2023-07-14T18:58:18.481 INFO:root:teuthology version: 0.0.1.dev145+gaa6a80e 2023-07-14T18:58:18.482 DEBUG:teuthology.run:Teuthology command: teuthology --name pdonnell-2023-07-14_16:44:47-fs:workload-wip-pdonnell-testing-20230714.134335-distro-default-smithi --description fs:workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/crc wsync/yes} objectstore-ec/bluestore-bitmap omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{balancer/random export-check n/3 replication/default} standby-replay tasks/{0-subvolume/{with-no-extra-options} 1-check-counter 2-scrub/yes 3-snaps/yes 4-flush/yes 5-workunit/suites/ffsb}} --owner scheduled_pdonnell@teuthology --verbose --archive /home/teuthworker/archive/pdonnell-2023-07-14_16:44:47-fs:workload-wip-pdonnell-testing-20230714.134335-distro-default-smithi/7338055 -- /home/teuthworker/archive/pdonnell-2023-07-14_16:44:47-fs:workload-wip-pdonnell-testing-20230714.134335-distro-default-smithi/7338055/orig.config.yaml 2023-07-14T18:58:18.554 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2023-07-14T18:58:23.655 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/pdonnell-2023-07-14_16:44:47-fs:workload-wip-pdonnell-testing-20230714.134335-distro-default-smithi/7338055 branch: wip-pdonnell-testing-20230714.134335 description: fs:workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/crc wsync/yes} objectstore-ec/bluestore-bitmap omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{balancer/random export-check n/3 replication/default} standby-replay tasks/{0-subvolume/{with-no-extra-options} 1-check-counter 2-scrub/yes 3-snaps/yes 4-flush/yes 5-workunit/suites/ffsb}} email: pdonnell@redhat.com first_in_suite: false job_id: '7338055' kernel: client: sha1: distro kdb: true sha1: distro ktype: distro last_in_suite: false log-rotate: ceph-mds: 10G ceph-osd: 10G machine_type: smithi name: pdonnell-2023-07-14_16:44:47-fs:workload-wip-pdonnell-testing-20230714.134335-distro-default-smithi no_nested_subset: false nuke-on-error: true openstack: - volumes: count: 4 size: 20 - machine: disk: 200 os_type: rhel os_version: '8.6' overrides: admin_socket: branch: wip-pdonnell-testing-20230714.134335 ceph: cephfs: max_mds: 3 session_timeout: 300 standby_replay: true conf: client: client mount timeout: 600 debug client: 20 debug ms: 1 rados mon op timeout: 900 rados osd op timeout: 900 global: ms die on skipped message: false mds: debug mds: 20 debug mds balancer: 20 debug ms: 1 mds bal fragment size max: 10000 mds bal merge size: 5 mds bal split bits: 3 mds bal split size: 100 mds debug frag: true mds debug scatterstat: true mds op complaint time: 180 mds verify scatter: true osd op complaint time: 180 rados mon op timeout: 900 rados osd op timeout: 900 mgr: cephsqlite lock renewal timeout: 900000 debug cephsqlite: 20 debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon op complaint time: 120 osd: bdev async discard: true bdev enable discard: true bluestore allocator: bitmap bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 1/20 debug bluestore: 1/20 debug ms: 1 debug osd: 20 debug rocksdb: 4/10 filestore flush min: 0 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd heartbeat grace: 60 osd objectstore: bluestore osd op complaint time: 180 osd shutdown pgref assert: true osd_max_omap_entries_per_request: 10 flavor: default fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - overall HEALTH_ - \(FS_DEGRADED\) - \(MDS_FAILED\) - \(MDS_DEGRADED\) - \(FS_WITH_FAILED_MDS\) - \(MDS_DAMAGE\) - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - \(FS_INLINE_DATA_DEPRECATED\) - overall HEALTH_ - \(OSD_DOWN\) - \(OSD_ - but it is still running - is not responding - slow metadata IO - SLOW_OPS - slow request - SLOW_OPS - slow request sha1: 81c9387d2e7576da065747ec9c16d1c7d69016ec subvols: create: 2 ceph-deploy: bluestore: true conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: osd default pool size: 2 osd: bdev async discard: true bdev enable discard: true bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 1/20 debug bluestore: 1/20 debug rocksdb: 4/10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore fs: xfs ceph-fuse: client.0: mount_subvol_num: 0 check-counter: counters: mds: - mds.exported - mds.imported - mds.root_rsnaps - mds_server.req_mksnap_latency.avgcount - mds_server.req_rmsnap_latency.avgcount - mds.dir_split install: ceph: flavor: default sha1: 81c9387d2e7576da065747ec9c16d1c7d69016ec kclient: client.0: mount_subvol_num: 1 mntopts: - ms_mode=crc - wsync syntax: v2 selinux: whitelist: - scontext=system_u:system_r:logrotate_t:s0 - scontext=system_u:system_r:logrotate_t:s0 thrashosds: bdev_inject_crash: 2 bdev_inject_crash_probability: 0.5 workunit: branch: wip-pdonnell-testing-20230714.134335 sha1: 81c9387d2e7576da065747ec9c16d1c7d69016ec owner: scheduled_pdonnell@teuthology priority: 75 repo: https://git.ceph.com/ceph-ci.git roles: - - mon.a - mgr.x - mds.a - mds.d - mds.g - mds.j - osd.0 - osd.3 - osd.6 - osd.9 - client.0 - - mon.b - mgr.y - mds.b - mds.e - mds.h - mds.k - osd.1 - osd.4 - osd.7 - osd.10 - - mon.c - mgr.z - mds.c - mds.f - mds.i - mds.l - osd.2 - osd.5 - osd.8 - osd.11 seed: 5179 sha1: 81c9387d2e7576da065747ec9c16d1c7d69016ec sleep_before_teardown: 0 subset: 285/512 suite: fs:workload suite_branch: wip-pdonnell-testing-20230714.134335 suite_path: /home/teuthworker/src/git.ceph.com_ceph-c_81c9387d2e7576da065747ec9c16d1c7d69016ec/qa suite_relpath: qa suite_repo: https://git.ceph.com/ceph-ci.git suite_sha1: 81c9387d2e7576da065747ec9c16d1c7d69016ec syslog: ignorelist: - WARNING*.*check_session_state - WARNING*.*__ceph_remove_cap targets: smithi098.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBLVvN3+LmzPZ4ixiwyCkS3KZLZ9R6MOPm+kPtaJbl8fExpzbk8TudXCcJWHBL7DGPWFvye2jT6sei9eiX0EUQYs= smithi101.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCpUxo5ay/ucdkX9rleNMm8Ofp24XRXUtJPOK3Ji3XBpTiUgmmwneaY6316+3CG+k4K0p3IebanuyTvZApjm02/dNCvu8yD3rjEpWVgIuqYH8O3lEdN9DXcCX+7xHJE0m18+ZPsJQVsNrNJxY+5zqyNnBc8jpZq2LStsWwPOKPWhauANRkOBFSGDoLppWZbLx/yJs+R0qoNLLK9LpKQTUnqsapXvbD27OpPahzj/qbNxFPPyY6XrrldZ6qEO+EjJOujFLH3gzv0kJu10DCuzxrpz6ston4RPi1O3GgcHnF4Mqr/W9VwZ94UTMh+k4dIFRVqJEKrcwRHa5kFsvH/WhVx0Ygr/MiV/Q86mmV7JT+/MpzIRTtWyei5L9IBeZIE9cwzj+mOk6SD/t1roIVEGVV/uF2AyFDVBPr6F8oZ822OGpUyyaDsWTitPNdTX4TC3zdSrLc9tyDa1P74eH/ZLeTSf+C6pM/cepOhQaa6XzpDiPSuJUm9HOQxUlEAvcqa7Ac= smithi139.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCpUxo5ay/ucdkX9rleNMm8Ofp24XRXUtJPOK3Ji3XBpTiUgmmwneaY6316+3CG+k4K0p3IebanuyTvZApjm02/dNCvu8yD3rjEpWVgIuqYH8O3lEdN9DXcCX+7xHJE0m18+ZPsJQVsNrNJxY+5zqyNnBc8jpZq2LStsWwPOKPWhauANRkOBFSGDoLppWZbLx/yJs+R0qoNLLK9LpKQTUnqsapXvbD27OpPahzj/qbNxFPPyY6XrrldZ6qEO+EjJOujFLH3gzv0kJu10DCuzxrpz6ston4RPi1O3GgcHnF4Mqr/W9VwZ94UTMh+k4dIFRVqJEKrcwRHa5kFsvH/WhVx0Ygr/MiV/Q86mmV7JT+/MpzIRTtWyei5L9IBeZIE9cwzj+mOk6SD/t1roIVEGVV/uF2AyFDVBPr6F8oZ822OGpUyyaDsWTitPNdTX4TC3zdSrLc9tyDa1P74eH/ZLeTSf+C6pM/cepOhQaa6XzpDiPSuJUm9HOQxUlEAvcqa7Ac= tasks: - pexec: all: - sudo cp /etc/containers/registries.conf /etc/containers/registries.conf.backup - sudo dnf -y module reset container-tools - sudo dnf -y module install container-tools:rhel8 --allowerasing --nobest - sudo cp /etc/containers/registries.conf.backup /etc/containers/registries.conf - install: extra_packages: deb: - python3-cephfs - cephfs-shell - cephfs-top - cephfs-mirror rpm: - python3-cephfs - cephfs-top - cephfs-mirror extra_system_packages: deb: - bison - flex - libelf-dev - libssl-dev - network-manager - iproute2 - util-linux - dump - indent - libaio-dev - libtool-bin - uuid-dev - xfslibs-dev - postgresql - postgresql-client - postgresql-common - postgresql-contrib rpm: - bison - flex - elfutils-libelf-devel - openssl-devel - NetworkManager - iproute - util-linux - libacl-devel - libaio-devel - libattr-devel - libtool - libuuid-devel - xfsdump - xfsprogs - xfsprogs-devel - libaio-devel - libtool - libuuid-devel - xfsprogs-devel - postgresql - postgresql-server - postgresql-contrib - cephadm: roleless: false - cephadm.shell: mon.a: - ceph orch status - ceph orch ps - ceph orch ls - ceph orch host ls - ceph orch device ls - cephadm.shell: mon.a: - ceph fs dump - ceph osd dump - fs.ready: timeout: 300 - kclient: null - exec: mon.a: - ceph fs set cephfs balance_automate false - ceph fs subvolumegroup pin cephfs _nogroup random 0.01 - check-counter: {} - fwd_scrub: scrub_timeout: 900 sleep_between_iterations: 1 - exec: mon.a: - ceph mgr module enable snap_schedule - ceph config set mgr mgr/snap_schedule/allow_m_granularity true - ceph config set mgr mgr/snap_schedule/dump_on_update true - ceph fs snap-schedule add --fs=cephfs --path=/ --snap_schedule=1M - ceph fs snap-schedule retention add --fs=cephfs --path=/ --retention-spec-or-period=6M3h - ceph fs snap-schedule status --fs=cephfs --path=/ - ceph fs snap-schedule list --fs=cephfs --path=/ --recursive=true - date +%s > START_TIME - full_sequential_finally: - exec: mon.a: - date +%s > END_TIME - START_TIME=$(cat START_TIME); END_TIME=$(cat END_TIME); DIFF_TIME=$((600-(END_TIME-START_TIME))); if [ "$DIFF_TIME" -gt 0 ]; then sleep "$DIFF_TIME"; fi - ceph fs snap-schedule status --fs=cephfs --path=/ - ceph fs snap-schedule list --fs=cephfs --path=/ --recursive=true - background_exec: mon.a: - while sleep 13; do ceph tell mds.cephfs:0 flush journal; done - workunit: clients: all: - suites/ffsb.sh teuthology: fragments_dropped: [] meta: {} postmerge: - local function is_kupstream() return yaml.ktype == 'upstream' end - local function is_kdistro() return yaml.ktype == 'distro' end - local function is_fuse() return false end - local function syntax_version() return yaml.overrides.kclient.syntax end teuthology_branch: main teuthology_sha1: aa6a80e6cc655fff48452a7eca02070e61d074fc timestamp: 2023-07-14_16:44:47 tube: smithi user: pdonnell verbose: false worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.23459 2023-07-14T18:58:23.656 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/git.ceph.com_ceph-c_81c9387d2e7576da065747ec9c16d1c7d69016ec/qa; will attempt to use it 2023-07-14T18:58:23.656 INFO:teuthology.run:Found tasks at /home/teuthworker/src/git.ceph.com_ceph-c_81c9387d2e7576da065747ec9c16d1c7d69016ec/qa/tasks 2023-07-14T18:58:23.657 INFO:teuthology.run_tasks:Running task internal.check_packages... 2023-07-14T18:58:23.659 INFO:teuthology.task.internal:Checking packages... 2023-07-14T18:58:23.687 INFO:teuthology.task.internal:Checking packages for os_type 'rhel', flavor 'default' and ceph hash '81c9387d2e7576da065747ec9c16d1c7d69016ec' 2023-07-14T18:58:23.688 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2023-07-14T18:58:23.688 INFO:teuthology.packaging:ref: None 2023-07-14T18:58:23.688 INFO:teuthology.packaging:tag: None 2023-07-14T18:58:23.689 INFO:teuthology.packaging:branch: wip-pdonnell-testing-20230714.134335 2023-07-14T18:58:23.689 INFO:teuthology.packaging:sha1: 81c9387d2e7576da065747ec9c16d1c7d69016ec 2023-07-14T18:58:23.689 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&ref=wip-pdonnell-testing-20230714.134335 2023-07-14T18:58:38.922 INFO:teuthology.task.internal:Found packages for ceph version 18.0.0-4940.g81c9387d 2023-07-14T18:58:38.922 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2023-07-14T18:58:38.935 INFO:teuthology.task.internal:no buildpackages task found 2023-07-14T18:58:38.935 INFO:teuthology.run_tasks:Running task internal.save_config... 2023-07-14T18:58:38.941 INFO:teuthology.task.internal:Saving configuration 2023-07-14T18:58:38.963 INFO:teuthology.run_tasks:Running task internal.check_lock... 2023-07-14T18:58:38.969 INFO:teuthology.task.internal.check_lock:Checking locks... 2023-07-14T18:58:44.002 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi098.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/pdonnell-2023-07-14_16:44:47-fs:workload-wip-pdonnell-testing-20230714.134335-distro-default-smithi/7338055', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'rhel', 'os_version': '8.6', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-07-14 18:51:51.079221', 'locked_by': 'scheduled_pdonnell@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBLVvN3+LmzPZ4ixiwyCkS3KZLZ9R6MOPm+kPtaJbl8fExpzbk8TudXCcJWHBL7DGPWFvye2jT6sei9eiX0EUQYs='} 2023-07-14T18:58:49.033 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi101.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/pdonnell-2023-07-14_16:44:47-fs:workload-wip-pdonnell-testing-20230714.134335-distro-default-smithi/7338055', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'rhel', 'os_version': '8.6', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-07-14 18:51:51.082748', 'locked_by': 'scheduled_pdonnell@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBLVvN3+LmzPZ4ixiwyCkS3KZLZ9R6MOPm+kPtaJbl8fExpzbk8TudXCcJWHBL7DGPWFvye2jT6sei9eiX0EUQYs='} 2023-07-14T18:58:54.067 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi139.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/pdonnell-2023-07-14_16:44:47-fs:workload-wip-pdonnell-testing-20230714.134335-distro-default-smithi/7338055', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'rhel', 'os_version': '8.6', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-07-14 18:51:51.081325', 'locked_by': 'scheduled_pdonnell@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBLVvN3+LmzPZ4ixiwyCkS3KZLZ9R6MOPm+kPtaJbl8fExpzbk8TudXCcJWHBL7DGPWFvye2jT6sei9eiX0EUQYs='} 2023-07-14T18:58:54.067 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2023-07-14T18:58:54.080 INFO:teuthology.task.internal:roles: ubuntu@smithi098.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'mds.a', 'mds.d', 'mds.g', 'mds.j', 'osd.0', 'osd.3', 'osd.6', 'osd.9', 'client.0'] 2023-07-14T18:58:54.081 INFO:teuthology.task.internal:roles: ubuntu@smithi101.front.sepia.ceph.com - ['mon.b', 'mgr.y', 'mds.b', 'mds.e', 'mds.h', 'mds.k', 'osd.1', 'osd.4', 'osd.7', 'osd.10'] 2023-07-14T18:58:54.081 INFO:teuthology.task.internal:roles: ubuntu@smithi139.front.sepia.ceph.com - ['mon.c', 'mgr.z', 'mds.c', 'mds.f', 'mds.i', 'mds.l', 'osd.2', 'osd.5', 'osd.8', 'osd.11'] 2023-07-14T18:58:54.082 INFO:teuthology.run_tasks:Running task console_log... 2023-07-14T18:59:09.221 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7f6df21bb430>, signals=[15]) 2023-07-14T18:59:09.222 INFO:teuthology.run_tasks:Running task internal.connect... 2023-07-14T18:59:09.230 INFO:teuthology.task.internal:Opening connections... 2023-07-14T18:59:09.231 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi098.front.sepia.ceph.com 2023-07-14T18:59:09.233 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi098.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-07-14T18:59:14.304 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi101.front.sepia.ceph.com 2023-07-14T18:59:14.305 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi101.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-07-14T18:59:19.378 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi139.front.sepia.ceph.com 2023-07-14T18:59:19.379 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi139.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-07-14T18:59:24.453 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2023-07-14T18:59:24.483 DEBUG:teuthology.orchestra.run.smithi098:> uname -m 2023-07-14T18:59:24.505 INFO:teuthology.orchestra.run.smithi098.stdout:x86_64 2023-07-14T18:59:24.505 DEBUG:teuthology.orchestra.run.smithi098:> cat /etc/os-release 2023-07-14T18:59:24.564 INFO:teuthology.orchestra.run.smithi098.stdout:NAME="Red Hat Enterprise Linux" 2023-07-14T18:59:24.565 INFO:teuthology.orchestra.run.smithi098.stdout:VERSION="8.6 (Ootpa)" 2023-07-14T18:59:24.565 INFO:teuthology.orchestra.run.smithi098.stdout:ID="rhel" 2023-07-14T18:59:24.565 INFO:teuthology.orchestra.run.smithi098.stdout:ID_LIKE="fedora" 2023-07-14T18:59:24.565 INFO:teuthology.orchestra.run.smithi098.stdout:VERSION_ID="8.6" 2023-07-14T18:59:24.565 INFO:teuthology.orchestra.run.smithi098.stdout:PLATFORM_ID="platform:el8" 2023-07-14T18:59:24.565 INFO:teuthology.orchestra.run.smithi098.stdout:PRETTY_NAME="Red Hat Enterprise Linux 8.6 (Ootpa)" 2023-07-14T18:59:24.566 INFO:teuthology.orchestra.run.smithi098.stdout:ANSI_COLOR="0;31" 2023-07-14T18:59:24.566 INFO:teuthology.orchestra.run.smithi098.stdout:CPE_NAME="cpe:/o:redhat:enterprise_linux:8::baseos" 2023-07-14T18:59:24.566 INFO:teuthology.orchestra.run.smithi098.stdout:HOME_URL="https://www.redhat.com/" 2023-07-14T18:59:24.566 INFO:teuthology.orchestra.run.smithi098.stdout:DOCUMENTATION_URL="https://access.redhat.com/documentation/red_hat_enterprise_linux/8/" 2023-07-14T18:59:24.566 INFO:teuthology.orchestra.run.smithi098.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2023-07-14T18:59:24.566 INFO:teuthology.orchestra.run.smithi098.stdout: 2023-07-14T18:59:24.566 INFO:teuthology.orchestra.run.smithi098.stdout:REDHAT_BUGZILLA_PRODUCT="Red Hat Enterprise Linux 8" 2023-07-14T18:59:24.567 INFO:teuthology.orchestra.run.smithi098.stdout:REDHAT_BUGZILLA_PRODUCT_VERSION=8.6 2023-07-14T18:59:24.567 INFO:teuthology.orchestra.run.smithi098.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux" 2023-07-14T18:59:24.567 INFO:teuthology.orchestra.run.smithi098.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="8.6" 2023-07-14T18:59:24.568 INFO:teuthology.lock.ops:Updating smithi098.front.sepia.ceph.com on lock server 2023-07-14T18:59:29.602 DEBUG:teuthology.orchestra.run.smithi101:> uname -m 2023-07-14T18:59:29.623 INFO:teuthology.orchestra.run.smithi101.stdout:x86_64 2023-07-14T18:59:29.624 DEBUG:teuthology.orchestra.run.smithi101:> cat /etc/os-release 2023-07-14T18:59:29.684 INFO:teuthology.orchestra.run.smithi101.stdout:NAME="Red Hat Enterprise Linux" 2023-07-14T18:59:29.684 INFO:teuthology.orchestra.run.smithi101.stdout:VERSION="8.6 (Ootpa)" 2023-07-14T18:59:29.684 INFO:teuthology.orchestra.run.smithi101.stdout:ID="rhel" 2023-07-14T18:59:29.685 INFO:teuthology.orchestra.run.smithi101.stdout:ID_LIKE="fedora" 2023-07-14T18:59:29.685 INFO:teuthology.orchestra.run.smithi101.stdout:VERSION_ID="8.6" 2023-07-14T18:59:29.685 INFO:teuthology.orchestra.run.smithi101.stdout:PLATFORM_ID="platform:el8" 2023-07-14T18:59:29.686 INFO:teuthology.orchestra.run.smithi101.stdout:PRETTY_NAME="Red Hat Enterprise Linux 8.6 (Ootpa)" 2023-07-14T18:59:29.686 INFO:teuthology.orchestra.run.smithi101.stdout:ANSI_COLOR="0;31" 2023-07-14T18:59:29.686 INFO:teuthology.orchestra.run.smithi101.stdout:CPE_NAME="cpe:/o:redhat:enterprise_linux:8::baseos" 2023-07-14T18:59:29.686 INFO:teuthology.orchestra.run.smithi101.stdout:HOME_URL="https://www.redhat.com/" 2023-07-14T18:59:29.687 INFO:teuthology.orchestra.run.smithi101.stdout:DOCUMENTATION_URL="https://access.redhat.com/documentation/red_hat_enterprise_linux/8/" 2023-07-14T18:59:29.687 INFO:teuthology.orchestra.run.smithi101.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2023-07-14T18:59:29.687 INFO:teuthology.orchestra.run.smithi101.stdout: 2023-07-14T18:59:29.688 INFO:teuthology.orchestra.run.smithi101.stdout:REDHAT_BUGZILLA_PRODUCT="Red Hat Enterprise Linux 8" 2023-07-14T18:59:29.688 INFO:teuthology.orchestra.run.smithi101.stdout:REDHAT_BUGZILLA_PRODUCT_VERSION=8.6 2023-07-14T18:59:29.688 INFO:teuthology.orchestra.run.smithi101.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux" 2023-07-14T18:59:29.689 INFO:teuthology.orchestra.run.smithi101.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="8.6" 2023-07-14T18:59:29.689 INFO:teuthology.lock.ops:Updating smithi101.front.sepia.ceph.com on lock server 2023-07-14T18:59:34.719 DEBUG:teuthology.orchestra.run.smithi139:> uname -m 2023-07-14T18:59:34.741 INFO:teuthology.orchestra.run.smithi139.stdout:x86_64 2023-07-14T18:59:34.741 DEBUG:teuthology.orchestra.run.smithi139:> cat /etc/os-release 2023-07-14T18:59:34.800 INFO:teuthology.orchestra.run.smithi139.stdout:NAME="Red Hat Enterprise Linux" 2023-07-14T18:59:34.801 INFO:teuthology.orchestra.run.smithi139.stdout:VERSION="8.6 (Ootpa)" 2023-07-14T18:59:34.801 INFO:teuthology.orchestra.run.smithi139.stdout:ID="rhel" 2023-07-14T18:59:34.801 INFO:teuthology.orchestra.run.smithi139.stdout:ID_LIKE="fedora" 2023-07-14T18:59:34.801 INFO:teuthology.orchestra.run.smithi139.stdout:VERSION_ID="8.6" 2023-07-14T18:59:34.801 INFO:teuthology.orchestra.run.smithi139.stdout:PLATFORM_ID="platform:el8" 2023-07-14T18:59:34.801 INFO:teuthology.orchestra.run.smithi139.stdout:PRETTY_NAME="Red Hat Enterprise Linux 8.6 (Ootpa)" 2023-07-14T18:59:34.802 INFO:teuthology.orchestra.run.smithi139.stdout:ANSI_COLOR="0;31" 2023-07-14T18:59:34.802 INFO:teuthology.orchestra.run.smithi139.stdout:CPE_NAME="cpe:/o:redhat:enterprise_linux:8::baseos" 2023-07-14T18:59:34.802 INFO:teuthology.orchestra.run.smithi139.stdout:HOME_URL="https://www.redhat.com/" 2023-07-14T18:59:34.802 INFO:teuthology.orchestra.run.smithi139.stdout:DOCUMENTATION_URL="https://access.redhat.com/documentation/red_hat_enterprise_linux/8/" 2023-07-14T18:59:34.802 INFO:teuthology.orchestra.run.smithi139.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2023-07-14T18:59:34.802 INFO:teuthology.orchestra.run.smithi139.stdout: 2023-07-14T18:59:34.802 INFO:teuthology.orchestra.run.smithi139.stdout:REDHAT_BUGZILLA_PRODUCT="Red Hat Enterprise Linux 8" 2023-07-14T18:59:34.803 INFO:teuthology.orchestra.run.smithi139.stdout:REDHAT_BUGZILLA_PRODUCT_VERSION=8.6 2023-07-14T18:59:34.803 INFO:teuthology.orchestra.run.smithi139.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux" 2023-07-14T18:59:34.803 INFO:teuthology.orchestra.run.smithi139.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="8.6" 2023-07-14T18:59:34.803 INFO:teuthology.lock.ops:Updating smithi139.front.sepia.ceph.com on lock server 2023-07-14T18:59:39.834 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2023-07-14T18:59:39.852 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2023-07-14T18:59:39.858 INFO:teuthology.task.internal:Checking for old test directory... 2023-07-14T18:59:39.858 DEBUG:teuthology.orchestra.run.smithi098:> test '!' -e /home/ubuntu/cephtest 2023-07-14T18:59:39.861 DEBUG:teuthology.orchestra.run.smithi101:> test '!' -e /home/ubuntu/cephtest 2023-07-14T18:59:39.863 DEBUG:teuthology.orchestra.run.smithi139:> test '!' -e /home/ubuntu/cephtest 2023-07-14T18:59:39.882 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2023-07-14T18:59:39.888 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2023-07-14T18:59:39.889 DEBUG:teuthology.orchestra.run.smithi098:> test -z $(ls -A /var/lib/ceph) 2023-07-14T18:59:39.920 DEBUG:teuthology.orchestra.run.smithi101:> test -z $(ls -A /var/lib/ceph) 2023-07-14T18:59:39.922 DEBUG:teuthology.orchestra.run.smithi139:> test -z $(ls -A /var/lib/ceph) 2023-07-14T18:59:39.960 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2023-07-14T18:59:49.432 INFO:teuthology.run_tasks:Running task kernel... 2023-07-14T18:59:49.449 INFO:teuthology.task.kernel:normalize config orig: {'client': {'sha1': 'distro'}, 'kdb': True, 'sha1': 'distro'} 2023-07-14T18:59:49.450 INFO:teuthology.task.kernel:normalize config final: {'client.0': {'sha1': 'distro'}} 2023-07-14T18:59:49.451 INFO:teuthology.task.kernel:config {'client.0': {'sha1': 'distro'}}, timeout 300 2023-07-14T18:59:49.451 DEBUG:teuthology.orchestra.run.smithi098:> test -f /run/.containerenv -o -f /.dockerenv 2023-07-14T18:59:49.470 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-07-14T18:59:49.471 DEBUG:teuthology.orchestra.run.smithi098:> uname -r 2023-07-14T18:59:49.531 INFO:teuthology.orchestra.run.smithi098.stdout:4.18.0-372.9.1.el8.x86_64 2023-07-14T18:59:49.531 INFO:teuthology.task.kernel:Running kernel on smithi098: 4.18.0-372.9.1.el8.x86_64 2023-07-14T18:59:49.531 DEBUG:teuthology.orchestra.run.smithi098:> sudo yum install -y kernel 2023-07-14T18:59:50.791 INFO:teuthology.orchestra.run.smithi098.stdout:Updating Subscription Management repositories. 2023-07-14T18:59:50.792 INFO:teuthology.orchestra.run.smithi098.stdout:Unable to read consumer identity 2023-07-14T18:59:52.626 INFO:teuthology.orchestra.run.smithi098.stdout:Copr repo for python3-asyncssh owned by ceph 7.3 kB/s | 3.5 kB 00:00 2023-07-14T18:59:57.392 INFO:teuthology.orchestra.run.smithi098.stdout:Extra Packages for Enterprise Linux 3.3 MB/s | 16 MB 00:04 2023-07-14T19:00:00.818 INFO:teuthology.orchestra.run.smithi098.stdout:lab-extras 444 kB/s | 24 kB 00:00 2023-07-14T19:00:01.512 INFO:teuthology.orchestra.run.smithi098.stdout:Package kernel-4.18.0-372.9.1.el8.x86_64 is already installed. 2023-07-14T19:00:01.567 INFO:teuthology.orchestra.run.smithi098.stdout:Dependencies resolved. 2023-07-14T19:00:01.568 INFO:teuthology.orchestra.run.smithi098.stdout:Nothing to do. 2023-07-14T19:00:01.568 INFO:teuthology.orchestra.run.smithi098.stdout:Complete! 2023-07-14T19:00:01.691 DEBUG:teuthology.orchestra.run.smithi098:> echo no | sudo yum reinstall kernel || true 2023-07-14T19:00:02.022 INFO:teuthology.orchestra.run.smithi098.stdout:Updating Subscription Management repositories. 2023-07-14T19:00:02.022 INFO:teuthology.orchestra.run.smithi098.stdout:Unable to read consumer identity 2023-07-14T19:00:02.290 INFO:teuthology.orchestra.run.smithi098.stdout:Last metadata expiration check: 0:00:02 ago on Fri 14 Jul 2023 07:00:00 PM UTC. 2023-07-14T19:00:02.331 INFO:teuthology.orchestra.run.smithi098.stdout:Installed package kernel-4.18.0-372.9.1.el8.x86_64 (from anaconda) not available. 2023-07-14T19:00:02.372 INFO:teuthology.orchestra.run.smithi098.stderr:Error: No packages marked for reinstall. 2023-07-14T19:00:02.425 DEBUG:teuthology.orchestra.run.smithi098:> sudo yum reinstall -y kernel || true 2023-07-14T19:00:02.753 INFO:teuthology.orchestra.run.smithi098.stdout:Updating Subscription Management repositories. 2023-07-14T19:00:02.754 INFO:teuthology.orchestra.run.smithi098.stdout:Unable to read consumer identity 2023-07-14T19:00:03.006 INFO:teuthology.orchestra.run.smithi098.stdout:Last metadata expiration check: 0:00:03 ago on Fri 14 Jul 2023 07:00:00 PM UTC. 2023-07-14T19:00:03.045 INFO:teuthology.orchestra.run.smithi098.stdout:Installed package kernel-4.18.0-372.9.1.el8.x86_64 (from anaconda) not available. 2023-07-14T19:00:03.076 INFO:teuthology.orchestra.run.smithi098.stderr:Error: No packages marked for reinstall. 2023-07-14T19:00:03.122 DEBUG:teuthology.orchestra.run.smithi098:> rpm -q kernel | sort -rV | head -n 1 2023-07-14T19:00:03.203 INFO:teuthology.orchestra.run.smithi098.stdout:kernel-4.18.0-372.9.1.el8.x86_64 2023-07-14T19:00:03.204 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 4.18.0-372.9.1.el8.x86_64 2023-07-14T19:00:03.204 INFO:teuthology.task.kernel:Newest distro kernel installed and running 2023-07-14T19:00:03.204 DEBUG:teuthology.parallel:result is None 2023-07-14T19:00:03.204 INFO:teuthology.run_tasks:Running task internal.base... 2023-07-14T19:00:03.218 INFO:teuthology.task.internal:Creating test directory... 2023-07-14T19:00:03.219 DEBUG:teuthology.orchestra.run.smithi098:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2023-07-14T19:00:03.221 DEBUG:teuthology.orchestra.run.smithi101:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2023-07-14T19:00:03.223 DEBUG:teuthology.orchestra.run.smithi139:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2023-07-14T19:00:03.244 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2023-07-14T19:00:03.254 INFO:teuthology.run_tasks:Running task internal.archive... 2023-07-14T19:00:03.264 INFO:teuthology.task.internal:Creating archive directory... 2023-07-14T19:00:03.265 DEBUG:teuthology.orchestra.run.smithi098:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2023-07-14T19:00:03.279 DEBUG:teuthology.orchestra.run.smithi101:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2023-07-14T19:00:03.283 DEBUG:teuthology.orchestra.run.smithi139:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2023-07-14T19:00:03.326 INFO:teuthology.run_tasks:Running task internal.coredump... 2023-07-14T19:00:03.334 INFO:teuthology.task.internal:Enabling coredump saving... 2023-07-14T19:00:03.334 DEBUG:teuthology.orchestra.run.smithi101:> test -f /run/.containerenv -o -f /.dockerenv 2023-07-14T19:00:03.371 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-07-14T19:00:03.372 DEBUG:teuthology.orchestra.run.smithi139:> test -f /run/.containerenv -o -f /.dockerenv 2023-07-14T19:00:03.391 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-07-14T19:00:03.393 DEBUG:teuthology.orchestra.run.smithi098:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2023-07-14T19:00:03.396 DEBUG:teuthology.orchestra.run.smithi101:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2023-07-14T19:00:03.415 DEBUG:teuthology.orchestra.run.smithi139:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2023-07-14T19:00:03.431 INFO:teuthology.orchestra.run.smithi098.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-07-14T19:00:03.442 INFO:teuthology.orchestra.run.smithi098.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-07-14T19:00:03.481 INFO:teuthology.orchestra.run.smithi101.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-07-14T19:00:03.502 INFO:teuthology.orchestra.run.smithi101.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-07-14T19:00:03.503 INFO:teuthology.orchestra.run.smithi139.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-07-14T19:00:03.526 INFO:teuthology.orchestra.run.smithi139.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-07-14T19:00:03.528 INFO:teuthology.run_tasks:Running task internal.sudo... 2023-07-14T19:00:03.540 INFO:teuthology.task.internal:Configuring sudo... 2023-07-14T19:00:03.541 DEBUG:teuthology.orchestra.run.smithi098:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2023-07-14T19:00:03.543 DEBUG:teuthology.orchestra.run.smithi101:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2023-07-14T19:00:03.545 DEBUG:teuthology.orchestra.run.smithi139:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2023-07-14T19:00:03.574 INFO:teuthology.run_tasks:Running task internal.syslog... 2023-07-14T19:00:03.582 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2023-07-14T19:00:03.583 DEBUG:teuthology.orchestra.run.smithi098:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2023-07-14T19:00:03.609 DEBUG:teuthology.orchestra.run.smithi101:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2023-07-14T19:00:03.613 DEBUG:teuthology.orchestra.run.smithi139:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2023-07-14T19:00:03.632 DEBUG:teuthology.orchestra.run.smithi098:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2023-07-14T19:00:08.716 DEBUG:teuthology.orchestra.run.smithi098:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2023-07-14T19:00:08.753 DEBUG:teuthology.orchestra.run.smithi098:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2023-07-14T19:00:11.447 DEBUG:teuthology.orchestra.run.smithi098:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2023-07-14T19:00:11.479 DEBUG:teuthology.orchestra.run.smithi098:> set -ex 2023-07-14T19:00:11.480 DEBUG:teuthology.orchestra.run.smithi098:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2023-07-14T19:00:11.551 DEBUG:teuthology.orchestra.run.smithi101:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2023-07-14T19:00:11.601 DEBUG:teuthology.orchestra.run.smithi101:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2023-07-14T19:00:11.653 DEBUG:teuthology.orchestra.run.smithi101:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2023-07-14T19:00:11.740 DEBUG:teuthology.orchestra.run.smithi101:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2023-07-14T19:00:11.782 DEBUG:teuthology.orchestra.run.smithi101:> set -ex 2023-07-14T19:00:11.782 DEBUG:teuthology.orchestra.run.smithi101:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2023-07-14T19:00:11.851 DEBUG:teuthology.orchestra.run.smithi139:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2023-07-14T19:00:16.881 DEBUG:teuthology.orchestra.run.smithi139:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2023-07-14T19:00:16.926 DEBUG:teuthology.orchestra.run.smithi139:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2023-07-14T19:00:17.014 DEBUG:teuthology.orchestra.run.smithi139:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2023-07-14T19:00:17.057 DEBUG:teuthology.orchestra.run.smithi139:> set -ex 2023-07-14T19:00:17.057 DEBUG:teuthology.orchestra.run.smithi139:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2023-07-14T19:00:17.126 DEBUG:teuthology.orchestra.run.smithi098:> sudo service rsyslog restart 2023-07-14T19:00:17.128 DEBUG:teuthology.orchestra.run.smithi101:> sudo service rsyslog restart 2023-07-14T19:00:17.130 DEBUG:teuthology.orchestra.run.smithi139:> sudo service rsyslog restart 2023-07-14T19:00:17.190 INFO:teuthology.orchestra.run.smithi101.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2023-07-14T19:00:17.192 INFO:teuthology.orchestra.run.smithi098.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2023-07-14T19:00:17.222 INFO:teuthology.orchestra.run.smithi139.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2023-07-14T19:00:17.546 INFO:teuthology.run_tasks:Running task internal.timer... 2023-07-14T19:00:17.554 INFO:teuthology.task.internal:Starting timer... 2023-07-14T19:00:17.554 INFO:teuthology.run_tasks:Running task pcp... 2023-07-14T19:00:17.565 INFO:teuthology.run_tasks:Running task selinux... 2023-07-14T19:00:17.573 DEBUG:teuthology.task:Applying overrides for task selinux: {'whitelist': ['scontext=system_u:system_r:logrotate_t:s0', 'scontext=system_u:system_r:logrotate_t:s0']} 2023-07-14T19:00:17.574 DEBUG:teuthology.orchestra.run.smithi098:> sudo service auditd rotate 2023-07-14T19:00:17.643 INFO:teuthology.orchestra.run.smithi098.stdout:Rotating logs: 2023-07-14T19:00:17.644 DEBUG:teuthology.orchestra.run.smithi101:> sudo service auditd rotate 2023-07-14T19:00:17.707 INFO:teuthology.orchestra.run.smithi101.stdout:Rotating logs: 2023-07-14T19:00:17.709 DEBUG:teuthology.orchestra.run.smithi139:> sudo service auditd rotate 2023-07-14T19:00:17.767 INFO:teuthology.orchestra.run.smithi139.stdout:Rotating logs: 2023-07-14T19:00:17.769 DEBUG:teuthology.task.selinux:Getting current SELinux state 2023-07-14T19:00:17.769 DEBUG:teuthology.orchestra.run.smithi098:> /usr/sbin/getenforce 2023-07-14T19:00:17.791 INFO:teuthology.orchestra.run.smithi098.stdout:Permissive 2023-07-14T19:00:17.791 DEBUG:teuthology.orchestra.run.smithi101:> /usr/sbin/getenforce 2023-07-14T19:00:17.824 INFO:teuthology.orchestra.run.smithi101.stdout:Permissive 2023-07-14T19:00:17.825 DEBUG:teuthology.orchestra.run.smithi139:> /usr/sbin/getenforce 2023-07-14T19:00:17.854 INFO:teuthology.orchestra.run.smithi139.stdout:Permissive 2023-07-14T19:00:17.854 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi098.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi101.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi139.front.sepia.ceph.com': 'permissive'} 2023-07-14T19:00:17.854 DEBUG:teuthology.orchestra.run.smithi098:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2023-07-14T19:00:17.881 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-07-14T19:00:17.882 DEBUG:teuthology.orchestra.run.smithi101:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2023-07-14T19:00:17.908 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-07-14T19:00:17.909 DEBUG:teuthology.orchestra.run.smithi139:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2023-07-14T19:00:17.937 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-07-14T19:00:17.937 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2023-07-14T19:00:17.938 DEBUG:teuthology.orchestra.run.smithi098:> sudo /usr/sbin/setenforce permissive 2023-07-14T19:00:17.965 DEBUG:teuthology.orchestra.run.smithi101:> sudo /usr/sbin/setenforce permissive 2023-07-14T19:00:17.992 DEBUG:teuthology.orchestra.run.smithi139:> sudo /usr/sbin/setenforce permissive 2023-07-14T19:00:18.020 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2023-07-14T19:00:18.029 INFO:teuthology.repo_utils:/home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main was just updated or references a specific commit; assuming it is current 2023-07-14T19:00:18.029 INFO:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2023-07-14T19:00:18.049 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2023-07-14T19:00:18.051 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi098.front.sepia.ceph.com,smithi101.front.sepia.ceph.com,smithi139.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2023-07-14T19:15:19.585 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi098.front.sepia.ceph.com'), Remote(name='ubuntu@smithi101.front.sepia.ceph.com'), Remote(name='ubuntu@smithi139.front.sepia.ceph.com')] 2023-07-14T19:15:19.587 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi098.front.sepia.ceph.com' 2023-07-14T19:15:19.588 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi098.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-07-14T19:15:19.656 DEBUG:teuthology.orchestra.run.smithi098:> true 2023-07-14T19:15:19.680 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi098.front.sepia.ceph.com' 2023-07-14T19:15:19.681 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi101.front.sepia.ceph.com' 2023-07-14T19:15:19.681 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi101.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-07-14T19:15:19.748 DEBUG:teuthology.orchestra.run.smithi101:> true 2023-07-14T19:15:19.818 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi101.front.sepia.ceph.com' 2023-07-14T19:15:19.819 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi139.front.sepia.ceph.com' 2023-07-14T19:15:19.820 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi139.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-07-14T19:15:24.890 DEBUG:teuthology.orchestra.run.smithi139:> true 2023-07-14T19:15:24.964 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi139.front.sepia.ceph.com' 2023-07-14T19:15:24.965 INFO:teuthology.run_tasks:Running task clock... 2023-07-14T19:15:24.977 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2023-07-14T19:15:24.977 INFO:teuthology.orchestra.run:Running command with timeout 360 2023-07-14T19:15:24.978 DEBUG:teuthology.orchestra.run.smithi098:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2023-07-14T19:15:24.981 INFO:teuthology.orchestra.run:Running command with timeout 360 2023-07-14T19:15:24.981 DEBUG:teuthology.orchestra.run.smithi101:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2023-07-14T19:15:24.984 INFO:teuthology.orchestra.run:Running command with timeout 360 2023-07-14T19:15:24.984 DEBUG:teuthology.orchestra.run.smithi139:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2023-07-14T19:15:25.015 INFO:teuthology.orchestra.run.smithi098.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2023-07-14T19:15:25.016 INFO:teuthology.orchestra.run.smithi101.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2023-07-14T19:15:25.031 INFO:teuthology.orchestra.run.smithi098.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2023-07-14T19:15:25.033 INFO:teuthology.orchestra.run.smithi101.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2023-07-14T19:15:25.042 INFO:teuthology.orchestra.run.smithi139.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2023-07-14T19:15:25.058 INFO:teuthology.orchestra.run.smithi139.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2023-07-14T19:15:25.100 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: ntpd: command not found 2023-07-14T19:15:25.104 INFO:teuthology.orchestra.run.smithi101.stderr:sudo: ntpd: command not found 2023-07-14T19:15:25.112 INFO:teuthology.orchestra.run.smithi098.stdout:506 Cannot talk to daemon 2023-07-14T19:15:25.117 INFO:teuthology.orchestra.run.smithi101.stdout:506 Cannot talk to daemon 2023-07-14T19:15:25.127 INFO:teuthology.orchestra.run.smithi098.stderr:Failed to start ntp.service: Unit ntp.service not found. 2023-07-14T19:15:25.132 INFO:teuthology.orchestra.run.smithi101.stderr:Failed to start ntp.service: Unit ntp.service not found. 2023-07-14T19:15:25.139 INFO:teuthology.orchestra.run.smithi139.stderr:sudo: ntpd: command not found 2023-07-14T19:15:25.141 INFO:teuthology.orchestra.run.smithi098.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2023-07-14T19:15:25.146 INFO:teuthology.orchestra.run.smithi101.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2023-07-14T19:15:25.151 INFO:teuthology.orchestra.run.smithi139.stdout:506 Cannot talk to daemon 2023-07-14T19:15:25.166 INFO:teuthology.orchestra.run.smithi139.stderr:Failed to start ntp.service: Unit ntp.service not found. 2023-07-14T19:15:25.181 INFO:teuthology.orchestra.run.smithi139.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2023-07-14T19:15:25.220 INFO:teuthology.orchestra.run.smithi098.stderr:bash: ntpq: command not found 2023-07-14T19:15:25.224 INFO:teuthology.orchestra.run.smithi098.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2023-07-14T19:15:25.224 INFO:teuthology.orchestra.run.smithi098.stdout:=============================================================================== 2023-07-14T19:15:25.224 INFO:teuthology.orchestra.run.smithi098.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-07-14T19:15:25.224 INFO:teuthology.orchestra.run.smithi098.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-07-14T19:15:25.224 INFO:teuthology.orchestra.run.smithi098.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-07-14T19:15:25.225 INFO:teuthology.orchestra.run.smithi098.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-07-14T19:15:25.233 INFO:teuthology.orchestra.run.smithi101.stderr:bash: ntpq: command not found 2023-07-14T19:15:25.237 INFO:teuthology.orchestra.run.smithi101.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2023-07-14T19:15:25.237 INFO:teuthology.orchestra.run.smithi101.stdout:=============================================================================== 2023-07-14T19:15:25.237 INFO:teuthology.orchestra.run.smithi101.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-07-14T19:15:25.238 INFO:teuthology.orchestra.run.smithi101.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-07-14T19:15:25.239 INFO:teuthology.orchestra.run.smithi101.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-07-14T19:15:25.239 INFO:teuthology.orchestra.run.smithi101.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-07-14T19:15:25.272 INFO:teuthology.orchestra.run.smithi139.stderr:bash: ntpq: command not found 2023-07-14T19:15:25.276 INFO:teuthology.orchestra.run.smithi139.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2023-07-14T19:15:25.276 INFO:teuthology.orchestra.run.smithi139.stdout:=============================================================================== 2023-07-14T19:15:25.277 INFO:teuthology.orchestra.run.smithi139.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-07-14T19:15:25.277 INFO:teuthology.orchestra.run.smithi139.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-07-14T19:15:25.277 INFO:teuthology.orchestra.run.smithi139.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-07-14T19:15:25.278 INFO:teuthology.orchestra.run.smithi139.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-07-14T19:15:25.279 INFO:teuthology.run_tasks:Running task pexec... 2023-07-14T19:15:25.292 INFO:teuthology.task.pexec:Executing custom commands... 2023-07-14T19:15:25.293 INFO:teuthology.task.pexec:Running commands on host ubuntu@smithi098.front.sepia.ceph.com 2023-07-14T19:15:25.293 DEBUG:teuthology.orchestra.run.smithi098:> TESTDIR=/home/ubuntu/cephtest bash -s 2023-07-14T19:15:25.293 INFO:teuthology.task.pexec:Running commands on host ubuntu@smithi101.front.sepia.ceph.com 2023-07-14T19:15:25.294 DEBUG:teuthology.orchestra.run.smithi101:> TESTDIR=/home/ubuntu/cephtest bash -s 2023-07-14T19:15:25.294 INFO:teuthology.task.pexec:Running commands on host ubuntu@smithi139.front.sepia.ceph.com 2023-07-14T19:15:25.294 DEBUG:teuthology.orchestra.run.smithi139:> TESTDIR=/home/ubuntu/cephtest bash -s 2023-07-14T19:15:25.667 INFO:teuthology.orchestra.run.smithi098.stdout:Updating Subscription Management repositories. 2023-07-14T19:15:25.669 INFO:teuthology.orchestra.run.smithi101.stdout:Updating Subscription Management repositories. 2023-07-14T19:15:25.690 INFO:teuthology.orchestra.run.smithi139.stdout:Updating Subscription Management repositories. 2023-07-14T19:15:26.524 INFO:teuthology.orchestra.run.smithi098.stdout:Last metadata expiration check: 0:05:42 ago on Fri 14 Jul 2023 07:09:44 PM UTC. 2023-07-14T19:15:26.533 INFO:teuthology.orchestra.run.smithi101.stdout:Last metadata expiration check: 0:01:30 ago on Fri 14 Jul 2023 07:13:56 PM UTC. 2023-07-14T19:15:26.540 INFO:teuthology.orchestra.run.smithi139.stdout:Last metadata expiration check: 0:06:14 ago on Fri 14 Jul 2023 07:09:12 PM UTC. 2023-07-14T19:15:28.287 INFO:teuthology.orchestra.run.smithi139.stdout:Dependencies resolved. 2023-07-14T19:15:28.288 INFO:teuthology.orchestra.run.smithi139.stdout:================================================================================ 2023-07-14T19:15:28.288 INFO:teuthology.orchestra.run.smithi139.stdout: Package Architecture Version Repository Size 2023-07-14T19:15:28.288 INFO:teuthology.orchestra.run.smithi139.stdout:================================================================================ 2023-07-14T19:15:28.289 INFO:teuthology.orchestra.run.smithi139.stdout:Resetting modules: 2023-07-14T19:15:28.289 INFO:teuthology.orchestra.run.smithi139.stdout: container-tools 2023-07-14T19:15:28.289 INFO:teuthology.orchestra.run.smithi139.stdout: 2023-07-14T19:15:28.289 INFO:teuthology.orchestra.run.smithi139.stdout:Transaction Summary 2023-07-14T19:15:28.290 INFO:teuthology.orchestra.run.smithi139.stdout:================================================================================ 2023-07-14T19:15:28.290 INFO:teuthology.orchestra.run.smithi139.stdout: 2023-07-14T19:15:28.293 INFO:teuthology.orchestra.run.smithi101.stdout:Dependencies resolved. 2023-07-14T19:15:28.294 INFO:teuthology.orchestra.run.smithi101.stdout:================================================================================ 2023-07-14T19:15:28.295 INFO:teuthology.orchestra.run.smithi101.stdout: Package Architecture Version Repository Size 2023-07-14T19:15:28.295 INFO:teuthology.orchestra.run.smithi101.stdout:================================================================================ 2023-07-14T19:15:28.295 INFO:teuthology.orchestra.run.smithi101.stdout:Resetting modules: 2023-07-14T19:15:28.295 INFO:teuthology.orchestra.run.smithi101.stdout: container-tools 2023-07-14T19:15:28.296 INFO:teuthology.orchestra.run.smithi101.stdout: 2023-07-14T19:15:28.296 INFO:teuthology.orchestra.run.smithi101.stdout:Transaction Summary 2023-07-14T19:15:28.296 INFO:teuthology.orchestra.run.smithi101.stdout:================================================================================ 2023-07-14T19:15:28.296 INFO:teuthology.orchestra.run.smithi101.stdout: 2023-07-14T19:15:28.408 INFO:teuthology.orchestra.run.smithi098.stdout:Dependencies resolved. 2023-07-14T19:15:28.410 INFO:teuthology.orchestra.run.smithi098.stdout:================================================================================ 2023-07-14T19:15:28.410 INFO:teuthology.orchestra.run.smithi098.stdout: Package Architecture Version Repository Size 2023-07-14T19:15:28.410 INFO:teuthology.orchestra.run.smithi098.stdout:================================================================================ 2023-07-14T19:15:28.410 INFO:teuthology.orchestra.run.smithi098.stdout:Resetting modules: 2023-07-14T19:15:28.410 INFO:teuthology.orchestra.run.smithi098.stdout: container-tools 2023-07-14T19:15:28.410 INFO:teuthology.orchestra.run.smithi098.stdout: 2023-07-14T19:15:28.411 INFO:teuthology.orchestra.run.smithi098.stdout:Transaction Summary 2023-07-14T19:15:28.411 INFO:teuthology.orchestra.run.smithi098.stdout:================================================================================ 2023-07-14T19:15:28.411 INFO:teuthology.orchestra.run.smithi098.stdout: 2023-07-14T19:15:29.265 INFO:teuthology.orchestra.run.smithi101.stdout:Last metadata expiration check: 0:01:32 ago on Fri 14 Jul 2023 07:13:56 PM UTC. 2023-07-14T19:15:29.291 INFO:teuthology.orchestra.run.smithi139.stdout:Last metadata expiration check: 0:06:16 ago on Fri 14 Jul 2023 07:09:12 PM UTC. 2023-07-14T19:15:29.374 INFO:teuthology.orchestra.run.smithi098.stdout:Last metadata expiration check: 0:05:44 ago on Fri 14 Jul 2023 07:09:44 PM UTC. 2023-07-14T19:15:32.641 INFO:teuthology.orchestra.run.smithi098.stdout:Complete! 2023-07-14T19:15:32.764 INFO:teuthology.orchestra.run.smithi101.stdout:Complete! 2023-07-14T19:15:32.785 INFO:teuthology.orchestra.run.smithi139.stdout:Complete! 2023-07-14T19:15:33.094 INFO:teuthology.orchestra.run.smithi098.stdout:Updating Subscription Management repositories. 2023-07-14T19:15:33.239 INFO:teuthology.orchestra.run.smithi101.stdout:Updating Subscription Management repositories. 2023-07-14T19:15:33.296 INFO:teuthology.orchestra.run.smithi139.stdout:Updating Subscription Management repositories. 2023-07-14T19:15:33.899 INFO:teuthology.orchestra.run.smithi098.stdout:Last metadata expiration check: 0:05:49 ago on Fri 14 Jul 2023 07:09:44 PM UTC. 2023-07-14T19:15:34.115 INFO:teuthology.orchestra.run.smithi101.stdout:Last metadata expiration check: 0:01:37 ago on Fri 14 Jul 2023 07:13:56 PM UTC. 2023-07-14T19:15:34.133 INFO:teuthology.orchestra.run.smithi139.stdout:Last metadata expiration check: 0:06:21 ago on Fri 14 Jul 2023 07:09:12 PM UTC. 2023-07-14T19:15:35.767 INFO:teuthology.orchestra.run.smithi098.stdout:Dependencies resolved. 2023-07-14T19:15:35.863 INFO:teuthology.orchestra.run.smithi098.stdout:============================================================================================================ 2023-07-14T19:15:35.863 INFO:teuthology.orchestra.run.smithi098.stdout: Package Arch Version Repository Size 2023-07-14T19:15:35.863 INFO:teuthology.orchestra.run.smithi098.stdout:============================================================================================================ 2023-07-14T19:15:35.863 INFO:teuthology.orchestra.run.smithi098.stdout:Installing group/module packages: 2023-07-14T19:15:35.863 INFO:teuthology.orchestra.run.smithi098.stdout: buildah x86_64 1:1.26.2-1.module+el8.6.0+16771+28dfca77 rhel-8-for-x86_64-appstream-rpms 8.0 M 2023-07-14T19:15:35.864 INFO:teuthology.orchestra.run.smithi098.stdout: cockpit-podman noarch 49.1-1.module+el8.6.0+16771+28dfca77 rhel-8-for-x86_64-appstream-rpms 524 k 2023-07-14T19:15:35.864 INFO:teuthology.orchestra.run.smithi098.stdout: crun x86_64 1.5-1.module+el8.6.0+16771+28dfca77 rhel-8-for-x86_64-appstream-rpms 213 k 2023-07-14T19:15:35.864 INFO:teuthology.orchestra.run.smithi098.stdout: python3-podman noarch 4.0.1-1.module+el8.6.0+16837+85b82211 rhel-8-for-x86_64-appstream-rpms 149 k 2023-07-14T19:15:35.864 INFO:teuthology.orchestra.run.smithi098.stdout: skopeo x86_64 2:1.9.1-1.module+el8.6.0+16771+28dfca77 rhel-8-for-x86_64-appstream-rpms 7.0 M 2023-07-14T19:15:35.864 INFO:teuthology.orchestra.run.smithi098.stdout: toolbox x86_64 0.0.99.3-0.6.module+el8.6.0+16771+28dfca77 rhel-8-for-x86_64-appstream-rpms 2.2 M 2023-07-14T19:15:35.864 INFO:teuthology.orchestra.run.smithi098.stdout: udica noarch 0.2.6-4.module+el8.6.0+16771+28dfca77 rhel-8-for-x86_64-appstream-rpms 49 k 2023-07-14T19:15:35.864 INFO:teuthology.orchestra.run.smithi098.stdout:Installing dependencies: 2023-07-14T19:15:35.865 INFO:teuthology.orchestra.run.smithi098.stdout: python3-pytoml noarch 0.1.14-5.git7dea353.el8 rhel-8-for-x86_64-appstream-rpms 25 k 2023-07-14T19:15:35.865 INFO:teuthology.orchestra.run.smithi098.stdout: python3-pyxdg noarch 0.25-16.el8 rhel-8-for-x86_64-appstream-rpms 94 k 2023-07-14T19:15:35.865 INFO:teuthology.orchestra.run.smithi098.stdout: yajl x86_64 2.1.0-10.el8 rhel-8-for-x86_64-appstream-rpms 41 k 2023-07-14T19:15:35.865 INFO:teuthology.orchestra.run.smithi098.stdout:Installing module profiles: 2023-07-14T19:15:35.865 INFO:teuthology.orchestra.run.smithi098.stdout: container-tools/common 2023-07-14T19:15:35.865 INFO:teuthology.orchestra.run.smithi098.stdout: 2023-07-14T19:15:35.865 INFO:teuthology.orchestra.run.smithi098.stdout:Enabling module streams: 2023-07-14T19:15:35.866 INFO:teuthology.orchestra.run.smithi098.stdout: container-tools rhel8 2023-07-14T19:15:35.866 INFO:teuthology.orchestra.run.smithi098.stdout: 2023-07-14T19:15:35.866 INFO:teuthology.orchestra.run.smithi098.stdout:Transaction Summary 2023-07-14T19:15:35.866 INFO:teuthology.orchestra.run.smithi098.stdout:============================================================================================================ 2023-07-14T19:15:35.866 INFO:teuthology.orchestra.run.smithi098.stdout:Install 10 Packages 2023-07-14T19:15:35.866 INFO:teuthology.orchestra.run.smithi098.stdout: 2023-07-14T19:15:35.867 INFO:teuthology.orchestra.run.smithi098.stdout:Total download size: 18 M 2023-07-14T19:15:35.867 INFO:teuthology.orchestra.run.smithi098.stdout:Installed size: 65 M 2023-07-14T19:15:35.867 INFO:teuthology.orchestra.run.smithi098.stdout:Downloading Packages: 2023-07-14T19:15:35.871 INFO:teuthology.orchestra.run.smithi101.stdout:Dependencies resolved. 2023-07-14T19:15:35.941 INFO:teuthology.orchestra.run.smithi098.stdout:(1/10): python3-pyxdg-0.25-16.el8.noarch.rpm 1.2 MB/s | 94 kB 00:00 2023-07-14T19:15:35.965 INFO:teuthology.orchestra.run.smithi101.stdout:============================================================================================================ 2023-07-14T19:15:35.965 INFO:teuthology.orchestra.run.smithi101.stdout: Package Arch Version Repository Size 2023-07-14T19:15:35.966 INFO:teuthology.orchestra.run.smithi101.stdout:============================================================================================================ 2023-07-14T19:15:35.966 INFO:teuthology.orchestra.run.smithi101.stdout:Installing group/module packages: 2023-07-14T19:15:35.966 INFO:teuthology.orchestra.run.smithi101.stdout: buildah x86_64 1:1.26.2-1.module+el8.6.0+16771+28dfca77 rhel-8-for-x86_64-appstream-rpms 8.0 M 2023-07-14T19:15:35.967 INFO:teuthology.orchestra.run.smithi101.stdout: cockpit-podman noarch 49.1-1.module+el8.6.0+16771+28dfca77 rhel-8-for-x86_64-appstream-rpms 524 k 2023-07-14T19:15:35.967 INFO:teuthology.orchestra.run.smithi101.stdout: crun x86_64 1.5-1.module+el8.6.0+16771+28dfca77 rhel-8-for-x86_64-appstream-rpms 213 k 2023-07-14T19:15:35.967 INFO:teuthology.orchestra.run.smithi101.stdout: python3-podman noarch 4.0.1-1.module+el8.6.0+16837+85b82211 rhel-8-for-x86_64-appstream-rpms 149 k 2023-07-14T19:15:35.967 INFO:teuthology.orchestra.run.smithi101.stdout: skopeo x86_64 2:1.9.1-1.module+el8.6.0+16771+28dfca77 rhel-8-for-x86_64-appstream-rpms 7.0 M 2023-07-14T19:15:35.968 INFO:teuthology.orchestra.run.smithi101.stdout: toolbox x86_64 0.0.99.3-0.6.module+el8.6.0+16771+28dfca77 rhel-8-for-x86_64-appstream-rpms 2.2 M 2023-07-14T19:15:35.968 INFO:teuthology.orchestra.run.smithi101.stdout: udica noarch 0.2.6-4.module+el8.6.0+16771+28dfca77 rhel-8-for-x86_64-appstream-rpms 49 k 2023-07-14T19:15:35.968 INFO:teuthology.orchestra.run.smithi101.stdout:Installing dependencies: 2023-07-14T19:15:35.969 INFO:teuthology.orchestra.run.smithi101.stdout: python3-pytoml noarch 0.1.14-5.git7dea353.el8 rhel-8-for-x86_64-appstream-rpms 25 k 2023-07-14T19:15:35.969 INFO:teuthology.orchestra.run.smithi101.stdout: python3-pyxdg noarch 0.25-16.el8 rhel-8-for-x86_64-appstream-rpms 94 k 2023-07-14T19:15:35.969 INFO:teuthology.orchestra.run.smithi101.stdout: yajl x86_64 2.1.0-10.el8 rhel-8-for-x86_64-appstream-rpms 41 k 2023-07-14T19:15:35.970 INFO:teuthology.orchestra.run.smithi101.stdout:Installing module profiles: 2023-07-14T19:15:35.970 INFO:teuthology.orchestra.run.smithi101.stdout: container-tools/common 2023-07-14T19:15:35.970 INFO:teuthology.orchestra.run.smithi101.stdout: 2023-07-14T19:15:35.971 INFO:teuthology.orchestra.run.smithi101.stdout:Enabling module streams: 2023-07-14T19:15:35.971 INFO:teuthology.orchestra.run.smithi101.stdout: container-tools rhel8 2023-07-14T19:15:35.971 INFO:teuthology.orchestra.run.smithi101.stdout: 2023-07-14T19:15:35.972 INFO:teuthology.orchestra.run.smithi101.stdout:Transaction Summary 2023-07-14T19:15:35.972 INFO:teuthology.orchestra.run.smithi101.stdout:============================================================================================================ 2023-07-14T19:15:35.972 INFO:teuthology.orchestra.run.smithi101.stdout:Install 10 Packages 2023-07-14T19:15:35.973 INFO:teuthology.orchestra.run.smithi101.stdout: 2023-07-14T19:15:35.974 INFO:teuthology.orchestra.run.smithi098.stdout:(2/10): python3-pytoml-0.1.14-5.git7dea353.el8. 245 kB/s | 25 kB 00:00 2023-07-14T19:15:35.974 INFO:teuthology.orchestra.run.smithi101.stdout:Total download size: 18 M 2023-07-14T19:15:35.975 INFO:teuthology.orchestra.run.smithi101.stdout:Installed size: 65 M 2023-07-14T19:15:35.975 INFO:teuthology.orchestra.run.smithi101.stdout:Downloading Packages: 2023-07-14T19:15:35.992 INFO:teuthology.orchestra.run.smithi098.stdout:(3/10): yajl-2.1.0-10.el8.x86_64.rpm 322 kB/s | 41 kB 00:00 2023-07-14T19:15:35.998 INFO:teuthology.orchestra.run.smithi139.stdout:Dependencies resolved. 2023-07-14T19:15:36.017 INFO:teuthology.orchestra.run.smithi098.stdout:(4/10): udica-0.2.6-4.module+el8.6.0+16771+28df 653 kB/s | 49 kB 00:00 2023-07-14T19:15:36.043 INFO:teuthology.orchestra.run.smithi101.stdout:(1/10): python3-pyxdg-0.25-16.el8.noarch.rpm 1.2 MB/s | 94 kB 00:00 2023-07-14T19:15:36.051 INFO:teuthology.orchestra.run.smithi098.stdout:(5/10): cockpit-podman-49.1-1.module+el8.6.0+16 6.1 MB/s | 524 kB 00:00 2023-07-14T19:15:36.069 INFO:teuthology.orchestra.run.smithi101.stdout:(2/10): python3-pytoml-0.1.14-5.git7dea353.el8. 246 kB/s | 25 kB 00:00 2023-07-14T19:15:36.076 INFO:teuthology.orchestra.run.smithi098.stdout:(6/10): python3-podman-4.0.1-1.module+el8.6.0+1 2.5 MB/s | 149 kB 00:00 2023-07-14T19:15:36.090 INFO:teuthology.orchestra.run.smithi139.stdout:============================================================================================================ 2023-07-14T19:15:36.091 INFO:teuthology.orchestra.run.smithi139.stdout: Package Arch Version Repository Size 2023-07-14T19:15:36.091 INFO:teuthology.orchestra.run.smithi139.stdout:============================================================================================================ 2023-07-14T19:15:36.092 INFO:teuthology.orchestra.run.smithi139.stdout:Installing group/module packages: 2023-07-14T19:15:36.092 INFO:teuthology.orchestra.run.smithi139.stdout: buildah x86_64 1:1.26.2-1.module+el8.6.0+16771+28dfca77 rhel-8-for-x86_64-appstream-rpms 8.0 M 2023-07-14T19:15:36.092 INFO:teuthology.orchestra.run.smithi139.stdout: cockpit-podman noarch 49.1-1.module+el8.6.0+16771+28dfca77 rhel-8-for-x86_64-appstream-rpms 524 k 2023-07-14T19:15:36.092 INFO:teuthology.orchestra.run.smithi139.stdout: crun x86_64 1.5-1.module+el8.6.0+16771+28dfca77 rhel-8-for-x86_64-appstream-rpms 213 k 2023-07-14T19:15:36.092 INFO:teuthology.orchestra.run.smithi139.stdout: python3-podman noarch 4.0.1-1.module+el8.6.0+16837+85b82211 rhel-8-for-x86_64-appstream-rpms 149 k 2023-07-14T19:15:36.092 INFO:teuthology.orchestra.run.smithi139.stdout: skopeo x86_64 2:1.9.1-1.module+el8.6.0+16771+28dfca77 rhel-8-for-x86_64-appstream-rpms 7.0 M 2023-07-14T19:15:36.093 INFO:teuthology.orchestra.run.smithi139.stdout: toolbox x86_64 0.0.99.3-0.6.module+el8.6.0+16771+28dfca77 rhel-8-for-x86_64-appstream-rpms 2.2 M 2023-07-14T19:15:36.093 INFO:teuthology.orchestra.run.smithi139.stdout: udica noarch 0.2.6-4.module+el8.6.0+16771+28dfca77 rhel-8-for-x86_64-appstream-rpms 49 k 2023-07-14T19:15:36.093 INFO:teuthology.orchestra.run.smithi139.stdout:Installing dependencies: 2023-07-14T19:15:36.093 INFO:teuthology.orchestra.run.smithi139.stdout: python3-pytoml noarch 0.1.14-5.git7dea353.el8 rhel-8-for-x86_64-appstream-rpms 25 k 2023-07-14T19:15:36.093 INFO:teuthology.orchestra.run.smithi139.stdout: python3-pyxdg noarch 0.25-16.el8 rhel-8-for-x86_64-appstream-rpms 94 k 2023-07-14T19:15:36.093 INFO:teuthology.orchestra.run.smithi139.stdout: yajl x86_64 2.1.0-10.el8 rhel-8-for-x86_64-appstream-rpms 41 k 2023-07-14T19:15:36.093 INFO:teuthology.orchestra.run.smithi139.stdout:Installing module profiles: 2023-07-14T19:15:36.094 INFO:teuthology.orchestra.run.smithi139.stdout: container-tools/common 2023-07-14T19:15:36.094 INFO:teuthology.orchestra.run.smithi139.stdout: 2023-07-14T19:15:36.094 INFO:teuthology.orchestra.run.smithi139.stdout:Enabling module streams: 2023-07-14T19:15:36.094 INFO:teuthology.orchestra.run.smithi139.stdout: container-tools rhel8 2023-07-14T19:15:36.094 INFO:teuthology.orchestra.run.smithi139.stdout: 2023-07-14T19:15:36.094 INFO:teuthology.orchestra.run.smithi139.stdout:Transaction Summary 2023-07-14T19:15:36.094 INFO:teuthology.orchestra.run.smithi139.stdout:============================================================================================================ 2023-07-14T19:15:36.095 INFO:teuthology.orchestra.run.smithi139.stdout:Install 10 Packages 2023-07-14T19:15:36.095 INFO:teuthology.orchestra.run.smithi139.stdout: 2023-07-14T19:15:36.095 INFO:teuthology.orchestra.run.smithi101.stdout:(3/10): yajl-2.1.0-10.el8.x86_64.rpm 323 kB/s | 41 kB 00:00 2023-07-14T19:15:36.096 INFO:teuthology.orchestra.run.smithi139.stdout:Total download size: 18 M 2023-07-14T19:15:36.096 INFO:teuthology.orchestra.run.smithi139.stdout:Installed size: 65 M 2023-07-14T19:15:36.096 INFO:teuthology.orchestra.run.smithi139.stdout:Downloading Packages: 2023-07-14T19:15:36.101 INFO:teuthology.orchestra.run.smithi098.stdout:(7/10): crun-1.5-1.module+el8.6.0+16771+28dfca7 4.2 MB/s | 213 kB 00:00 2023-07-14T19:15:36.119 INFO:teuthology.orchestra.run.smithi101.stdout:(4/10): udica-0.2.6-4.module+el8.6.0+16771+28df 653 kB/s | 49 kB 00:00 2023-07-14T19:15:36.178 INFO:teuthology.orchestra.run.smithi101.stdout:(5/10): cockpit-podman-49.1-1.module+el8.6.0+16 4.7 MB/s | 524 kB 00:00 2023-07-14T19:15:36.178 INFO:teuthology.orchestra.run.smithi139.stdout:(1/10): python3-pyxdg-0.25-16.el8.noarch.rpm 1.1 MB/s | 94 kB 00:00 2023-07-14T19:15:36.193 INFO:teuthology.orchestra.run.smithi098.stdout:(8/10): toolbox-0.0.99.3-0.6.module+el8.6.0+167 24 MB/s | 2.2 MB 00:00 2023-07-14T19:15:36.203 INFO:teuthology.orchestra.run.smithi101.stdout:(6/10): python3-podman-4.0.1-1.module+el8.6.0+1 1.7 MB/s | 149 kB 00:00 2023-07-14T19:15:36.204 INFO:teuthology.orchestra.run.smithi139.stdout:(2/10): python3-pytoml-0.1.14-5.git7dea353.el8. 224 kB/s | 25 kB 00:00 2023-07-14T19:15:36.229 INFO:teuthology.orchestra.run.smithi101.stdout:(7/10): crun-1.5-1.module+el8.6.0+16771+28dfca7 4.1 MB/s | 213 kB 00:00 2023-07-14T19:15:36.229 INFO:teuthology.orchestra.run.smithi139.stdout:(3/10): yajl-2.1.0-10.el8.x86_64.rpm 300 kB/s | 41 kB 00:00 2023-07-14T19:15:36.254 INFO:teuthology.orchestra.run.smithi139.stdout:(4/10): udica-0.2.6-4.module+el8.6.0+16771+28df 656 kB/s | 49 kB 00:00 2023-07-14T19:15:36.302 INFO:teuthology.orchestra.run.smithi098.stdout:(9/10): skopeo-1.9.1-1.module+el8.6.0+16771+28d 31 MB/s | 7.0 MB 00:00 2023-07-14T19:15:36.320 INFO:teuthology.orchestra.run.smithi101.stdout:(8/10): toolbox-0.0.99.3-0.6.module+el8.6.0+167 24 MB/s | 2.2 MB 00:00 2023-07-14T19:15:36.321 INFO:teuthology.orchestra.run.smithi139.stdout:(5/10): cockpit-podman-49.1-1.module+el8.6.0+16 4.4 MB/s | 524 kB 00:00 2023-07-14T19:15:36.347 INFO:teuthology.orchestra.run.smithi139.stdout:(6/10): python3-podman-4.0.1-1.module+el8.6.0+1 1.6 MB/s | 149 kB 00:00 2023-07-14T19:15:36.372 INFO:teuthology.orchestra.run.smithi139.stdout:(7/10): crun-1.5-1.module+el8.6.0+16771+28dfca7 4.1 MB/s | 213 kB 00:00 2023-07-14T19:15:36.394 INFO:teuthology.orchestra.run.smithi098.stdout:(10/10): buildah-1.26.2-1.module+el8.6.0+16771+ 20 MB/s | 8.0 MB 00:00 2023-07-14T19:15:36.395 INFO:teuthology.orchestra.run.smithi098.stdout:-------------------------------------------------------------------------------- 2023-07-14T19:15:36.395 INFO:teuthology.orchestra.run.smithi098.stdout:Total 34 MB/s | 18 MB 00:00 2023-07-14T19:15:36.429 INFO:teuthology.orchestra.run.smithi101.stdout:(9/10): skopeo-1.9.1-1.module+el8.6.0+16771+28d 31 MB/s | 7.0 MB 00:00 2023-07-14T19:15:36.464 INFO:teuthology.orchestra.run.smithi139.stdout:(8/10): toolbox-0.0.99.3-0.6.module+el8.6.0+167 24 MB/s | 2.2 MB 00:00 2023-07-14T19:15:36.521 INFO:teuthology.orchestra.run.smithi101.stdout:(10/10): buildah-1.26.2-1.module+el8.6.0+16771+ 19 MB/s | 8.0 MB 00:00 2023-07-14T19:15:36.521 INFO:teuthology.orchestra.run.smithi101.stdout:-------------------------------------------------------------------------------- 2023-07-14T19:15:36.522 INFO:teuthology.orchestra.run.smithi101.stdout:Total 33 MB/s | 18 MB 00:00 2023-07-14T19:15:36.581 INFO:teuthology.orchestra.run.smithi139.stdout:(9/10): skopeo-1.9.1-1.module+el8.6.0+16771+28d 30 MB/s | 7.0 MB 00:00 2023-07-14T19:15:36.594 INFO:teuthology.orchestra.run.smithi098.stdout:Running transaction check 2023-07-14T19:15:36.613 INFO:teuthology.orchestra.run.smithi098.stdout:Transaction check succeeded. 2023-07-14T19:15:36.613 INFO:teuthology.orchestra.run.smithi098.stdout:Running transaction test 2023-07-14T19:15:36.681 INFO:teuthology.orchestra.run.smithi139.stdout:(10/10): buildah-1.26.2-1.module+el8.6.0+16771+ 18 MB/s | 8.0 MB 00:00 2023-07-14T19:15:36.682 INFO:teuthology.orchestra.run.smithi139.stdout:-------------------------------------------------------------------------------- 2023-07-14T19:15:36.682 INFO:teuthology.orchestra.run.smithi139.stdout:Total 31 MB/s | 18 MB 00:00 2023-07-14T19:15:36.723 INFO:teuthology.orchestra.run.smithi101.stdout:Running transaction check 2023-07-14T19:15:36.740 INFO:teuthology.orchestra.run.smithi101.stdout:Transaction check succeeded. 2023-07-14T19:15:36.740 INFO:teuthology.orchestra.run.smithi101.stdout:Running transaction test 2023-07-14T19:15:36.794 INFO:teuthology.orchestra.run.smithi098.stdout:Transaction test succeeded. 2023-07-14T19:15:36.798 INFO:teuthology.orchestra.run.smithi098.stdout:Running transaction 2023-07-14T19:15:36.880 INFO:teuthology.orchestra.run.smithi139.stdout:Running transaction check 2023-07-14T19:15:36.899 INFO:teuthology.orchestra.run.smithi139.stdout:Transaction check succeeded. 2023-07-14T19:15:36.900 INFO:teuthology.orchestra.run.smithi139.stdout:Running transaction test 2023-07-14T19:15:36.922 INFO:teuthology.orchestra.run.smithi101.stdout:Transaction test succeeded. 2023-07-14T19:15:36.925 INFO:teuthology.orchestra.run.smithi101.stdout:Running transaction 2023-07-14T19:15:37.080 INFO:teuthology.orchestra.run.smithi139.stdout:Transaction test succeeded. 2023-07-14T19:15:37.084 INFO:teuthology.orchestra.run.smithi139.stdout:Running transaction 2023-07-14T19:15:37.319 INFO:teuthology.orchestra.run.smithi098.stdout: Preparing : 1/1 2023-07-14T19:15:37.449 INFO:teuthology.orchestra.run.smithi101.stdout: Preparing : 1/1 2023-07-14T19:15:37.506 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-pytoml-0.1.14-5.git7dea353.el8.noarch 1/10 2023-07-14T19:15:37.627 INFO:teuthology.orchestra.run.smithi139.stdout: Preparing : 1/1 2023-07-14T19:15:37.639 INFO:teuthology.orchestra.run.smithi101.stdout: Installing : python3-pytoml-0.1.14-5.git7dea353.el8.noarch 1/10 2023-07-14T19:15:37.654 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : yajl-2.1.0-10.el8.x86_64 2/10 2023-07-14T19:15:37.851 INFO:teuthology.orchestra.run.smithi101.stdout: Installing : yajl-2.1.0-10.el8.x86_64 2/10 2023-07-14T19:15:37.852 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : python3-pytoml-0.1.14-5.git7dea353.el8.noarch 1/10 2023-07-14T19:15:37.861 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-pyxdg-0.25-16.el8.noarch 3/10 2023-07-14T19:15:38.027 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : yajl-2.1.0-10.el8.x86_64 2/10 2023-07-14T19:15:38.053 INFO:teuthology.orchestra.run.smithi101.stdout: Installing : python3-pyxdg-0.25-16.el8.noarch 3/10 2023-07-14T19:15:38.060 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-podman-4.0.1-1.module+el8.6.0+16837+85b822 4/10 2023-07-14T19:15:38.229 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : python3-pyxdg-0.25-16.el8.noarch 3/10 2023-07-14T19:15:38.295 INFO:teuthology.orchestra.run.smithi101.stdout: Installing : python3-podman-4.0.1-1.module+el8.6.0+16837+85b822 4/10 2023-07-14T19:15:38.400 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : crun-1.5-1.module+el8.6.0+16771+28dfca77.x86_64 5/10 2023-07-14T19:15:38.474 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : python3-podman-4.0.1-1.module+el8.6.0+16837+85b822 4/10 2023-07-14T19:15:38.654 INFO:teuthology.orchestra.run.smithi101.stdout: Installing : crun-1.5-1.module+el8.6.0+16771+28dfca77.x86_64 5/10 2023-07-14T19:15:38.806 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : crun-1.5-1.module+el8.6.0+16771+28dfca77.x86_64 5/10 2023-07-14T19:15:39.146 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : toolbox-0.0.99.3-0.6.module+el8.6.0+16771+28dfca77 6/10 2023-07-14T19:15:39.400 INFO:teuthology.orchestra.run.smithi101.stdout: Installing : toolbox-0.0.99.3-0.6.module+el8.6.0+16771+28dfca77 6/10 2023-07-14T19:15:39.553 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : toolbox-0.0.99.3-0.6.module+el8.6.0+16771+28dfca77 6/10 2023-07-14T19:15:40.070 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : skopeo-2:1.9.1-1.module+el8.6.0+16771+28dfca77.x86 7/10 2023-07-14T19:15:40.290 INFO:teuthology.orchestra.run.smithi101.stdout: Installing : skopeo-2:1.9.1-1.module+el8.6.0+16771+28dfca77.x86 7/10 2023-07-14T19:15:40.317 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : buildah-1:1.26.2-1.module+el8.6.0+16771+28dfca77.x 8/10 2023-07-14T19:15:40.434 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : skopeo-2:1.9.1-1.module+el8.6.0+16771+28dfca77.x86 7/10 2023-07-14T19:15:40.487 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : cockpit-podman-49.1-1.module+el8.6.0+16771+28dfca7 9/10 2023-07-14T19:15:40.532 INFO:teuthology.orchestra.run.smithi101.stdout: Installing : buildah-1:1.26.2-1.module+el8.6.0+16771+28dfca77.x 8/10 2023-07-14T19:15:40.656 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : buildah-1:1.26.2-1.module+el8.6.0+16771+28dfca77.x 8/10 2023-07-14T19:15:40.663 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : udica-0.2.6-4.module+el8.6.0+16771+28dfca77.noarch 10/10 2023-07-14T19:15:40.750 INFO:teuthology.orchestra.run.smithi101.stdout: Installing : cockpit-podman-49.1-1.module+el8.6.0+16771+28dfca7 9/10 2023-07-14T19:15:40.850 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : cockpit-podman-49.1-1.module+el8.6.0+16771+28dfca7 9/10 2023-07-14T19:15:40.932 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: udica-0.2.6-4.module+el8.6.0+16771+28dfca77.noarch 10/10 2023-07-14T19:15:40.932 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-pyxdg-0.25-16.el8.noarch 1/10 2023-07-14T19:15:40.932 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : yajl-2.1.0-10.el8.x86_64 2/10 2023-07-14T19:15:40.933 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-pytoml-0.1.14-5.git7dea353.el8.noarch 3/10 2023-07-14T19:15:40.933 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : udica-0.2.6-4.module+el8.6.0+16771+28dfca77.noarch 4/10 2023-07-14T19:15:40.933 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : cockpit-podman-49.1-1.module+el8.6.0+16771+28dfca7 5/10 2023-07-14T19:15:40.933 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : buildah-1:1.26.2-1.module+el8.6.0+16771+28dfca77.x 6/10 2023-07-14T19:15:40.933 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-podman-4.0.1-1.module+el8.6.0+16837+85b822 7/10 2023-07-14T19:15:40.933 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : crun-1.5-1.module+el8.6.0+16771+28dfca77.x86_64 8/10 2023-07-14T19:15:40.933 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : skopeo-2:1.9.1-1.module+el8.6.0+16771+28dfca77.x86 9/10 2023-07-14T19:15:40.952 INFO:teuthology.orchestra.run.smithi101.stdout: Installing : udica-0.2.6-4.module+el8.6.0+16771+28dfca77.noarch 10/10 2023-07-14T19:15:41.027 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : udica-0.2.6-4.module+el8.6.0+16771+28dfca77.noarch 10/10 2023-07-14T19:15:41.261 INFO:teuthology.orchestra.run.smithi101.stdout: Running scriptlet: udica-0.2.6-4.module+el8.6.0+16771+28dfca77.noarch 10/10 2023-07-14T19:15:41.261 INFO:teuthology.orchestra.run.smithi101.stdout: Verifying : python3-pyxdg-0.25-16.el8.noarch 1/10 2023-07-14T19:15:41.261 INFO:teuthology.orchestra.run.smithi101.stdout: Verifying : yajl-2.1.0-10.el8.x86_64 2/10 2023-07-14T19:15:41.261 INFO:teuthology.orchestra.run.smithi101.stdout: Verifying : python3-pytoml-0.1.14-5.git7dea353.el8.noarch 3/10 2023-07-14T19:15:41.262 INFO:teuthology.orchestra.run.smithi101.stdout: Verifying : udica-0.2.6-4.module+el8.6.0+16771+28dfca77.noarch 4/10 2023-07-14T19:15:41.262 INFO:teuthology.orchestra.run.smithi101.stdout: Verifying : cockpit-podman-49.1-1.module+el8.6.0+16771+28dfca7 5/10 2023-07-14T19:15:41.262 INFO:teuthology.orchestra.run.smithi101.stdout: Verifying : buildah-1:1.26.2-1.module+el8.6.0+16771+28dfca77.x 6/10 2023-07-14T19:15:41.262 INFO:teuthology.orchestra.run.smithi101.stdout: Verifying : python3-podman-4.0.1-1.module+el8.6.0+16837+85b822 7/10 2023-07-14T19:15:41.262 INFO:teuthology.orchestra.run.smithi101.stdout: Verifying : crun-1.5-1.module+el8.6.0+16771+28dfca77.x86_64 8/10 2023-07-14T19:15:41.262 INFO:teuthology.orchestra.run.smithi101.stdout: Verifying : skopeo-2:1.9.1-1.module+el8.6.0+16771+28dfca77.x86 9/10 2023-07-14T19:15:41.289 INFO:teuthology.orchestra.run.smithi139.stdout: Running scriptlet: udica-0.2.6-4.module+el8.6.0+16771+28dfca77.noarch 10/10 2023-07-14T19:15:41.289 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : python3-pyxdg-0.25-16.el8.noarch 1/10 2023-07-14T19:15:41.290 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : yajl-2.1.0-10.el8.x86_64 2/10 2023-07-14T19:15:41.290 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : python3-pytoml-0.1.14-5.git7dea353.el8.noarch 3/10 2023-07-14T19:15:41.290 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : udica-0.2.6-4.module+el8.6.0+16771+28dfca77.noarch 4/10 2023-07-14T19:15:41.290 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : cockpit-podman-49.1-1.module+el8.6.0+16771+28dfca7 5/10 2023-07-14T19:15:41.290 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : buildah-1:1.26.2-1.module+el8.6.0+16771+28dfca77.x 6/10 2023-07-14T19:15:41.290 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : python3-podman-4.0.1-1.module+el8.6.0+16837+85b822 7/10 2023-07-14T19:15:41.290 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : crun-1.5-1.module+el8.6.0+16771+28dfca77.x86_64 8/10 2023-07-14T19:15:41.291 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : skopeo-2:1.9.1-1.module+el8.6.0+16771+28dfca77.x86 9/10 2023-07-14T19:15:42.056 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : toolbox-0.0.99.3-0.6.module+el8.6.0+16771+28dfca77 10/10 2023-07-14T19:15:42.056 INFO:teuthology.orchestra.run.smithi098.stdout:Last metadata expiration check: 0:05:57 ago on Fri 14 Jul 2023 07:09:44 PM UTC. 2023-07-14T19:15:42.427 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : toolbox-0.0.99.3-0.6.module+el8.6.0+16771+28dfca77 10/10 2023-07-14T19:15:42.428 INFO:teuthology.orchestra.run.smithi139.stdout:Last metadata expiration check: 0:06:29 ago on Fri 14 Jul 2023 07:09:12 PM UTC. 2023-07-14T19:15:42.451 INFO:teuthology.orchestra.run.smithi101.stdout: Verifying : toolbox-0.0.99.3-0.6.module+el8.6.0+16771+28dfca77 10/10 2023-07-14T19:15:42.452 INFO:teuthology.orchestra.run.smithi101.stdout:Last metadata expiration check: 0:01:45 ago on Fri 14 Jul 2023 07:13:56 PM UTC. 2023-07-14T19:15:46.406 INFO:teuthology.orchestra.run.smithi101.stdout:Installed products updated. 2023-07-14T19:15:46.446 INFO:teuthology.orchestra.run.smithi098.stdout:Installed products updated. 2023-07-14T19:15:46.489 INFO:teuthology.orchestra.run.smithi139.stdout:Installed products updated. 2023-07-14T19:15:46.613 INFO:teuthology.orchestra.run.smithi101.stdout: 2023-07-14T19:15:46.613 INFO:teuthology.orchestra.run.smithi101.stdout:Installed: 2023-07-14T19:15:46.613 INFO:teuthology.orchestra.run.smithi101.stdout: buildah-1:1.26.2-1.module+el8.6.0+16771+28dfca77.x86_64 2023-07-14T19:15:46.613 INFO:teuthology.orchestra.run.smithi101.stdout: cockpit-podman-49.1-1.module+el8.6.0+16771+28dfca77.noarch 2023-07-14T19:15:46.613 INFO:teuthology.orchestra.run.smithi101.stdout: crun-1.5-1.module+el8.6.0+16771+28dfca77.x86_64 2023-07-14T19:15:46.613 INFO:teuthology.orchestra.run.smithi101.stdout: python3-podman-4.0.1-1.module+el8.6.0+16837+85b82211.noarch 2023-07-14T19:15:46.614 INFO:teuthology.orchestra.run.smithi101.stdout: python3-pytoml-0.1.14-5.git7dea353.el8.noarch 2023-07-14T19:15:46.614 INFO:teuthology.orchestra.run.smithi101.stdout: python3-pyxdg-0.25-16.el8.noarch 2023-07-14T19:15:46.614 INFO:teuthology.orchestra.run.smithi101.stdout: skopeo-2:1.9.1-1.module+el8.6.0+16771+28dfca77.x86_64 2023-07-14T19:15:46.614 INFO:teuthology.orchestra.run.smithi101.stdout: toolbox-0.0.99.3-0.6.module+el8.6.0+16771+28dfca77.x86_64 2023-07-14T19:15:46.614 INFO:teuthology.orchestra.run.smithi101.stdout: udica-0.2.6-4.module+el8.6.0+16771+28dfca77.noarch 2023-07-14T19:15:46.614 INFO:teuthology.orchestra.run.smithi101.stdout: yajl-2.1.0-10.el8.x86_64 2023-07-14T19:15:46.614 INFO:teuthology.orchestra.run.smithi101.stdout: 2023-07-14T19:15:46.614 INFO:teuthology.orchestra.run.smithi101.stdout:Complete! 2023-07-14T19:15:46.651 INFO:teuthology.orchestra.run.smithi098.stdout: 2023-07-14T19:15:46.651 INFO:teuthology.orchestra.run.smithi098.stdout:Installed: 2023-07-14T19:15:46.651 INFO:teuthology.orchestra.run.smithi098.stdout: buildah-1:1.26.2-1.module+el8.6.0+16771+28dfca77.x86_64 2023-07-14T19:15:46.651 INFO:teuthology.orchestra.run.smithi098.stdout: cockpit-podman-49.1-1.module+el8.6.0+16771+28dfca77.noarch 2023-07-14T19:15:46.651 INFO:teuthology.orchestra.run.smithi098.stdout: crun-1.5-1.module+el8.6.0+16771+28dfca77.x86_64 2023-07-14T19:15:46.651 INFO:teuthology.orchestra.run.smithi098.stdout: python3-podman-4.0.1-1.module+el8.6.0+16837+85b82211.noarch 2023-07-14T19:15:46.652 INFO:teuthology.orchestra.run.smithi098.stdout: python3-pytoml-0.1.14-5.git7dea353.el8.noarch 2023-07-14T19:15:46.652 INFO:teuthology.orchestra.run.smithi098.stdout: python3-pyxdg-0.25-16.el8.noarch 2023-07-14T19:15:46.652 INFO:teuthology.orchestra.run.smithi098.stdout: skopeo-2:1.9.1-1.module+el8.6.0+16771+28dfca77.x86_64 2023-07-14T19:15:46.652 INFO:teuthology.orchestra.run.smithi098.stdout: toolbox-0.0.99.3-0.6.module+el8.6.0+16771+28dfca77.x86_64 2023-07-14T19:15:46.652 INFO:teuthology.orchestra.run.smithi098.stdout: udica-0.2.6-4.module+el8.6.0+16771+28dfca77.noarch 2023-07-14T19:15:46.652 INFO:teuthology.orchestra.run.smithi098.stdout: yajl-2.1.0-10.el8.x86_64 2023-07-14T19:15:46.652 INFO:teuthology.orchestra.run.smithi098.stdout: 2023-07-14T19:15:46.653 INFO:teuthology.orchestra.run.smithi098.stdout:Complete! 2023-07-14T19:15:46.750 INFO:teuthology.orchestra.run.smithi139.stdout: 2023-07-14T19:15:46.751 INFO:teuthology.orchestra.run.smithi139.stdout:Installed: 2023-07-14T19:15:46.751 INFO:teuthology.orchestra.run.smithi139.stdout: buildah-1:1.26.2-1.module+el8.6.0+16771+28dfca77.x86_64 2023-07-14T19:15:46.751 INFO:teuthology.orchestra.run.smithi139.stdout: cockpit-podman-49.1-1.module+el8.6.0+16771+28dfca77.noarch 2023-07-14T19:15:46.751 INFO:teuthology.orchestra.run.smithi139.stdout: crun-1.5-1.module+el8.6.0+16771+28dfca77.x86_64 2023-07-14T19:15:46.751 INFO:teuthology.orchestra.run.smithi139.stdout: python3-podman-4.0.1-1.module+el8.6.0+16837+85b82211.noarch 2023-07-14T19:15:46.752 INFO:teuthology.orchestra.run.smithi139.stdout: python3-pytoml-0.1.14-5.git7dea353.el8.noarch 2023-07-14T19:15:46.752 INFO:teuthology.orchestra.run.smithi139.stdout: python3-pyxdg-0.25-16.el8.noarch 2023-07-14T19:15:46.752 INFO:teuthology.orchestra.run.smithi139.stdout: skopeo-2:1.9.1-1.module+el8.6.0+16771+28dfca77.x86_64 2023-07-14T19:15:46.752 INFO:teuthology.orchestra.run.smithi139.stdout: toolbox-0.0.99.3-0.6.module+el8.6.0+16771+28dfca77.x86_64 2023-07-14T19:15:46.752 INFO:teuthology.orchestra.run.smithi139.stdout: udica-0.2.6-4.module+el8.6.0+16771+28dfca77.noarch 2023-07-14T19:15:46.752 INFO:teuthology.orchestra.run.smithi139.stdout: yajl-2.1.0-10.el8.x86_64 2023-07-14T19:15:46.752 INFO:teuthology.orchestra.run.smithi139.stdout: 2023-07-14T19:15:46.753 INFO:teuthology.orchestra.run.smithi139.stdout:Complete! 2023-07-14T19:15:46.783 DEBUG:teuthology.parallel:result is None 2023-07-14T19:15:46.834 DEBUG:teuthology.parallel:result is None 2023-07-14T19:15:46.921 DEBUG:teuthology.parallel:result is None 2023-07-14T19:15:46.921 INFO:teuthology.run_tasks:Running task install... 2023-07-14T19:15:46.929 DEBUG:teuthology.task.install:project ceph 2023-07-14T19:15:46.930 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': '81c9387d2e7576da065747ec9c16d1c7d69016ec'}} 2023-07-14T19:15:46.930 DEBUG:teuthology.task.install:config {'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'flavor': 'default', 'sha1': '81c9387d2e7576da065747ec9c16d1c7d69016ec'} 2023-07-14T19:15:46.931 INFO:teuthology.task.install:Using flavor: default 2023-07-14T19:15:46.936 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2023-07-14T19:15:46.937 INFO:teuthology.task.install:extra packages: {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-top', 'cephfs-mirror']} 2023-07-14T19:15:46.937 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '81c9387d2e7576da065747ec9c16d1c7d69016ec', 'tag': None, 'wait_for_package': False} 2023-07-14T19:15:46.938 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=81c9387d2e7576da065747ec9c16d1c7d69016ec 2023-07-14T19:15:46.940 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '81c9387d2e7576da065747ec9c16d1c7d69016ec', 'tag': None, 'wait_for_package': False} 2023-07-14T19:15:46.940 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=81c9387d2e7576da065747ec9c16d1c7d69016ec 2023-07-14T19:15:46.941 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '81c9387d2e7576da065747ec9c16d1c7d69016ec', 'tag': None, 'wait_for_package': False} 2023-07-14T19:15:46.942 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=81c9387d2e7576da065747ec9c16d1c7d69016ec 2023-07-14T19:15:57.302 INFO:teuthology.task.install.rpm:Pulling from https://4.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20230714.134335/81c9387d2e7576da065747ec9c16d1c7d69016ec/centos/8/flavors/default/ 2023-07-14T19:15:57.303 INFO:teuthology.task.install.rpm:Package version is 18.0.0-4940.g81c9387d 2023-07-14T19:16:02.314 INFO:teuthology.task.install.rpm:Pulling from https://4.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20230714.134335/81c9387d2e7576da065747ec9c16d1c7d69016ec/centos/8/flavors/default/ 2023-07-14T19:16:02.315 INFO:teuthology.task.install.rpm:Package version is 18.0.0-4940.g81c9387d 2023-07-14T19:16:02.316 INFO:teuthology.task.install.rpm:Pulling from https://4.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20230714.134335/81c9387d2e7576da065747ec9c16d1c7d69016ec/centos/8/flavors/default/ 2023-07-14T19:16:02.316 INFO:teuthology.task.install.rpm:Package version is 18.0.0-4940.g81c9387d 2023-07-14T19:16:12.676 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://4.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20230714.134335/81c9387d2e7576da065747ec9c16d1c7d69016ec/centos/8/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://4.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20230714.134335/81c9387d2e7576da065747ec9c16d1c7d69016ec/centos/8/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://4.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20230714.134335/81c9387d2e7576da065747ec9c16d1c7d69016ec/centos/8/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2023-07-14T19:16:12.677 DEBUG:teuthology.orchestra.run.smithi098:> set -ex 2023-07-14T19:16:12.677 DEBUG:teuthology.orchestra.run.smithi098:> sudo dd of=/etc/yum.repos.d/ceph.repo 2023-07-14T19:16:12.679 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://4.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20230714.134335/81c9387d2e7576da065747ec9c16d1c7d69016ec/centos/8/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://4.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20230714.134335/81c9387d2e7576da065747ec9c16d1c7d69016ec/centos/8/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://4.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20230714.134335/81c9387d2e7576da065747ec9c16d1c7d69016ec/centos/8/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2023-07-14T19:16:12.679 DEBUG:teuthology.orchestra.run.smithi101:> set -ex 2023-07-14T19:16:12.679 DEBUG:teuthology.orchestra.run.smithi101:> sudo dd of=/etc/yum.repos.d/ceph.repo 2023-07-14T19:16:12.713 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, python3-cephfs, cephfs-top, cephfs-mirror, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib on remote rpm x86_64 2023-07-14T19:16:12.714 DEBUG:teuthology.orchestra.run.smithi098:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/81c9387d2e7576da065747ec9c16d1c7d69016ec/;g' /etc/yum.repos.d/ceph.repo ; fi 2023-07-14T19:16:12.715 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, python3-cephfs, cephfs-top, cephfs-mirror, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib on remote rpm x86_64 2023-07-14T19:16:12.715 DEBUG:teuthology.orchestra.run.smithi101:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/81c9387d2e7576da065747ec9c16d1c7d69016ec/;g' /etc/yum.repos.d/ceph.repo ; fi 2023-07-14T19:16:12.786 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://4.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20230714.134335/81c9387d2e7576da065747ec9c16d1c7d69016ec/centos/8/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://4.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20230714.134335/81c9387d2e7576da065747ec9c16d1c7d69016ec/centos/8/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://4.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20230714.134335/81c9387d2e7576da065747ec9c16d1c7d69016ec/centos/8/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2023-07-14T19:16:12.787 DEBUG:teuthology.orchestra.run.smithi139:> set -ex 2023-07-14T19:16:12.787 DEBUG:teuthology.orchestra.run.smithi139:> sudo dd of=/etc/yum.repos.d/ceph.repo 2023-07-14T19:16:12.788 DEBUG:teuthology.orchestra.run.smithi101:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2023-07-14T19:16:12.789 DEBUG:teuthology.orchestra.run.smithi098:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2023-07-14T19:16:12.822 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, python3-cephfs, cephfs-top, cephfs-mirror, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib on remote rpm x86_64 2023-07-14T19:16:12.822 DEBUG:teuthology.orchestra.run.smithi139:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/81c9387d2e7576da065747ec9c16d1c7d69016ec/;g' /etc/yum.repos.d/ceph.repo ; fi 2023-07-14T19:16:12.869 DEBUG:teuthology.orchestra.run.smithi098:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2023-07-14T19:16:12.870 DEBUG:teuthology.orchestra.run.smithi101:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2023-07-14T19:16:12.894 DEBUG:teuthology.orchestra.run.smithi139:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2023-07-14T19:16:12.937 INFO:teuthology.orchestra.run.smithi098.stdout:check_obsoletes = 1 2023-07-14T19:16:12.938 DEBUG:teuthology.orchestra.run.smithi098:> sudo yum clean all 2023-07-14T19:16:12.939 INFO:teuthology.orchestra.run.smithi101.stdout:check_obsoletes = 1 2023-07-14T19:16:12.940 DEBUG:teuthology.orchestra.run.smithi101:> sudo yum clean all 2023-07-14T19:16:12.976 DEBUG:teuthology.orchestra.run.smithi139:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2023-07-14T19:16:13.044 INFO:teuthology.orchestra.run.smithi139.stdout:check_obsoletes = 1 2023-07-14T19:16:13.045 DEBUG:teuthology.orchestra.run.smithi139:> sudo yum clean all 2023-07-14T19:16:13.311 INFO:teuthology.orchestra.run.smithi098.stdout:Updating Subscription Management repositories. 2023-07-14T19:16:13.312 INFO:teuthology.orchestra.run.smithi101.stdout:Updating Subscription Management repositories. 2023-07-14T19:16:13.422 INFO:teuthology.orchestra.run.smithi139.stdout:Updating Subscription Management repositories. 2023-07-14T19:16:13.544 INFO:teuthology.orchestra.run.smithi101.stdout:44 files removed 2023-07-14T19:16:13.553 INFO:teuthology.orchestra.run.smithi098.stdout:44 files removed 2023-07-14T19:16:13.590 DEBUG:teuthology.orchestra.run.smithi101:> sudo yum -y install ceph-radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-fuse rbd-mirror rbd-nbd python3-cephfs cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib 2023-07-14T19:16:13.598 DEBUG:teuthology.orchestra.run.smithi098:> sudo yum -y install ceph-radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-fuse rbd-mirror rbd-nbd python3-cephfs cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib 2023-07-14T19:16:13.645 INFO:teuthology.orchestra.run.smithi139.stdout:44 files removed 2023-07-14T19:16:13.691 DEBUG:teuthology.orchestra.run.smithi139:> sudo yum -y install ceph-radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-fuse rbd-mirror rbd-nbd python3-cephfs cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib 2023-07-14T19:16:13.920 INFO:teuthology.orchestra.run.smithi101.stdout:Updating Subscription Management repositories. 2023-07-14T19:16:13.928 INFO:teuthology.orchestra.run.smithi098.stdout:Updating Subscription Management repositories. 2023-07-14T19:16:14.024 INFO:teuthology.orchestra.run.smithi139.stdout:Updating Subscription Management repositories. 2023-07-14T19:16:15.427 INFO:teuthology.orchestra.run.smithi101.stdout:ceph packages for x86_64 70 kB/s | 81 kB 00:01 2023-07-14T19:16:15.440 INFO:teuthology.orchestra.run.smithi098.stdout:ceph packages for x86_64 70 kB/s | 81 kB 00:01 2023-07-14T19:16:15.519 INFO:teuthology.orchestra.run.smithi139.stdout:ceph packages for x86_64 71 kB/s | 81 kB 00:01 2023-07-14T19:16:16.174 INFO:teuthology.orchestra.run.smithi098.stdout:ceph noarch packages 13 kB/s | 9.4 kB 00:00 2023-07-14T19:16:16.178 INFO:teuthology.orchestra.run.smithi101.stdout:ceph noarch packages 13 kB/s | 9.4 kB 00:00 2023-07-14T19:16:16.244 INFO:teuthology.orchestra.run.smithi139.stdout:ceph noarch packages 14 kB/s | 9.4 kB 00:00 2023-07-14T19:16:17.004 INFO:teuthology.orchestra.run.smithi101.stdout:ceph source packages 1.9 kB/s | 1.5 kB 00:00 2023-07-14T19:16:17.008 INFO:teuthology.orchestra.run.smithi098.stdout:ceph source packages 1.8 kB/s | 1.5 kB 00:00 2023-07-14T19:16:17.094 INFO:teuthology.orchestra.run.smithi139.stdout:ceph source packages 1.8 kB/s | 1.5 kB 00:00 2023-07-14T19:16:17.712 INFO:teuthology.orchestra.run.smithi139.stdout:Copr repo for python3-asyncssh owned by ceph 5.8 kB/s | 3.5 kB 00:00 2023-07-14T19:16:17.730 INFO:teuthology.orchestra.run.smithi101.stdout:Copr repo for python3-asyncssh owned by ceph 4.9 kB/s | 3.5 kB 00:00 2023-07-14T19:16:17.758 INFO:teuthology.orchestra.run.smithi098.stdout:Copr repo for python3-asyncssh owned by ceph 4.7 kB/s | 3.5 kB 00:00 2023-07-14T19:16:20.139 INFO:teuthology.orchestra.run.smithi101.stdout:Extra Packages for Enterprise Linux 6.5 MB/s | 16 MB 00:02 2023-07-14T19:16:23.672 INFO:teuthology.orchestra.run.smithi101.stdout:lab-extras 277 kB/s | 24 kB 00:00 2023-07-14T19:16:24.591 INFO:teuthology.orchestra.run.smithi101.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 63 MB/s | 53 MB 00:00 2023-07-14T19:16:32.842 INFO:teuthology.orchestra.run.smithi101.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 59 MB/s | 47 MB 00:00 2023-07-14T19:16:44.008 INFO:teuthology.orchestra.run.smithi101.stdout:Package librados2-1:12.2.7-9.el8.x86_64 is already installed. 2023-07-14T19:16:44.009 INFO:teuthology.orchestra.run.smithi101.stdout:Package librbd1-1:12.2.7-9.el8.x86_64 is already installed. 2023-07-14T19:16:44.023 INFO:teuthology.orchestra.run.smithi101.stdout:Package NetworkManager-1:1.36.0-4.el8.x86_64 is already installed. 2023-07-14T19:16:44.024 INFO:teuthology.orchestra.run.smithi101.stdout:Package iproute-5.15.0-4.el8.x86_64 is already installed. 2023-07-14T19:16:44.025 INFO:teuthology.orchestra.run.smithi101.stdout:Package util-linux-2.32.1-35.el8.x86_64 is already installed. 2023-07-14T19:16:44.027 INFO:teuthology.orchestra.run.smithi101.stdout:Package libacl-devel-2.2.53-1.el8.x86_64 is already installed. 2023-07-14T19:16:44.030 INFO:teuthology.orchestra.run.smithi101.stdout:Package libattr-devel-2.4.48-3.el8.x86_64 is already installed. 2023-07-14T19:16:44.033 INFO:teuthology.orchestra.run.smithi101.stdout:Package xfsdump-3.1.8-4.el8.x86_64 is already installed. 2023-07-14T19:16:44.034 INFO:teuthology.orchestra.run.smithi101.stdout:Package xfsprogs-5.0.0-10.el8.x86_64 is already installed. 2023-07-14T19:16:44.165 INFO:teuthology.orchestra.run.smithi101.stdout:Dependencies resolved. 2023-07-14T19:16:44.178 INFO:teuthology.orchestra.run.smithi101.stdout:=========================================================================================================================== 2023-07-14T19:16:44.178 INFO:teuthology.orchestra.run.smithi101.stdout: Package Arch Version Repository Size 2023-07-14T19:16:44.178 INFO:teuthology.orchestra.run.smithi101.stdout:=========================================================================================================================== 2023-07-14T19:16:44.179 INFO:teuthology.orchestra.run.smithi101.stdout:Installing: 2023-07-14T19:16:44.179 INFO:teuthology.orchestra.run.smithi101.stdout: NetworkManager-initscripts-updown noarch 1:1.36.0-9.el8_6 rhel-8-for-x86_64-baseos-rpms 138 k 2023-07-14T19:16:44.179 INFO:teuthology.orchestra.run.smithi101.stdout: bison x86_64 3.0.4-10.el8 rhel-8-for-x86_64-appstream-rpms 688 k 2023-07-14T19:16:44.180 INFO:teuthology.orchestra.run.smithi101.stdout: ceph x86_64 2:18.0.0-4940.g81c9387d.el8 ceph 6.4 k 2023-07-14T19:16:44.180 INFO:teuthology.orchestra.run.smithi101.stdout: ceph-base x86_64 2:18.0.0-4940.g81c9387d.el8 ceph 5.4 M 2023-07-14T19:16:44.180 INFO:teuthology.orchestra.run.smithi101.stdout: ceph-fuse x86_64 2:18.0.0-4940.g81c9387d.el8 ceph 850 k 2023-07-14T19:16:44.180 INFO:teuthology.orchestra.run.smithi101.stdout: ceph-immutable-object-cache x86_64 2:18.0.0-4940.g81c9387d.el8 ceph 167 k 2023-07-14T19:16:44.181 INFO:teuthology.orchestra.run.smithi101.stdout: ceph-mgr x86_64 2:18.0.0-4940.g81c9387d.el8 ceph 1.6 M 2023-07-14T19:16:44.181 INFO:teuthology.orchestra.run.smithi101.stdout: ceph-mgr-cephadm noarch 2:18.0.0-4940.g81c9387d.el8 ceph-noarch 129 k 2023-07-14T19:16:44.181 INFO:teuthology.orchestra.run.smithi101.stdout: ceph-mgr-dashboard noarch 2:18.0.0-4940.g81c9387d.el8 ceph-noarch 1.7 M 2023-07-14T19:16:44.182 INFO:teuthology.orchestra.run.smithi101.stdout: ceph-mgr-diskprediction-local noarch 2:18.0.0-4940.g81c9387d.el8 ceph-noarch 7.4 M 2023-07-14T19:16:44.182 INFO:teuthology.orchestra.run.smithi101.stdout: ceph-mgr-rook noarch 2:18.0.0-4940.g81c9387d.el8 ceph-noarch 47 k 2023-07-14T19:16:44.182 INFO:teuthology.orchestra.run.smithi101.stdout: ceph-radosgw x86_64 2:18.0.0-4940.g81c9387d.el8 ceph 14 M 2023-07-14T19:16:44.182 INFO:teuthology.orchestra.run.smithi101.stdout: ceph-test x86_64 2:18.0.0-4940.g81c9387d.el8 ceph 48 M 2023-07-14T19:16:44.183 INFO:teuthology.orchestra.run.smithi101.stdout: ceph-volume noarch 2:18.0.0-4940.g81c9387d.el8 ceph-noarch 258 k 2023-07-14T19:16:44.183 INFO:teuthology.orchestra.run.smithi101.stdout: cephadm noarch 2:18.0.0-4940.g81c9387d.el8 ceph-noarch 165 k 2023-07-14T19:16:44.183 INFO:teuthology.orchestra.run.smithi101.stdout: cephfs-mirror x86_64 2:18.0.0-4940.g81c9387d.el8 ceph 260 k 2023-07-14T19:16:44.184 INFO:teuthology.orchestra.run.smithi101.stdout: cephfs-top noarch 2:18.0.0-4940.g81c9387d.el8 ceph-noarch 18 k 2023-07-14T19:16:44.184 INFO:teuthology.orchestra.run.smithi101.stdout: elfutils-libelf-devel x86_64 0.186-1.el8 rhel-8-for-x86_64-baseos-rpms 60 k 2023-07-14T19:16:44.184 INFO:teuthology.orchestra.run.smithi101.stdout: flex x86_64 2.6.1-9.el8 rhel-8-for-x86_64-appstream-rpms 320 k 2023-07-14T19:16:44.184 INFO:teuthology.orchestra.run.smithi101.stdout: libaio-devel x86_64 0.3.112-1.el8 rhel-8-for-x86_64-baseos-rpms 19 k 2023-07-14T19:16:44.185 INFO:teuthology.orchestra.run.smithi101.stdout: libcephfs-devel x86_64 2:18.0.0-4940.g81c9387d.el8 ceph 31 k 2023-07-14T19:16:44.185 INFO:teuthology.orchestra.run.smithi101.stdout: libcephfs2 x86_64 2:18.0.0-4940.g81c9387d.el8 ceph 772 k 2023-07-14T19:16:44.185 INFO:teuthology.orchestra.run.smithi101.stdout: librados-devel x86_64 2:18.0.0-4940.g81c9387d.el8 ceph 139 k 2023-07-14T19:16:44.185 INFO:teuthology.orchestra.run.smithi101.stdout: libtool x86_64 2.4.6-25.el8 rhel-8-for-x86_64-appstream-rpms 709 k 2023-07-14T19:16:44.186 INFO:teuthology.orchestra.run.smithi101.stdout: libuuid-devel x86_64 2.32.1-35.el8 rhel-8-for-x86_64-baseos-rpms 98 k 2023-07-14T19:16:44.186 INFO:teuthology.orchestra.run.smithi101.stdout: openssl-devel x86_64 1:1.1.1k-7.el8_6 rhel-8-for-x86_64-baseos-rpms 2.3 M 2023-07-14T19:16:44.186 INFO:teuthology.orchestra.run.smithi101.stdout: postgresql x86_64 10.21-2.module+el8.6.0+15342+53518fac rhel-8-for-x86_64-appstream-rpms 1.5 M 2023-07-14T19:16:44.187 INFO:teuthology.orchestra.run.smithi101.stdout: postgresql-contrib x86_64 10.21-2.module+el8.6.0+15342+53518fac rhel-8-for-x86_64-appstream-rpms 808 k 2023-07-14T19:16:44.187 INFO:teuthology.orchestra.run.smithi101.stdout: postgresql-server x86_64 10.21-2.module+el8.6.0+15342+53518fac rhel-8-for-x86_64-appstream-rpms 5.1 M 2023-07-14T19:16:44.187 INFO:teuthology.orchestra.run.smithi101.stdout: python3-cephfs x86_64 2:18.0.0-4940.g81c9387d.el8 ceph 197 k 2023-07-14T19:16:44.187 INFO:teuthology.orchestra.run.smithi101.stdout: python3-rados x86_64 2:18.0.0-4940.g81c9387d.el8 ceph 371 k 2023-07-14T19:16:44.188 INFO:teuthology.orchestra.run.smithi101.stdout: python3-rbd x86_64 2:18.0.0-4940.g81c9387d.el8 ceph 364 k 2023-07-14T19:16:44.188 INFO:teuthology.orchestra.run.smithi101.stdout: python3-rgw x86_64 2:18.0.0-4940.g81c9387d.el8 ceph 107 k 2023-07-14T19:16:44.188 INFO:teuthology.orchestra.run.smithi101.stdout: rbd-fuse x86_64 2:18.0.0-4940.g81c9387d.el8 ceph 94 k 2023-07-14T19:16:44.188 INFO:teuthology.orchestra.run.smithi101.stdout: rbd-mirror x86_64 2:18.0.0-4940.g81c9387d.el8 ceph 3.5 M 2023-07-14T19:16:44.189 INFO:teuthology.orchestra.run.smithi101.stdout: rbd-nbd x86_64 2:18.0.0-4940.g81c9387d.el8 ceph 179 k 2023-07-14T19:16:44.189 INFO:teuthology.orchestra.run.smithi101.stdout: xfsprogs-devel x86_64 5.0.0-10.el8 rhel-8-for-x86_64-baseos-rpms 72 k 2023-07-14T19:16:44.189 INFO:teuthology.orchestra.run.smithi101.stdout:Upgrading: 2023-07-14T19:16:44.189 INFO:teuthology.orchestra.run.smithi101.stdout: iproute x86_64 5.15.0-4.el8_6.1 rhel-8-for-x86_64-baseos-rpms 798 k 2023-07-14T19:16:44.190 INFO:teuthology.orchestra.run.smithi101.stdout: librados2 x86_64 2:18.0.0-4940.g81c9387d.el8 ceph 3.7 M 2023-07-14T19:16:44.190 INFO:teuthology.orchestra.run.smithi101.stdout: librbd1 x86_64 2:18.0.0-4940.g81c9387d.el8 ceph 3.6 M 2023-07-14T19:16:44.190 INFO:teuthology.orchestra.run.smithi101.stdout:Installing dependencies: 2023-07-14T19:16:44.191 INFO:teuthology.orchestra.run.smithi101.stdout: automake noarch 1.16.1-7.el8 rhel-8-for-x86_64-appstream-rpms 713 k 2023-07-14T19:16:44.191 INFO:teuthology.orchestra.run.smithi101.stdout: ceph-common x86_64 2:18.0.0-4940.g81c9387d.el8 ceph 22 M 2023-07-14T19:16:44.191 INFO:teuthology.orchestra.run.smithi101.stdout: ceph-grafana-dashboards noarch 2:18.0.0-4940.g81c9387d.el8 ceph-noarch 23 k 2023-07-14T19:16:44.191 INFO:teuthology.orchestra.run.smithi101.stdout: ceph-mds x86_64 2:18.0.0-4940.g81c9387d.el8 ceph 2.3 M 2023-07-14T19:16:44.192 INFO:teuthology.orchestra.run.smithi101.stdout: ceph-mgr-modules-core noarch 2:18.0.0-4940.g81c9387d.el8 ceph-noarch 241 k 2023-07-14T19:16:44.192 INFO:teuthology.orchestra.run.smithi101.stdout: ceph-mon x86_64 2:18.0.0-4940.g81c9387d.el8 ceph 4.6 M 2023-07-14T19:16:44.192 INFO:teuthology.orchestra.run.smithi101.stdout: ceph-osd x86_64 2:18.0.0-4940.g81c9387d.el8 ceph 18 M 2023-07-14T19:16:44.192 INFO:teuthology.orchestra.run.smithi101.stdout: ceph-prometheus-alerts noarch 2:18.0.0-4940.g81c9387d.el8 ceph-noarch 15 k 2023-07-14T19:16:44.192 INFO:teuthology.orchestra.run.smithi101.stdout: ceph-selinux x86_64 2:18.0.0-4940.g81c9387d.el8 ceph 24 k 2023-07-14T19:16:44.193 INFO:teuthology.orchestra.run.smithi101.stdout: fuse x86_64 2.9.7-15.el8 rhel-8-for-x86_64-baseos-rpms 83 k 2023-07-14T19:16:44.193 INFO:teuthology.orchestra.run.smithi101.stdout: gperftools-libs x86_64 1:2.7-9.el8 epel 306 k 2023-07-14T19:16:44.193 INFO:teuthology.orchestra.run.smithi101.stdout: jq x86_64 1.6-3.el8 rhel-8-for-x86_64-appstream-rpms 202 k 2023-07-14T19:16:44.194 INFO:teuthology.orchestra.run.smithi101.stdout: keyutils-libs-devel x86_64 1.5.10-9.el8 rhel-8-for-x86_64-baseos-rpms 48 k 2023-07-14T19:16:44.194 INFO:teuthology.orchestra.run.smithi101.stdout: krb5-devel x86_64 1.18.2-14.el8 rhel-8-for-x86_64-baseos-rpms 560 k 2023-07-14T19:16:44.194 INFO:teuthology.orchestra.run.smithi101.stdout: libcephsqlite x86_64 2:18.0.0-4940.g81c9387d.el8 ceph 135 k 2023-07-14T19:16:44.194 INFO:teuthology.orchestra.run.smithi101.stdout: libcom_err-devel x86_64 1.45.6-4.el8 rhel-8-for-x86_64-baseos-rpms 39 k 2023-07-14T19:16:44.195 INFO:teuthology.orchestra.run.smithi101.stdout: libgfortran x86_64 8.5.0-10.1.el8_6 rhel-8-for-x86_64-baseos-rpms 644 k 2023-07-14T19:16:44.195 INFO:teuthology.orchestra.run.smithi101.stdout: liboath x86_64 2.6.2-3.el8 epel 59 k 2023-07-14T19:16:44.195 INFO:teuthology.orchestra.run.smithi101.stdout: libpq x86_64 13.5-1.el8 rhel-8-for-x86_64-appstream-rpms 198 k 2023-07-14T19:16:44.195 INFO:teuthology.orchestra.run.smithi101.stdout: libquadmath x86_64 8.5.0-10.1.el8_6 rhel-8-for-x86_64-baseos-rpms 171 k 2023-07-14T19:16:44.195 INFO:teuthology.orchestra.run.smithi101.stdout: librabbitmq x86_64 0.9.0-3.el8 rhel-8-for-x86_64-baseos-rpms 47 k 2023-07-14T19:16:44.195 INFO:teuthology.orchestra.run.smithi101.stdout: libradosstriper1 x86_64 2:18.0.0-4940.g81c9387d.el8 ceph 473 k 2023-07-14T19:16:44.196 INFO:teuthology.orchestra.run.smithi101.stdout: librdkafka x86_64 0.11.4-3.el8 rhel-8-for-x86_64-appstream-rpms 354 k 2023-07-14T19:16:44.196 INFO:teuthology.orchestra.run.smithi101.stdout: librgw2 x86_64 2:18.0.0-4940.g81c9387d.el8 ceph 6.8 M 2023-07-14T19:16:44.196 INFO:teuthology.orchestra.run.smithi101.stdout: libselinux-devel x86_64 2.9-5.el8 rhel-8-for-x86_64-baseos-rpms 200 k 2023-07-14T19:16:44.196 INFO:teuthology.orchestra.run.smithi101.stdout: libsepol-devel x86_64 2.9-3.el8 rhel-8-for-x86_64-baseos-rpms 87 k 2023-07-14T19:16:44.196 INFO:teuthology.orchestra.run.smithi101.stdout: libunwind x86_64 1.3.1-3.el8 epel 75 k 2023-07-14T19:16:44.196 INFO:teuthology.orchestra.run.smithi101.stdout: libverto-devel x86_64 0.3.0-5.el8 rhel-8-for-x86_64-baseos-rpms 18 k 2023-07-14T19:16:44.196 INFO:teuthology.orchestra.run.smithi101.stdout: lttng-ust x86_64 2.8.1-11.el8 rhel-8-for-x86_64-appstream-rpms 259 k 2023-07-14T19:16:44.197 INFO:teuthology.orchestra.run.smithi101.stdout: oniguruma x86_64 6.8.2-2.el8 rhel-8-for-x86_64-appstream-rpms 187 k 2023-07-14T19:16:44.197 INFO:teuthology.orchestra.run.smithi101.stdout: openblas x86_64 0.3.15-3.el8 rhel-8-for-x86_64-appstream-rpms 4.7 M 2023-07-14T19:16:44.197 INFO:teuthology.orchestra.run.smithi101.stdout: openblas-threads x86_64 0.3.15-3.el8 rhel-8-for-x86_64-appstream-rpms 4.9 M 2023-07-14T19:16:44.197 INFO:teuthology.orchestra.run.smithi101.stdout: pcre2-devel x86_64 10.32-2.el8 rhel-8-for-x86_64-baseos-rpms 605 k 2023-07-14T19:16:44.197 INFO:teuthology.orchestra.run.smithi101.stdout: pcre2-utf16 x86_64 10.32-2.el8 rhel-8-for-x86_64-baseos-rpms 229 k 2023-07-14T19:16:44.197 INFO:teuthology.orchestra.run.smithi101.stdout: pcre2-utf32 x86_64 10.32-2.el8 rhel-8-for-x86_64-baseos-rpms 220 k 2023-07-14T19:16:44.197 INFO:teuthology.orchestra.run.smithi101.stdout: python3-asyncssh noarch 2.7.0-2.el8 epel 476 k 2023-07-14T19:16:44.198 INFO:teuthology.orchestra.run.smithi101.stdout: python3-babel noarch 2.5.1-7.el8 rhel-8-for-x86_64-appstream-rpms 4.8 M 2023-07-14T19:16:44.198 INFO:teuthology.orchestra.run.smithi101.stdout: python3-bcrypt x86_64 3.1.6-2.el8.1 epel 44 k 2023-07-14T19:16:44.198 INFO:teuthology.orchestra.run.smithi101.stdout: python3-beautifulsoup4 noarch 4.6.3-2.el8.1 epel 185 k 2023-07-14T19:16:44.198 INFO:teuthology.orchestra.run.smithi101.stdout: python3-cachetools noarch 3.1.1-4.el8 epel 33 k 2023-07-14T19:16:44.198 INFO:teuthology.orchestra.run.smithi101.stdout: python3-ceph-argparse x86_64 2:18.0.0-4940.g81c9387d.el8 ceph 45 k 2023-07-14T19:16:44.198 INFO:teuthology.orchestra.run.smithi101.stdout: python3-ceph-common x86_64 2:18.0.0-4940.g81c9387d.el8 ceph 122 k 2023-07-14T19:16:44.198 INFO:teuthology.orchestra.run.smithi101.stdout: python3-certifi noarch 2018.10.15-7.el8 epel 16 k 2023-07-14T19:16:44.198 INFO:teuthology.orchestra.run.smithi101.stdout: python3-cffi x86_64 1.11.5-5.el8 rhel-8-for-x86_64-baseos-rpms 238 k 2023-07-14T19:16:44.199 INFO:teuthology.orchestra.run.smithi101.stdout: python3-cheroot noarch 8.5.2-1.el8 epel 173 k 2023-07-14T19:16:44.199 INFO:teuthology.orchestra.run.smithi101.stdout: python3-cherrypy noarch 18.4.0-1.el8 epel 384 k 2023-07-14T19:16:44.199 INFO:teuthology.orchestra.run.smithi101.stdout: python3-cryptography x86_64 3.2.1-5.el8 rhel-8-for-x86_64-baseos-rpms 559 k 2023-07-14T19:16:44.199 INFO:teuthology.orchestra.run.smithi101.stdout: python3-google-auth noarch 1:1.1.1-10.el8 epel 81 k 2023-07-14T19:16:44.199 INFO:teuthology.orchestra.run.smithi101.stdout: python3-jaraco noarch 6.2-6.el8 epel 11 k 2023-07-14T19:16:44.199 INFO:teuthology.orchestra.run.smithi101.stdout: python3-jaraco-functools noarch 2.0-4.el8 epel 18 k 2023-07-14T19:16:44.199 INFO:teuthology.orchestra.run.smithi101.stdout: python3-jinja2 noarch 2.10.1-3.el8 rhel-8-for-x86_64-appstream-rpms 538 k 2023-07-14T19:16:44.200 INFO:teuthology.orchestra.run.smithi101.stdout: python3-jsonpatch noarch 1.21-2.el8 rhel-8-for-x86_64-appstream-rpms 27 k 2023-07-14T19:16:44.200 INFO:teuthology.orchestra.run.smithi101.stdout: python3-jsonpointer noarch 1.10-11.el8 rhel-8-for-x86_64-appstream-rpms 20 k 2023-07-14T19:16:44.200 INFO:teuthology.orchestra.run.smithi101.stdout: python3-jwt noarch 1.6.1-2.el8 rhel-8-for-x86_64-baseos-rpms 43 k 2023-07-14T19:16:44.200 INFO:teuthology.orchestra.run.smithi101.stdout: python3-kubernetes noarch 1:11.0.0-6.el8 epel 1.2 M 2023-07-14T19:16:44.200 INFO:teuthology.orchestra.run.smithi101.stdout: python3-logutils noarch 0.3.5-11.el8 epel 49 k 2023-07-14T19:16:44.200 INFO:teuthology.orchestra.run.smithi101.stdout: python3-mako noarch 1.0.6-13.el8 rhel-8-for-x86_64-appstream-rpms 157 k 2023-07-14T19:16:44.200 INFO:teuthology.orchestra.run.smithi101.stdout: python3-markupsafe x86_64 0.23-19.el8 rhel-8-for-x86_64-appstream-rpms 39 k 2023-07-14T19:16:44.201 INFO:teuthology.orchestra.run.smithi101.stdout: python3-more-itertools noarch 7.2.0-3.el8 epel 59 k 2023-07-14T19:16:44.201 INFO:teuthology.orchestra.run.smithi101.stdout: python3-natsort noarch 7.1.1-2.el8 epel 60 k 2023-07-14T19:16:44.201 INFO:teuthology.orchestra.run.smithi101.stdout: python3-numpy x86_64 1:1.14.3-10.el8 rhel-8-for-x86_64-appstream-rpms 3.7 M 2023-07-14T19:16:44.201 INFO:teuthology.orchestra.run.smithi101.stdout: python3-numpy-f2py x86_64 1:1.14.3-10.el8 rhel-8-for-x86_64-appstream-rpms 225 k 2023-07-14T19:16:44.201 INFO:teuthology.orchestra.run.smithi101.stdout: python3-oauthlib noarch 2.1.0-1.el8 rhel-8-for-x86_64-baseos-rpms 155 k 2023-07-14T19:16:44.201 INFO:teuthology.orchestra.run.smithi101.stdout: python3-pecan noarch 1.3.2-9.el8 epel 283 k 2023-07-14T19:16:44.201 INFO:teuthology.orchestra.run.smithi101.stdout: python3-portend noarch 2.6-1.el8 epel 16 k 2023-07-14T19:16:44.202 INFO:teuthology.orchestra.run.smithi101.stdout: python3-prettytable noarch 0.7.2-14.el8 rhel-8-for-x86_64-appstream-rpms 44 k 2023-07-14T19:16:44.202 INFO:teuthology.orchestra.run.smithi101.stdout: python3-pyOpenSSL noarch 19.0.0-1.el8 rhel-8-for-x86_64-appstream-rpms 103 k 2023-07-14T19:16:44.202 INFO:teuthology.orchestra.run.smithi101.stdout: python3-pyasn1 noarch 0.3.7-6.el8 rhel-8-for-x86_64-appstream-rpms 126 k 2023-07-14T19:16:44.202 INFO:teuthology.orchestra.run.smithi101.stdout: python3-pyasn1-modules noarch 0.3.7-6.el8 rhel-8-for-x86_64-appstream-rpms 110 k 2023-07-14T19:16:44.202 INFO:teuthology.orchestra.run.smithi101.stdout: python3-pycparser noarch 2.14-14.el8 rhel-8-for-x86_64-baseos-rpms 109 k 2023-07-14T19:16:44.202 INFO:teuthology.orchestra.run.smithi101.stdout: python3-pytz noarch 2017.2-9.el8 rhel-8-for-x86_64-appstream-rpms 54 k 2023-07-14T19:16:44.203 INFO:teuthology.orchestra.run.smithi101.stdout: python3-repoze-lru noarch 0.7-6.el8 epel 34 k 2023-07-14T19:16:44.203 INFO:teuthology.orchestra.run.smithi101.stdout: python3-requests-oauthlib noarch 1.0.0-1.el8 rhel-8-for-x86_64-baseos-rpms 43 k 2023-07-14T19:16:44.203 INFO:teuthology.orchestra.run.smithi101.stdout: python3-routes noarch 2.4.1-12.el8 epel 196 k 2023-07-14T19:16:44.203 INFO:teuthology.orchestra.run.smithi101.stdout: python3-rsa noarch 4.8-1.el8 epel 61 k 2023-07-14T19:16:44.203 INFO:teuthology.orchestra.run.smithi101.stdout: python3-scipy x86_64 1.0.0-21.module+el8.5.0+10916+41bd434d rhel-8-for-x86_64-appstream-rpms 14 M 2023-07-14T19:16:44.203 INFO:teuthology.orchestra.run.smithi101.stdout: python3-simplegeneric noarch 0.8.1-17.el8 epel 19 k 2023-07-14T19:16:44.203 INFO:teuthology.orchestra.run.smithi101.stdout: python3-singledispatch noarch 3.4.0.3-18.el8 epel 24 k 2023-07-14T19:16:44.203 INFO:teuthology.orchestra.run.smithi101.stdout: python3-tempora noarch 1.14.1-5.el8 epel 29 k 2023-07-14T19:16:44.204 INFO:teuthology.orchestra.run.smithi101.stdout: python3-trustme noarch 0.6.0-4.el8 epel 27 k 2023-07-14T19:16:44.204 INFO:teuthology.orchestra.run.smithi101.stdout: python3-waitress noarch 1.4.3-1.el8 epel 255 k 2023-07-14T19:16:44.204 INFO:teuthology.orchestra.run.smithi101.stdout: python3-webob noarch 1.8.5-1.el8.1 epel 251 k 2023-07-14T19:16:44.204 INFO:teuthology.orchestra.run.smithi101.stdout: python3-websocket-client noarch 0.56.0-5.el8 epel 61 k 2023-07-14T19:16:44.204 INFO:teuthology.orchestra.run.smithi101.stdout: python3-webtest noarch 2.0.33-1.el8 epel 85 k 2023-07-14T19:16:44.204 INFO:teuthology.orchestra.run.smithi101.stdout: python3-werkzeug noarch 0.12.2-4.el8 rhel-8-for-x86_64-appstream-rpms 457 k 2023-07-14T19:16:44.205 INFO:teuthology.orchestra.run.smithi101.stdout: python3-zc-lockfile noarch 2.0-2.el8 epel 23 k 2023-07-14T19:16:44.205 INFO:teuthology.orchestra.run.smithi101.stdout: socat x86_64 1.7.4.1-1.el8 rhel-8-for-x86_64-appstream-rpms 323 k 2023-07-14T19:16:44.205 INFO:teuthology.orchestra.run.smithi101.stdout: thrift x86_64 0.13.0-2.el8 epel 1.7 M 2023-07-14T19:16:44.205 INFO:teuthology.orchestra.run.smithi101.stdout: uuid x86_64 1.6.2-43.el8 rhel-8-for-x86_64-appstream-rpms 64 k 2023-07-14T19:16:44.206 INFO:teuthology.orchestra.run.smithi101.stdout: xmlstarlet x86_64 1.6.1-20.el8 lab-extras 70 k 2023-07-14T19:16:44.206 INFO:teuthology.orchestra.run.smithi101.stdout: zlib-devel x86_64 1.2.11-18.el8_5 rhel-8-for-x86_64-baseos-rpms 58 k 2023-07-14T19:16:44.206 INFO:teuthology.orchestra.run.smithi101.stdout:Enabling module streams: 2023-07-14T19:16:44.206 INFO:teuthology.orchestra.run.smithi101.stdout: postgresql 10 2023-07-14T19:16:44.207 INFO:teuthology.orchestra.run.smithi101.stdout: 2023-07-14T19:16:44.207 INFO:teuthology.orchestra.run.smithi101.stdout:Transaction Summary 2023-07-14T19:16:44.207 INFO:teuthology.orchestra.run.smithi101.stdout:=========================================================================================================================== 2023-07-14T19:16:44.208 INFO:teuthology.orchestra.run.smithi101.stdout:Install 128 Packages 2023-07-14T19:16:44.208 INFO:teuthology.orchestra.run.smithi101.stdout:Upgrade 3 Packages 2023-07-14T19:16:44.208 INFO:teuthology.orchestra.run.smithi101.stdout: 2023-07-14T19:16:44.209 INFO:teuthology.orchestra.run.smithi101.stdout:Total download size: 206 M 2023-07-14T19:16:44.210 INFO:teuthology.orchestra.run.smithi101.stdout:Downloading Packages: 2023-07-14T19:16:44.735 INFO:teuthology.orchestra.run.smithi101.stdout:(1/131): ceph-18.0.0-4940.g81c9387d.el8.x86_64. 12 kB/s | 6.4 kB 00:00 2023-07-14T19:16:50.860 INFO:teuthology.orchestra.run.smithi101.stdout:(2/131): ceph-fuse-18.0.0-4940.g81c9387d.el8.x8 139 kB/s | 850 kB 00:06 2023-07-14T19:16:52.810 INFO:teuthology.orchestra.run.smithi101.stdout:(3/131): ceph-immutable-object-cache-18.0.0-494 86 kB/s | 167 kB 00:01 2023-07-14T19:17:01.368 INFO:teuthology.orchestra.run.smithi101.stdout:(4/131): ceph-mds-18.0.0-4940.g81c9387d.el8.x86 275 kB/s | 2.3 MB 00:08 2023-07-14T19:17:06.399 INFO:teuthology.orchestra.run.smithi098.stdout:Extra Packages for Enterprise Linux 329 kB/s | 16 MB 00:48 2023-07-14T19:17:08.385 INFO:teuthology.orchestra.run.smithi101.stdout:(5/131): ceph-mgr-18.0.0-4940.g81c9387d.el8.x86 226 kB/s | 1.6 MB 00:07 2023-07-14T19:17:09.963 INFO:teuthology.orchestra.run.smithi098.stdout:lab-extras 139 kB/s | 24 kB 00:00 2023-07-14T19:17:10.898 INFO:teuthology.orchestra.run.smithi098.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 61 MB/s | 53 MB 00:00 2023-07-14T19:17:18.966 INFO:teuthology.orchestra.run.smithi098.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 58 MB/s | 47 MB 00:00 2023-07-14T19:17:19.034 INFO:teuthology.orchestra.run.smithi101.stdout:(6/131): ceph-base-18.0.0-4940.g81c9387d.el8.x8 158 kB/s | 5.4 MB 00:34 2023-07-14T19:17:24.159 INFO:teuthology.orchestra.run.smithi101.stdout:(7/131): ceph-mon-18.0.0-4940.g81c9387d.el8.x86 300 kB/s | 4.6 MB 00:15 2023-07-14T19:17:29.847 INFO:teuthology.orchestra.run.smithi098.stdout:Package librados2-1:12.2.7-9.el8.x86_64 is already installed. 2023-07-14T19:17:29.848 INFO:teuthology.orchestra.run.smithi098.stdout:Package librbd1-1:12.2.7-9.el8.x86_64 is already installed. 2023-07-14T19:17:29.862 INFO:teuthology.orchestra.run.smithi098.stdout:Package NetworkManager-1:1.36.0-4.el8.x86_64 is already installed. 2023-07-14T19:17:29.863 INFO:teuthology.orchestra.run.smithi098.stdout:Package iproute-5.15.0-4.el8.x86_64 is already installed. 2023-07-14T19:17:29.865 INFO:teuthology.orchestra.run.smithi098.stdout:Package util-linux-2.32.1-35.el8.x86_64 is already installed. 2023-07-14T19:17:29.866 INFO:teuthology.orchestra.run.smithi098.stdout:Package libacl-devel-2.2.53-1.el8.x86_64 is already installed. 2023-07-14T19:17:29.868 INFO:teuthology.orchestra.run.smithi098.stdout:Package libattr-devel-2.4.48-3.el8.x86_64 is already installed. 2023-07-14T19:17:29.871 INFO:teuthology.orchestra.run.smithi098.stdout:Package xfsdump-3.1.8-4.el8.x86_64 is already installed. 2023-07-14T19:17:29.872 INFO:teuthology.orchestra.run.smithi098.stdout:Package xfsprogs-5.0.0-10.el8.x86_64 is already installed. 2023-07-14T19:17:30.004 INFO:teuthology.orchestra.run.smithi098.stdout:Dependencies resolved. 2023-07-14T19:17:30.017 INFO:teuthology.orchestra.run.smithi098.stdout:=========================================================================================================================== 2023-07-14T19:17:30.017 INFO:teuthology.orchestra.run.smithi098.stdout: Package Arch Version Repository Size 2023-07-14T19:17:30.017 INFO:teuthology.orchestra.run.smithi098.stdout:=========================================================================================================================== 2023-07-14T19:17:30.017 INFO:teuthology.orchestra.run.smithi098.stdout:Installing: 2023-07-14T19:17:30.017 INFO:teuthology.orchestra.run.smithi098.stdout: NetworkManager-initscripts-updown noarch 1:1.36.0-9.el8_6 rhel-8-for-x86_64-baseos-rpms 138 k 2023-07-14T19:17:30.018 INFO:teuthology.orchestra.run.smithi098.stdout: bison x86_64 3.0.4-10.el8 rhel-8-for-x86_64-appstream-rpms 688 k 2023-07-14T19:17:30.018 INFO:teuthology.orchestra.run.smithi098.stdout: ceph x86_64 2:18.0.0-4940.g81c9387d.el8 ceph 6.4 k 2023-07-14T19:17:30.018 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-base x86_64 2:18.0.0-4940.g81c9387d.el8 ceph 5.4 M 2023-07-14T19:17:30.018 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-fuse x86_64 2:18.0.0-4940.g81c9387d.el8 ceph 850 k 2023-07-14T19:17:30.018 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-immutable-object-cache x86_64 2:18.0.0-4940.g81c9387d.el8 ceph 167 k 2023-07-14T19:17:30.018 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-mgr x86_64 2:18.0.0-4940.g81c9387d.el8 ceph 1.6 M 2023-07-14T19:17:30.018 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-mgr-cephadm noarch 2:18.0.0-4940.g81c9387d.el8 ceph-noarch 129 k 2023-07-14T19:17:30.019 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-mgr-dashboard noarch 2:18.0.0-4940.g81c9387d.el8 ceph-noarch 1.7 M 2023-07-14T19:17:30.019 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-mgr-diskprediction-local noarch 2:18.0.0-4940.g81c9387d.el8 ceph-noarch 7.4 M 2023-07-14T19:17:30.019 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-mgr-rook noarch 2:18.0.0-4940.g81c9387d.el8 ceph-noarch 47 k 2023-07-14T19:17:30.019 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-radosgw x86_64 2:18.0.0-4940.g81c9387d.el8 ceph 14 M 2023-07-14T19:17:30.019 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-test x86_64 2:18.0.0-4940.g81c9387d.el8 ceph 48 M 2023-07-14T19:17:30.019 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-volume noarch 2:18.0.0-4940.g81c9387d.el8 ceph-noarch 258 k 2023-07-14T19:17:30.019 INFO:teuthology.orchestra.run.smithi098.stdout: cephadm noarch 2:18.0.0-4940.g81c9387d.el8 ceph-noarch 165 k 2023-07-14T19:17:30.020 INFO:teuthology.orchestra.run.smithi098.stdout: cephfs-mirror x86_64 2:18.0.0-4940.g81c9387d.el8 ceph 260 k 2023-07-14T19:17:30.020 INFO:teuthology.orchestra.run.smithi098.stdout: cephfs-top noarch 2:18.0.0-4940.g81c9387d.el8 ceph-noarch 18 k 2023-07-14T19:17:30.020 INFO:teuthology.orchestra.run.smithi098.stdout: elfutils-libelf-devel x86_64 0.186-1.el8 rhel-8-for-x86_64-baseos-rpms 60 k 2023-07-14T19:17:30.020 INFO:teuthology.orchestra.run.smithi098.stdout: flex x86_64 2.6.1-9.el8 rhel-8-for-x86_64-appstream-rpms 320 k 2023-07-14T19:17:30.020 INFO:teuthology.orchestra.run.smithi098.stdout: libaio-devel x86_64 0.3.112-1.el8 rhel-8-for-x86_64-baseos-rpms 19 k 2023-07-14T19:17:30.020 INFO:teuthology.orchestra.run.smithi098.stdout: libcephfs-devel x86_64 2:18.0.0-4940.g81c9387d.el8 ceph 31 k 2023-07-14T19:17:30.020 INFO:teuthology.orchestra.run.smithi098.stdout: libcephfs2 x86_64 2:18.0.0-4940.g81c9387d.el8 ceph 772 k 2023-07-14T19:17:30.021 INFO:teuthology.orchestra.run.smithi098.stdout: librados-devel x86_64 2:18.0.0-4940.g81c9387d.el8 ceph 139 k 2023-07-14T19:17:30.021 INFO:teuthology.orchestra.run.smithi098.stdout: libtool x86_64 2.4.6-25.el8 rhel-8-for-x86_64-appstream-rpms 709 k 2023-07-14T19:17:30.021 INFO:teuthology.orchestra.run.smithi098.stdout: libuuid-devel x86_64 2.32.1-35.el8 rhel-8-for-x86_64-baseos-rpms 98 k 2023-07-14T19:17:30.021 INFO:teuthology.orchestra.run.smithi098.stdout: openssl-devel x86_64 1:1.1.1k-7.el8_6 rhel-8-for-x86_64-baseos-rpms 2.3 M 2023-07-14T19:17:30.021 INFO:teuthology.orchestra.run.smithi098.stdout: postgresql x86_64 10.21-2.module+el8.6.0+15342+53518fac rhel-8-for-x86_64-appstream-rpms 1.5 M 2023-07-14T19:17:30.021 INFO:teuthology.orchestra.run.smithi098.stdout: postgresql-contrib x86_64 10.21-2.module+el8.6.0+15342+53518fac rhel-8-for-x86_64-appstream-rpms 808 k 2023-07-14T19:17:30.021 INFO:teuthology.orchestra.run.smithi098.stdout: postgresql-server x86_64 10.21-2.module+el8.6.0+15342+53518fac rhel-8-for-x86_64-appstream-rpms 5.1 M 2023-07-14T19:17:30.022 INFO:teuthology.orchestra.run.smithi098.stdout: python3-cephfs x86_64 2:18.0.0-4940.g81c9387d.el8 ceph 197 k 2023-07-14T19:17:30.022 INFO:teuthology.orchestra.run.smithi098.stdout: python3-rados x86_64 2:18.0.0-4940.g81c9387d.el8 ceph 371 k 2023-07-14T19:17:30.022 INFO:teuthology.orchestra.run.smithi098.stdout: python3-rbd x86_64 2:18.0.0-4940.g81c9387d.el8 ceph 364 k 2023-07-14T19:17:30.022 INFO:teuthology.orchestra.run.smithi098.stdout: python3-rgw x86_64 2:18.0.0-4940.g81c9387d.el8 ceph 107 k 2023-07-14T19:17:30.022 INFO:teuthology.orchestra.run.smithi098.stdout: rbd-fuse x86_64 2:18.0.0-4940.g81c9387d.el8 ceph 94 k 2023-07-14T19:17:30.022 INFO:teuthology.orchestra.run.smithi098.stdout: rbd-mirror x86_64 2:18.0.0-4940.g81c9387d.el8 ceph 3.5 M 2023-07-14T19:17:30.023 INFO:teuthology.orchestra.run.smithi098.stdout: rbd-nbd x86_64 2:18.0.0-4940.g81c9387d.el8 ceph 179 k 2023-07-14T19:17:30.023 INFO:teuthology.orchestra.run.smithi098.stdout: xfsprogs-devel x86_64 5.0.0-10.el8 rhel-8-for-x86_64-baseos-rpms 72 k 2023-07-14T19:17:30.023 INFO:teuthology.orchestra.run.smithi098.stdout:Upgrading: 2023-07-14T19:17:30.023 INFO:teuthology.orchestra.run.smithi098.stdout: iproute x86_64 5.15.0-4.el8_6.1 rhel-8-for-x86_64-baseos-rpms 798 k 2023-07-14T19:17:30.023 INFO:teuthology.orchestra.run.smithi098.stdout: librados2 x86_64 2:18.0.0-4940.g81c9387d.el8 ceph 3.7 M 2023-07-14T19:17:30.023 INFO:teuthology.orchestra.run.smithi098.stdout: librbd1 x86_64 2:18.0.0-4940.g81c9387d.el8 ceph 3.6 M 2023-07-14T19:17:30.023 INFO:teuthology.orchestra.run.smithi098.stdout:Installing dependencies: 2023-07-14T19:17:30.024 INFO:teuthology.orchestra.run.smithi098.stdout: automake noarch 1.16.1-7.el8 rhel-8-for-x86_64-appstream-rpms 713 k 2023-07-14T19:17:30.024 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-common x86_64 2:18.0.0-4940.g81c9387d.el8 ceph 22 M 2023-07-14T19:17:30.024 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-grafana-dashboards noarch 2:18.0.0-4940.g81c9387d.el8 ceph-noarch 23 k 2023-07-14T19:17:30.024 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-mds x86_64 2:18.0.0-4940.g81c9387d.el8 ceph 2.3 M 2023-07-14T19:17:30.024 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-mgr-modules-core noarch 2:18.0.0-4940.g81c9387d.el8 ceph-noarch 241 k 2023-07-14T19:17:30.024 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-mon x86_64 2:18.0.0-4940.g81c9387d.el8 ceph 4.6 M 2023-07-14T19:17:30.024 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-osd x86_64 2:18.0.0-4940.g81c9387d.el8 ceph 18 M 2023-07-14T19:17:30.025 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-prometheus-alerts noarch 2:18.0.0-4940.g81c9387d.el8 ceph-noarch 15 k 2023-07-14T19:17:30.025 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-selinux x86_64 2:18.0.0-4940.g81c9387d.el8 ceph 24 k 2023-07-14T19:17:30.025 INFO:teuthology.orchestra.run.smithi098.stdout: fuse x86_64 2.9.7-15.el8 rhel-8-for-x86_64-baseos-rpms 83 k 2023-07-14T19:17:30.025 INFO:teuthology.orchestra.run.smithi098.stdout: gperftools-libs x86_64 1:2.7-9.el8 epel 306 k 2023-07-14T19:17:30.025 INFO:teuthology.orchestra.run.smithi098.stdout: jq x86_64 1.6-3.el8 rhel-8-for-x86_64-appstream-rpms 202 k 2023-07-14T19:17:30.025 INFO:teuthology.orchestra.run.smithi098.stdout: keyutils-libs-devel x86_64 1.5.10-9.el8 rhel-8-for-x86_64-baseos-rpms 48 k 2023-07-14T19:17:30.025 INFO:teuthology.orchestra.run.smithi098.stdout: krb5-devel x86_64 1.18.2-14.el8 rhel-8-for-x86_64-baseos-rpms 560 k 2023-07-14T19:17:30.026 INFO:teuthology.orchestra.run.smithi098.stdout: libcephsqlite x86_64 2:18.0.0-4940.g81c9387d.el8 ceph 135 k 2023-07-14T19:17:30.026 INFO:teuthology.orchestra.run.smithi098.stdout: libcom_err-devel x86_64 1.45.6-4.el8 rhel-8-for-x86_64-baseos-rpms 39 k 2023-07-14T19:17:30.026 INFO:teuthology.orchestra.run.smithi098.stdout: libgfortran x86_64 8.5.0-10.1.el8_6 rhel-8-for-x86_64-baseos-rpms 644 k 2023-07-14T19:17:30.026 INFO:teuthology.orchestra.run.smithi098.stdout: liboath x86_64 2.6.2-3.el8 epel 59 k 2023-07-14T19:17:30.026 INFO:teuthology.orchestra.run.smithi098.stdout: libpq x86_64 13.5-1.el8 rhel-8-for-x86_64-appstream-rpms 198 k 2023-07-14T19:17:30.026 INFO:teuthology.orchestra.run.smithi098.stdout: libquadmath x86_64 8.5.0-10.1.el8_6 rhel-8-for-x86_64-baseos-rpms 171 k 2023-07-14T19:17:30.026 INFO:teuthology.orchestra.run.smithi098.stdout: librabbitmq x86_64 0.9.0-3.el8 rhel-8-for-x86_64-baseos-rpms 47 k 2023-07-14T19:17:30.026 INFO:teuthology.orchestra.run.smithi098.stdout: libradosstriper1 x86_64 2:18.0.0-4940.g81c9387d.el8 ceph 473 k 2023-07-14T19:17:30.027 INFO:teuthology.orchestra.run.smithi098.stdout: librdkafka x86_64 0.11.4-3.el8 rhel-8-for-x86_64-appstream-rpms 354 k 2023-07-14T19:17:30.027 INFO:teuthology.orchestra.run.smithi098.stdout: librgw2 x86_64 2:18.0.0-4940.g81c9387d.el8 ceph 6.8 M 2023-07-14T19:17:30.027 INFO:teuthology.orchestra.run.smithi098.stdout: libselinux-devel x86_64 2.9-5.el8 rhel-8-for-x86_64-baseos-rpms 200 k 2023-07-14T19:17:30.027 INFO:teuthology.orchestra.run.smithi098.stdout: libsepol-devel x86_64 2.9-3.el8 rhel-8-for-x86_64-baseos-rpms 87 k 2023-07-14T19:17:30.027 INFO:teuthology.orchestra.run.smithi098.stdout: libunwind x86_64 1.3.1-3.el8 epel 75 k 2023-07-14T19:17:30.027 INFO:teuthology.orchestra.run.smithi098.stdout: libverto-devel x86_64 0.3.0-5.el8 rhel-8-for-x86_64-baseos-rpms 18 k 2023-07-14T19:17:30.027 INFO:teuthology.orchestra.run.smithi098.stdout: lttng-ust x86_64 2.8.1-11.el8 rhel-8-for-x86_64-appstream-rpms 259 k 2023-07-14T19:17:30.028 INFO:teuthology.orchestra.run.smithi098.stdout: oniguruma x86_64 6.8.2-2.el8 rhel-8-for-x86_64-appstream-rpms 187 k 2023-07-14T19:17:30.028 INFO:teuthology.orchestra.run.smithi098.stdout: openblas x86_64 0.3.15-3.el8 rhel-8-for-x86_64-appstream-rpms 4.7 M 2023-07-14T19:17:30.028 INFO:teuthology.orchestra.run.smithi098.stdout: openblas-threads x86_64 0.3.15-3.el8 rhel-8-for-x86_64-appstream-rpms 4.9 M 2023-07-14T19:17:30.028 INFO:teuthology.orchestra.run.smithi098.stdout: pcre2-devel x86_64 10.32-2.el8 rhel-8-for-x86_64-baseos-rpms 605 k 2023-07-14T19:17:30.028 INFO:teuthology.orchestra.run.smithi098.stdout: pcre2-utf16 x86_64 10.32-2.el8 rhel-8-for-x86_64-baseos-rpms 229 k 2023-07-14T19:17:30.028 INFO:teuthology.orchestra.run.smithi098.stdout: pcre2-utf32 x86_64 10.32-2.el8 rhel-8-for-x86_64-baseos-rpms 220 k 2023-07-14T19:17:30.028 INFO:teuthology.orchestra.run.smithi098.stdout: python3-asyncssh noarch 2.7.0-2.el8 epel 476 k 2023-07-14T19:17:30.029 INFO:teuthology.orchestra.run.smithi098.stdout: python3-babel noarch 2.5.1-7.el8 rhel-8-for-x86_64-appstream-rpms 4.8 M 2023-07-14T19:17:30.029 INFO:teuthology.orchestra.run.smithi098.stdout: python3-bcrypt x86_64 3.1.6-2.el8.1 epel 44 k 2023-07-14T19:17:30.029 INFO:teuthology.orchestra.run.smithi098.stdout: python3-beautifulsoup4 noarch 4.6.3-2.el8.1 epel 185 k 2023-07-14T19:17:30.029 INFO:teuthology.orchestra.run.smithi098.stdout: python3-cachetools noarch 3.1.1-4.el8 epel 33 k 2023-07-14T19:17:30.029 INFO:teuthology.orchestra.run.smithi098.stdout: python3-ceph-argparse x86_64 2:18.0.0-4940.g81c9387d.el8 ceph 45 k 2023-07-14T19:17:30.029 INFO:teuthology.orchestra.run.smithi098.stdout: python3-ceph-common x86_64 2:18.0.0-4940.g81c9387d.el8 ceph 122 k 2023-07-14T19:17:30.029 INFO:teuthology.orchestra.run.smithi098.stdout: python3-certifi noarch 2018.10.15-7.el8 epel 16 k 2023-07-14T19:17:30.029 INFO:teuthology.orchestra.run.smithi098.stdout: python3-cffi x86_64 1.11.5-5.el8 rhel-8-for-x86_64-baseos-rpms 238 k 2023-07-14T19:17:30.030 INFO:teuthology.orchestra.run.smithi098.stdout: python3-cheroot noarch 8.5.2-1.el8 epel 173 k 2023-07-14T19:17:30.030 INFO:teuthology.orchestra.run.smithi098.stdout: python3-cherrypy noarch 18.4.0-1.el8 epel 384 k 2023-07-14T19:17:30.030 INFO:teuthology.orchestra.run.smithi098.stdout: python3-cryptography x86_64 3.2.1-5.el8 rhel-8-for-x86_64-baseos-rpms 559 k 2023-07-14T19:17:30.030 INFO:teuthology.orchestra.run.smithi098.stdout: python3-google-auth noarch 1:1.1.1-10.el8 epel 81 k 2023-07-14T19:17:30.030 INFO:teuthology.orchestra.run.smithi098.stdout: python3-jaraco noarch 6.2-6.el8 epel 11 k 2023-07-14T19:17:30.030 INFO:teuthology.orchestra.run.smithi098.stdout: python3-jaraco-functools noarch 2.0-4.el8 epel 18 k 2023-07-14T19:17:30.030 INFO:teuthology.orchestra.run.smithi098.stdout: python3-jinja2 noarch 2.10.1-3.el8 rhel-8-for-x86_64-appstream-rpms 538 k 2023-07-14T19:17:30.031 INFO:teuthology.orchestra.run.smithi098.stdout: python3-jsonpatch noarch 1.21-2.el8 rhel-8-for-x86_64-appstream-rpms 27 k 2023-07-14T19:17:30.031 INFO:teuthology.orchestra.run.smithi098.stdout: python3-jsonpointer noarch 1.10-11.el8 rhel-8-for-x86_64-appstream-rpms 20 k 2023-07-14T19:17:30.031 INFO:teuthology.orchestra.run.smithi098.stdout: python3-jwt noarch 1.6.1-2.el8 rhel-8-for-x86_64-baseos-rpms 43 k 2023-07-14T19:17:30.031 INFO:teuthology.orchestra.run.smithi098.stdout: python3-kubernetes noarch 1:11.0.0-6.el8 epel 1.2 M 2023-07-14T19:17:30.031 INFO:teuthology.orchestra.run.smithi098.stdout: python3-logutils noarch 0.3.5-11.el8 epel 49 k 2023-07-14T19:17:30.031 INFO:teuthology.orchestra.run.smithi098.stdout: python3-mako noarch 1.0.6-13.el8 rhel-8-for-x86_64-appstream-rpms 157 k 2023-07-14T19:17:30.031 INFO:teuthology.orchestra.run.smithi098.stdout: python3-markupsafe x86_64 0.23-19.el8 rhel-8-for-x86_64-appstream-rpms 39 k 2023-07-14T19:17:30.031 INFO:teuthology.orchestra.run.smithi098.stdout: python3-more-itertools noarch 7.2.0-3.el8 epel 59 k 2023-07-14T19:17:30.032 INFO:teuthology.orchestra.run.smithi098.stdout: python3-natsort noarch 7.1.1-2.el8 epel 60 k 2023-07-14T19:17:30.032 INFO:teuthology.orchestra.run.smithi098.stdout: python3-numpy x86_64 1:1.14.3-10.el8 rhel-8-for-x86_64-appstream-rpms 3.7 M 2023-07-14T19:17:30.032 INFO:teuthology.orchestra.run.smithi098.stdout: python3-numpy-f2py x86_64 1:1.14.3-10.el8 rhel-8-for-x86_64-appstream-rpms 225 k 2023-07-14T19:17:30.032 INFO:teuthology.orchestra.run.smithi098.stdout: python3-oauthlib noarch 2.1.0-1.el8 rhel-8-for-x86_64-baseos-rpms 155 k 2023-07-14T19:17:30.032 INFO:teuthology.orchestra.run.smithi098.stdout: python3-pecan noarch 1.3.2-9.el8 epel 283 k 2023-07-14T19:17:30.033 INFO:teuthology.orchestra.run.smithi098.stdout: python3-portend noarch 2.6-1.el8 epel 16 k 2023-07-14T19:17:30.033 INFO:teuthology.orchestra.run.smithi098.stdout: python3-prettytable noarch 0.7.2-14.el8 rhel-8-for-x86_64-appstream-rpms 44 k 2023-07-14T19:17:30.033 INFO:teuthology.orchestra.run.smithi098.stdout: python3-pyOpenSSL noarch 19.0.0-1.el8 rhel-8-for-x86_64-appstream-rpms 103 k 2023-07-14T19:17:30.033 INFO:teuthology.orchestra.run.smithi098.stdout: python3-pyasn1 noarch 0.3.7-6.el8 rhel-8-for-x86_64-appstream-rpms 126 k 2023-07-14T19:17:30.033 INFO:teuthology.orchestra.run.smithi098.stdout: python3-pyasn1-modules noarch 0.3.7-6.el8 rhel-8-for-x86_64-appstream-rpms 110 k 2023-07-14T19:17:30.033 INFO:teuthology.orchestra.run.smithi098.stdout: python3-pycparser noarch 2.14-14.el8 rhel-8-for-x86_64-baseos-rpms 109 k 2023-07-14T19:17:30.033 INFO:teuthology.orchestra.run.smithi098.stdout: python3-pytz noarch 2017.2-9.el8 rhel-8-for-x86_64-appstream-rpms 54 k 2023-07-14T19:17:30.033 INFO:teuthology.orchestra.run.smithi098.stdout: python3-repoze-lru noarch 0.7-6.el8 epel 34 k 2023-07-14T19:17:30.034 INFO:teuthology.orchestra.run.smithi098.stdout: python3-requests-oauthlib noarch 1.0.0-1.el8 rhel-8-for-x86_64-baseos-rpms 43 k 2023-07-14T19:17:30.034 INFO:teuthology.orchestra.run.smithi098.stdout: python3-routes noarch 2.4.1-12.el8 epel 196 k 2023-07-14T19:17:30.034 INFO:teuthology.orchestra.run.smithi098.stdout: python3-rsa noarch 4.8-1.el8 epel 61 k 2023-07-14T19:17:30.034 INFO:teuthology.orchestra.run.smithi098.stdout: python3-scipy x86_64 1.0.0-21.module+el8.5.0+10916+41bd434d rhel-8-for-x86_64-appstream-rpms 14 M 2023-07-14T19:17:30.034 INFO:teuthology.orchestra.run.smithi098.stdout: python3-simplegeneric noarch 0.8.1-17.el8 epel 19 k 2023-07-14T19:17:30.034 INFO:teuthology.orchestra.run.smithi098.stdout: python3-singledispatch noarch 3.4.0.3-18.el8 epel 24 k 2023-07-14T19:17:30.034 INFO:teuthology.orchestra.run.smithi098.stdout: python3-tempora noarch 1.14.1-5.el8 epel 29 k 2023-07-14T19:17:30.035 INFO:teuthology.orchestra.run.smithi098.stdout: python3-trustme noarch 0.6.0-4.el8 epel 27 k 2023-07-14T19:17:30.035 INFO:teuthology.orchestra.run.smithi098.stdout: python3-waitress noarch 1.4.3-1.el8 epel 255 k 2023-07-14T19:17:30.035 INFO:teuthology.orchestra.run.smithi098.stdout: python3-webob noarch 1.8.5-1.el8.1 epel 251 k 2023-07-14T19:17:30.035 INFO:teuthology.orchestra.run.smithi098.stdout: python3-websocket-client noarch 0.56.0-5.el8 epel 61 k 2023-07-14T19:17:30.035 INFO:teuthology.orchestra.run.smithi098.stdout: python3-webtest noarch 2.0.33-1.el8 epel 85 k 2023-07-14T19:17:30.035 INFO:teuthology.orchestra.run.smithi098.stdout: python3-werkzeug noarch 0.12.2-4.el8 rhel-8-for-x86_64-appstream-rpms 457 k 2023-07-14T19:17:30.035 INFO:teuthology.orchestra.run.smithi098.stdout: python3-zc-lockfile noarch 2.0-2.el8 epel 23 k 2023-07-14T19:17:30.035 INFO:teuthology.orchestra.run.smithi098.stdout: socat x86_64 1.7.4.1-1.el8 rhel-8-for-x86_64-appstream-rpms 323 k 2023-07-14T19:17:30.036 INFO:teuthology.orchestra.run.smithi098.stdout: thrift x86_64 0.13.0-2.el8 epel 1.7 M 2023-07-14T19:17:30.036 INFO:teuthology.orchestra.run.smithi098.stdout: uuid x86_64 1.6.2-43.el8 rhel-8-for-x86_64-appstream-rpms 64 k 2023-07-14T19:17:30.036 INFO:teuthology.orchestra.run.smithi098.stdout: xmlstarlet x86_64 1.6.1-20.el8 lab-extras 70 k 2023-07-14T19:17:30.036 INFO:teuthology.orchestra.run.smithi098.stdout: zlib-devel x86_64 1.2.11-18.el8_5 rhel-8-for-x86_64-baseos-rpms 58 k 2023-07-14T19:17:30.036 INFO:teuthology.orchestra.run.smithi098.stdout:Enabling module streams: 2023-07-14T19:17:30.037 INFO:teuthology.orchestra.run.smithi098.stdout: postgresql 10 2023-07-14T19:17:30.037 INFO:teuthology.orchestra.run.smithi098.stdout: 2023-07-14T19:17:30.037 INFO:teuthology.orchestra.run.smithi098.stdout:Transaction Summary 2023-07-14T19:17:30.038 INFO:teuthology.orchestra.run.smithi098.stdout:=========================================================================================================================== 2023-07-14T19:17:30.038 INFO:teuthology.orchestra.run.smithi098.stdout:Install 128 Packages 2023-07-14T19:17:30.038 INFO:teuthology.orchestra.run.smithi098.stdout:Upgrade 3 Packages 2023-07-14T19:17:30.038 INFO:teuthology.orchestra.run.smithi098.stdout: 2023-07-14T19:17:30.040 INFO:teuthology.orchestra.run.smithi098.stdout:Total download size: 206 M 2023-07-14T19:17:30.040 INFO:teuthology.orchestra.run.smithi098.stdout:Downloading Packages: 2023-07-14T19:17:30.458 INFO:teuthology.orchestra.run.smithi098.stdout:(1/131): ceph-18.0.0-4940.g81c9387d.el8.x86_64. 16 kB/s | 6.4 kB 00:00 2023-07-14T19:17:31.038 INFO:teuthology.orchestra.run.smithi139.stdout:Extra Packages for Enterprise Linux 218 kB/s | 16 MB 01:13 2023-07-14T19:17:33.733 INFO:teuthology.orchestra.run.smithi098.stdout:(2/131): ceph-fuse-18.0.0-4940.g81c9387d.el8.x8 260 kB/s | 850 kB 00:03 2023-07-14T19:17:34.217 INFO:teuthology.orchestra.run.smithi098.stdout:(3/131): ceph-immutable-object-cache-18.0.0-494 345 kB/s | 167 kB 00:00 2023-07-14T19:17:34.501 INFO:teuthology.orchestra.run.smithi139.stdout:lab-extras 396 kB/s | 24 kB 00:00 2023-07-14T19:17:35.394 INFO:teuthology.orchestra.run.smithi139.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 64 MB/s | 53 MB 00:00 2023-07-14T19:17:43.479 INFO:teuthology.orchestra.run.smithi139.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 61 MB/s | 47 MB 00:00 2023-07-14T19:17:47.258 INFO:teuthology.orchestra.run.smithi098.stdout:(4/131): ceph-mds-18.0.0-4940.g81c9387d.el8.x86 180 kB/s | 2.3 MB 00:13 2023-07-14T19:17:54.551 INFO:teuthology.orchestra.run.smithi139.stdout:Package librados2-1:12.2.7-9.el8.x86_64 is already installed. 2023-07-14T19:17:54.552 INFO:teuthology.orchestra.run.smithi139.stdout:Package librbd1-1:12.2.7-9.el8.x86_64 is already installed. 2023-07-14T19:17:54.566 INFO:teuthology.orchestra.run.smithi139.stdout:Package NetworkManager-1:1.36.0-4.el8.x86_64 is already installed. 2023-07-14T19:17:54.567 INFO:teuthology.orchestra.run.smithi139.stdout:Package iproute-5.15.0-4.el8.x86_64 is already installed. 2023-07-14T19:17:54.568 INFO:teuthology.orchestra.run.smithi139.stdout:Package util-linux-2.32.1-35.el8.x86_64 is already installed. 2023-07-14T19:17:54.569 INFO:teuthology.orchestra.run.smithi139.stdout:Package libacl-devel-2.2.53-1.el8.x86_64 is already installed. 2023-07-14T19:17:54.571 INFO:teuthology.orchestra.run.smithi139.stdout:Package libattr-devel-2.4.48-3.el8.x86_64 is already installed. 2023-07-14T19:17:54.574 INFO:teuthology.orchestra.run.smithi139.stdout:Package xfsdump-3.1.8-4.el8.x86_64 is already installed. 2023-07-14T19:17:54.575 INFO:teuthology.orchestra.run.smithi139.stdout:Package xfsprogs-5.0.0-10.el8.x86_64 is already installed. 2023-07-14T19:17:54.708 INFO:teuthology.orchestra.run.smithi139.stdout:Dependencies resolved. 2023-07-14T19:17:54.720 INFO:teuthology.orchestra.run.smithi139.stdout:=========================================================================================================================== 2023-07-14T19:17:54.720 INFO:teuthology.orchestra.run.smithi139.stdout: Package Arch Version Repository Size 2023-07-14T19:17:54.721 INFO:teuthology.orchestra.run.smithi139.stdout:=========================================================================================================================== 2023-07-14T19:17:54.721 INFO:teuthology.orchestra.run.smithi139.stdout:Installing: 2023-07-14T19:17:54.721 INFO:teuthology.orchestra.run.smithi139.stdout: NetworkManager-initscripts-updown noarch 1:1.36.0-9.el8_6 rhel-8-for-x86_64-baseos-rpms 138 k 2023-07-14T19:17:54.721 INFO:teuthology.orchestra.run.smithi139.stdout: bison x86_64 3.0.4-10.el8 rhel-8-for-x86_64-appstream-rpms 688 k 2023-07-14T19:17:54.721 INFO:teuthology.orchestra.run.smithi139.stdout: ceph x86_64 2:18.0.0-4940.g81c9387d.el8 ceph 6.4 k 2023-07-14T19:17:54.721 INFO:teuthology.orchestra.run.smithi139.stdout: ceph-base x86_64 2:18.0.0-4940.g81c9387d.el8 ceph 5.4 M 2023-07-14T19:17:54.721 INFO:teuthology.orchestra.run.smithi139.stdout: ceph-fuse x86_64 2:18.0.0-4940.g81c9387d.el8 ceph 850 k 2023-07-14T19:17:54.722 INFO:teuthology.orchestra.run.smithi139.stdout: ceph-immutable-object-cache x86_64 2:18.0.0-4940.g81c9387d.el8 ceph 167 k 2023-07-14T19:17:54.722 INFO:teuthology.orchestra.run.smithi139.stdout: ceph-mgr x86_64 2:18.0.0-4940.g81c9387d.el8 ceph 1.6 M 2023-07-14T19:17:54.722 INFO:teuthology.orchestra.run.smithi139.stdout: ceph-mgr-cephadm noarch 2:18.0.0-4940.g81c9387d.el8 ceph-noarch 129 k 2023-07-14T19:17:54.722 INFO:teuthology.orchestra.run.smithi139.stdout: ceph-mgr-dashboard noarch 2:18.0.0-4940.g81c9387d.el8 ceph-noarch 1.7 M 2023-07-14T19:17:54.722 INFO:teuthology.orchestra.run.smithi139.stdout: ceph-mgr-diskprediction-local noarch 2:18.0.0-4940.g81c9387d.el8 ceph-noarch 7.4 M 2023-07-14T19:17:54.722 INFO:teuthology.orchestra.run.smithi139.stdout: ceph-mgr-rook noarch 2:18.0.0-4940.g81c9387d.el8 ceph-noarch 47 k 2023-07-14T19:17:54.723 INFO:teuthology.orchestra.run.smithi139.stdout: ceph-radosgw x86_64 2:18.0.0-4940.g81c9387d.el8 ceph 14 M 2023-07-14T19:17:54.723 INFO:teuthology.orchestra.run.smithi139.stdout: ceph-test x86_64 2:18.0.0-4940.g81c9387d.el8 ceph 48 M 2023-07-14T19:17:54.723 INFO:teuthology.orchestra.run.smithi139.stdout: ceph-volume noarch 2:18.0.0-4940.g81c9387d.el8 ceph-noarch 258 k 2023-07-14T19:17:54.723 INFO:teuthology.orchestra.run.smithi139.stdout: cephadm noarch 2:18.0.0-4940.g81c9387d.el8 ceph-noarch 165 k 2023-07-14T19:17:54.723 INFO:teuthology.orchestra.run.smithi139.stdout: cephfs-mirror x86_64 2:18.0.0-4940.g81c9387d.el8 ceph 260 k 2023-07-14T19:17:54.723 INFO:teuthology.orchestra.run.smithi139.stdout: cephfs-top noarch 2:18.0.0-4940.g81c9387d.el8 ceph-noarch 18 k 2023-07-14T19:17:54.723 INFO:teuthology.orchestra.run.smithi139.stdout: elfutils-libelf-devel x86_64 0.186-1.el8 rhel-8-for-x86_64-baseos-rpms 60 k 2023-07-14T19:17:54.724 INFO:teuthology.orchestra.run.smithi139.stdout: flex x86_64 2.6.1-9.el8 rhel-8-for-x86_64-appstream-rpms 320 k 2023-07-14T19:17:54.724 INFO:teuthology.orchestra.run.smithi139.stdout: libaio-devel x86_64 0.3.112-1.el8 rhel-8-for-x86_64-baseos-rpms 19 k 2023-07-14T19:17:54.724 INFO:teuthology.orchestra.run.smithi139.stdout: libcephfs-devel x86_64 2:18.0.0-4940.g81c9387d.el8 ceph 31 k 2023-07-14T19:17:54.724 INFO:teuthology.orchestra.run.smithi139.stdout: libcephfs2 x86_64 2:18.0.0-4940.g81c9387d.el8 ceph 772 k 2023-07-14T19:17:54.724 INFO:teuthology.orchestra.run.smithi139.stdout: librados-devel x86_64 2:18.0.0-4940.g81c9387d.el8 ceph 139 k 2023-07-14T19:17:54.724 INFO:teuthology.orchestra.run.smithi139.stdout: libtool x86_64 2.4.6-25.el8 rhel-8-for-x86_64-appstream-rpms 709 k 2023-07-14T19:17:54.724 INFO:teuthology.orchestra.run.smithi139.stdout: libuuid-devel x86_64 2.32.1-35.el8 rhel-8-for-x86_64-baseos-rpms 98 k 2023-07-14T19:17:54.725 INFO:teuthology.orchestra.run.smithi139.stdout: openssl-devel x86_64 1:1.1.1k-7.el8_6 rhel-8-for-x86_64-baseos-rpms 2.3 M 2023-07-14T19:17:54.725 INFO:teuthology.orchestra.run.smithi139.stdout: postgresql x86_64 10.21-2.module+el8.6.0+15342+53518fac rhel-8-for-x86_64-appstream-rpms 1.5 M 2023-07-14T19:17:54.725 INFO:teuthology.orchestra.run.smithi139.stdout: postgresql-contrib x86_64 10.21-2.module+el8.6.0+15342+53518fac rhel-8-for-x86_64-appstream-rpms 808 k 2023-07-14T19:17:54.725 INFO:teuthology.orchestra.run.smithi139.stdout: postgresql-server x86_64 10.21-2.module+el8.6.0+15342+53518fac rhel-8-for-x86_64-appstream-rpms 5.1 M 2023-07-14T19:17:54.725 INFO:teuthology.orchestra.run.smithi139.stdout: python3-cephfs x86_64 2:18.0.0-4940.g81c9387d.el8 ceph 197 k 2023-07-14T19:17:54.725 INFO:teuthology.orchestra.run.smithi139.stdout: python3-rados x86_64 2:18.0.0-4940.g81c9387d.el8 ceph 371 k 2023-07-14T19:17:54.726 INFO:teuthology.orchestra.run.smithi139.stdout: python3-rbd x86_64 2:18.0.0-4940.g81c9387d.el8 ceph 364 k 2023-07-14T19:17:54.726 INFO:teuthology.orchestra.run.smithi139.stdout: python3-rgw x86_64 2:18.0.0-4940.g81c9387d.el8 ceph 107 k 2023-07-14T19:17:54.726 INFO:teuthology.orchestra.run.smithi139.stdout: rbd-fuse x86_64 2:18.0.0-4940.g81c9387d.el8 ceph 94 k 2023-07-14T19:17:54.726 INFO:teuthology.orchestra.run.smithi139.stdout: rbd-mirror x86_64 2:18.0.0-4940.g81c9387d.el8 ceph 3.5 M 2023-07-14T19:17:54.726 INFO:teuthology.orchestra.run.smithi139.stdout: rbd-nbd x86_64 2:18.0.0-4940.g81c9387d.el8 ceph 179 k 2023-07-14T19:17:54.726 INFO:teuthology.orchestra.run.smithi139.stdout: xfsprogs-devel x86_64 5.0.0-10.el8 rhel-8-for-x86_64-baseos-rpms 72 k 2023-07-14T19:17:54.726 INFO:teuthology.orchestra.run.smithi139.stdout:Upgrading: 2023-07-14T19:17:54.727 INFO:teuthology.orchestra.run.smithi139.stdout: iproute x86_64 5.15.0-4.el8_6.1 rhel-8-for-x86_64-baseos-rpms 798 k 2023-07-14T19:17:54.727 INFO:teuthology.orchestra.run.smithi139.stdout: librados2 x86_64 2:18.0.0-4940.g81c9387d.el8 ceph 3.7 M 2023-07-14T19:17:54.727 INFO:teuthology.orchestra.run.smithi139.stdout: librbd1 x86_64 2:18.0.0-4940.g81c9387d.el8 ceph 3.6 M 2023-07-14T19:17:54.727 INFO:teuthology.orchestra.run.smithi139.stdout:Installing dependencies: 2023-07-14T19:17:54.727 INFO:teuthology.orchestra.run.smithi139.stdout: automake noarch 1.16.1-7.el8 rhel-8-for-x86_64-appstream-rpms 713 k 2023-07-14T19:17:54.728 INFO:teuthology.orchestra.run.smithi139.stdout: ceph-common x86_64 2:18.0.0-4940.g81c9387d.el8 ceph 22 M 2023-07-14T19:17:54.728 INFO:teuthology.orchestra.run.smithi139.stdout: ceph-grafana-dashboards noarch 2:18.0.0-4940.g81c9387d.el8 ceph-noarch 23 k 2023-07-14T19:17:54.728 INFO:teuthology.orchestra.run.smithi139.stdout: ceph-mds x86_64 2:18.0.0-4940.g81c9387d.el8 ceph 2.3 M 2023-07-14T19:17:54.728 INFO:teuthology.orchestra.run.smithi139.stdout: ceph-mgr-modules-core noarch 2:18.0.0-4940.g81c9387d.el8 ceph-noarch 241 k 2023-07-14T19:17:54.728 INFO:teuthology.orchestra.run.smithi139.stdout: ceph-mon x86_64 2:18.0.0-4940.g81c9387d.el8 ceph 4.6 M 2023-07-14T19:17:54.729 INFO:teuthology.orchestra.run.smithi139.stdout: ceph-osd x86_64 2:18.0.0-4940.g81c9387d.el8 ceph 18 M 2023-07-14T19:17:54.729 INFO:teuthology.orchestra.run.smithi139.stdout: ceph-prometheus-alerts noarch 2:18.0.0-4940.g81c9387d.el8 ceph-noarch 15 k 2023-07-14T19:17:54.729 INFO:teuthology.orchestra.run.smithi139.stdout: ceph-selinux x86_64 2:18.0.0-4940.g81c9387d.el8 ceph 24 k 2023-07-14T19:17:54.729 INFO:teuthology.orchestra.run.smithi139.stdout: fuse x86_64 2.9.7-15.el8 rhel-8-for-x86_64-baseos-rpms 83 k 2023-07-14T19:17:54.729 INFO:teuthology.orchestra.run.smithi139.stdout: gperftools-libs x86_64 1:2.7-9.el8 epel 306 k 2023-07-14T19:17:54.729 INFO:teuthology.orchestra.run.smithi139.stdout: jq x86_64 1.6-3.el8 rhel-8-for-x86_64-appstream-rpms 202 k 2023-07-14T19:17:54.729 INFO:teuthology.orchestra.run.smithi139.stdout: keyutils-libs-devel x86_64 1.5.10-9.el8 rhel-8-for-x86_64-baseos-rpms 48 k 2023-07-14T19:17:54.730 INFO:teuthology.orchestra.run.smithi139.stdout: krb5-devel x86_64 1.18.2-14.el8 rhel-8-for-x86_64-baseos-rpms 560 k 2023-07-14T19:17:54.730 INFO:teuthology.orchestra.run.smithi139.stdout: libcephsqlite x86_64 2:18.0.0-4940.g81c9387d.el8 ceph 135 k 2023-07-14T19:17:54.730 INFO:teuthology.orchestra.run.smithi139.stdout: libcom_err-devel x86_64 1.45.6-4.el8 rhel-8-for-x86_64-baseos-rpms 39 k 2023-07-14T19:17:54.730 INFO:teuthology.orchestra.run.smithi139.stdout: libgfortran x86_64 8.5.0-10.1.el8_6 rhel-8-for-x86_64-baseos-rpms 644 k 2023-07-14T19:17:54.730 INFO:teuthology.orchestra.run.smithi139.stdout: liboath x86_64 2.6.2-3.el8 epel 59 k 2023-07-14T19:17:54.731 INFO:teuthology.orchestra.run.smithi139.stdout: libpq x86_64 13.5-1.el8 rhel-8-for-x86_64-appstream-rpms 198 k 2023-07-14T19:17:54.731 INFO:teuthology.orchestra.run.smithi139.stdout: libquadmath x86_64 8.5.0-10.1.el8_6 rhel-8-for-x86_64-baseos-rpms 171 k 2023-07-14T19:17:54.731 INFO:teuthology.orchestra.run.smithi139.stdout: librabbitmq x86_64 0.9.0-3.el8 rhel-8-for-x86_64-baseos-rpms 47 k 2023-07-14T19:17:54.731 INFO:teuthology.orchestra.run.smithi139.stdout: libradosstriper1 x86_64 2:18.0.0-4940.g81c9387d.el8 ceph 473 k 2023-07-14T19:17:54.731 INFO:teuthology.orchestra.run.smithi139.stdout: librdkafka x86_64 0.11.4-3.el8 rhel-8-for-x86_64-appstream-rpms 354 k 2023-07-14T19:17:54.731 INFO:teuthology.orchestra.run.smithi139.stdout: librgw2 x86_64 2:18.0.0-4940.g81c9387d.el8 ceph 6.8 M 2023-07-14T19:17:54.732 INFO:teuthology.orchestra.run.smithi139.stdout: libselinux-devel x86_64 2.9-5.el8 rhel-8-for-x86_64-baseos-rpms 200 k 2023-07-14T19:17:54.732 INFO:teuthology.orchestra.run.smithi139.stdout: libsepol-devel x86_64 2.9-3.el8 rhel-8-for-x86_64-baseos-rpms 87 k 2023-07-14T19:17:54.732 INFO:teuthology.orchestra.run.smithi139.stdout: libunwind x86_64 1.3.1-3.el8 epel 75 k 2023-07-14T19:17:54.732 INFO:teuthology.orchestra.run.smithi139.stdout: libverto-devel x86_64 0.3.0-5.el8 rhel-8-for-x86_64-baseos-rpms 18 k 2023-07-14T19:17:54.732 INFO:teuthology.orchestra.run.smithi139.stdout: lttng-ust x86_64 2.8.1-11.el8 rhel-8-for-x86_64-appstream-rpms 259 k 2023-07-14T19:17:54.732 INFO:teuthology.orchestra.run.smithi139.stdout: oniguruma x86_64 6.8.2-2.el8 rhel-8-for-x86_64-appstream-rpms 187 k 2023-07-14T19:17:54.732 INFO:teuthology.orchestra.run.smithi139.stdout: openblas x86_64 0.3.15-3.el8 rhel-8-for-x86_64-appstream-rpms 4.7 M 2023-07-14T19:17:54.733 INFO:teuthology.orchestra.run.smithi139.stdout: openblas-threads x86_64 0.3.15-3.el8 rhel-8-for-x86_64-appstream-rpms 4.9 M 2023-07-14T19:17:54.733 INFO:teuthology.orchestra.run.smithi139.stdout: pcre2-devel x86_64 10.32-2.el8 rhel-8-for-x86_64-baseos-rpms 605 k 2023-07-14T19:17:54.733 INFO:teuthology.orchestra.run.smithi139.stdout: pcre2-utf16 x86_64 10.32-2.el8 rhel-8-for-x86_64-baseos-rpms 229 k 2023-07-14T19:17:54.733 INFO:teuthology.orchestra.run.smithi139.stdout: pcre2-utf32 x86_64 10.32-2.el8 rhel-8-for-x86_64-baseos-rpms 220 k 2023-07-14T19:17:54.733 INFO:teuthology.orchestra.run.smithi139.stdout: python3-asyncssh noarch 2.7.0-2.el8 epel 476 k 2023-07-14T19:17:54.733 INFO:teuthology.orchestra.run.smithi139.stdout: python3-babel noarch 2.5.1-7.el8 rhel-8-for-x86_64-appstream-rpms 4.8 M 2023-07-14T19:17:54.733 INFO:teuthology.orchestra.run.smithi139.stdout: python3-bcrypt x86_64 3.1.6-2.el8.1 epel 44 k 2023-07-14T19:17:54.734 INFO:teuthology.orchestra.run.smithi139.stdout: python3-beautifulsoup4 noarch 4.6.3-2.el8.1 epel 185 k 2023-07-14T19:17:54.734 INFO:teuthology.orchestra.run.smithi139.stdout: python3-cachetools noarch 3.1.1-4.el8 epel 33 k 2023-07-14T19:17:54.734 INFO:teuthology.orchestra.run.smithi139.stdout: python3-ceph-argparse x86_64 2:18.0.0-4940.g81c9387d.el8 ceph 45 k 2023-07-14T19:17:54.734 INFO:teuthology.orchestra.run.smithi139.stdout: python3-ceph-common x86_64 2:18.0.0-4940.g81c9387d.el8 ceph 122 k 2023-07-14T19:17:54.734 INFO:teuthology.orchestra.run.smithi139.stdout: python3-certifi noarch 2018.10.15-7.el8 epel 16 k 2023-07-14T19:17:54.734 INFO:teuthology.orchestra.run.smithi139.stdout: python3-cffi x86_64 1.11.5-5.el8 rhel-8-for-x86_64-baseos-rpms 238 k 2023-07-14T19:17:54.734 INFO:teuthology.orchestra.run.smithi139.stdout: python3-cheroot noarch 8.5.2-1.el8 epel 173 k 2023-07-14T19:17:54.735 INFO:teuthology.orchestra.run.smithi139.stdout: python3-cherrypy noarch 18.4.0-1.el8 epel 384 k 2023-07-14T19:17:54.735 INFO:teuthology.orchestra.run.smithi139.stdout: python3-cryptography x86_64 3.2.1-5.el8 rhel-8-for-x86_64-baseos-rpms 559 k 2023-07-14T19:17:54.735 INFO:teuthology.orchestra.run.smithi139.stdout: python3-google-auth noarch 1:1.1.1-10.el8 epel 81 k 2023-07-14T19:17:54.735 INFO:teuthology.orchestra.run.smithi139.stdout: python3-jaraco noarch 6.2-6.el8 epel 11 k 2023-07-14T19:17:54.735 INFO:teuthology.orchestra.run.smithi139.stdout: python3-jaraco-functools noarch 2.0-4.el8 epel 18 k 2023-07-14T19:17:54.735 INFO:teuthology.orchestra.run.smithi139.stdout: python3-jinja2 noarch 2.10.1-3.el8 rhel-8-for-x86_64-appstream-rpms 538 k 2023-07-14T19:17:54.735 INFO:teuthology.orchestra.run.smithi139.stdout: python3-jsonpatch noarch 1.21-2.el8 rhel-8-for-x86_64-appstream-rpms 27 k 2023-07-14T19:17:54.736 INFO:teuthology.orchestra.run.smithi139.stdout: python3-jsonpointer noarch 1.10-11.el8 rhel-8-for-x86_64-appstream-rpms 20 k 2023-07-14T19:17:54.736 INFO:teuthology.orchestra.run.smithi139.stdout: python3-jwt noarch 1.6.1-2.el8 rhel-8-for-x86_64-baseos-rpms 43 k 2023-07-14T19:17:54.736 INFO:teuthology.orchestra.run.smithi139.stdout: python3-kubernetes noarch 1:11.0.0-6.el8 epel 1.2 M 2023-07-14T19:17:54.736 INFO:teuthology.orchestra.run.smithi139.stdout: python3-logutils noarch 0.3.5-11.el8 epel 49 k 2023-07-14T19:17:54.736 INFO:teuthology.orchestra.run.smithi139.stdout: python3-mako noarch 1.0.6-13.el8 rhel-8-for-x86_64-appstream-rpms 157 k 2023-07-14T19:17:54.736 INFO:teuthology.orchestra.run.smithi139.stdout: python3-markupsafe x86_64 0.23-19.el8 rhel-8-for-x86_64-appstream-rpms 39 k 2023-07-14T19:17:54.736 INFO:teuthology.orchestra.run.smithi139.stdout: python3-more-itertools noarch 7.2.0-3.el8 epel 59 k 2023-07-14T19:17:54.737 INFO:teuthology.orchestra.run.smithi139.stdout: python3-natsort noarch 7.1.1-2.el8 epel 60 k 2023-07-14T19:17:54.737 INFO:teuthology.orchestra.run.smithi139.stdout: python3-numpy x86_64 1:1.14.3-10.el8 rhel-8-for-x86_64-appstream-rpms 3.7 M 2023-07-14T19:17:54.737 INFO:teuthology.orchestra.run.smithi139.stdout: python3-numpy-f2py x86_64 1:1.14.3-10.el8 rhel-8-for-x86_64-appstream-rpms 225 k 2023-07-14T19:17:54.737 INFO:teuthology.orchestra.run.smithi139.stdout: python3-oauthlib noarch 2.1.0-1.el8 rhel-8-for-x86_64-baseos-rpms 155 k 2023-07-14T19:17:54.737 INFO:teuthology.orchestra.run.smithi139.stdout: python3-pecan noarch 1.3.2-9.el8 epel 283 k 2023-07-14T19:17:54.737 INFO:teuthology.orchestra.run.smithi139.stdout: python3-portend noarch 2.6-1.el8 epel 16 k 2023-07-14T19:17:54.737 INFO:teuthology.orchestra.run.smithi139.stdout: python3-prettytable noarch 0.7.2-14.el8 rhel-8-for-x86_64-appstream-rpms 44 k 2023-07-14T19:17:54.738 INFO:teuthology.orchestra.run.smithi139.stdout: python3-pyOpenSSL noarch 19.0.0-1.el8 rhel-8-for-x86_64-appstream-rpms 103 k 2023-07-14T19:17:54.738 INFO:teuthology.orchestra.run.smithi139.stdout: python3-pyasn1 noarch 0.3.7-6.el8 rhel-8-for-x86_64-appstream-rpms 126 k 2023-07-14T19:17:54.738 INFO:teuthology.orchestra.run.smithi139.stdout: python3-pyasn1-modules noarch 0.3.7-6.el8 rhel-8-for-x86_64-appstream-rpms 110 k 2023-07-14T19:17:54.738 INFO:teuthology.orchestra.run.smithi139.stdout: python3-pycparser noarch 2.14-14.el8 rhel-8-for-x86_64-baseos-rpms 109 k 2023-07-14T19:17:54.738 INFO:teuthology.orchestra.run.smithi139.stdout: python3-pytz noarch 2017.2-9.el8 rhel-8-for-x86_64-appstream-rpms 54 k 2023-07-14T19:17:54.738 INFO:teuthology.orchestra.run.smithi139.stdout: python3-repoze-lru noarch 0.7-6.el8 epel 34 k 2023-07-14T19:17:54.739 INFO:teuthology.orchestra.run.smithi139.stdout: python3-requests-oauthlib noarch 1.0.0-1.el8 rhel-8-for-x86_64-baseos-rpms 43 k 2023-07-14T19:17:54.739 INFO:teuthology.orchestra.run.smithi139.stdout: python3-routes noarch 2.4.1-12.el8 epel 196 k 2023-07-14T19:17:54.739 INFO:teuthology.orchestra.run.smithi139.stdout: python3-rsa noarch 4.8-1.el8 epel 61 k 2023-07-14T19:17:54.739 INFO:teuthology.orchestra.run.smithi139.stdout: python3-scipy x86_64 1.0.0-21.module+el8.5.0+10916+41bd434d rhel-8-for-x86_64-appstream-rpms 14 M 2023-07-14T19:17:54.739 INFO:teuthology.orchestra.run.smithi139.stdout: python3-simplegeneric noarch 0.8.1-17.el8 epel 19 k 2023-07-14T19:17:54.739 INFO:teuthology.orchestra.run.smithi139.stdout: python3-singledispatch noarch 3.4.0.3-18.el8 epel 24 k 2023-07-14T19:17:54.739 INFO:teuthology.orchestra.run.smithi139.stdout: python3-tempora noarch 1.14.1-5.el8 epel 29 k 2023-07-14T19:17:54.740 INFO:teuthology.orchestra.run.smithi139.stdout: python3-trustme noarch 0.6.0-4.el8 epel 27 k 2023-07-14T19:17:54.740 INFO:teuthology.orchestra.run.smithi139.stdout: python3-waitress noarch 1.4.3-1.el8 epel 255 k 2023-07-14T19:17:54.740 INFO:teuthology.orchestra.run.smithi139.stdout: python3-webob noarch 1.8.5-1.el8.1 epel 251 k 2023-07-14T19:17:54.740 INFO:teuthology.orchestra.run.smithi139.stdout: python3-websocket-client noarch 0.56.0-5.el8 epel 61 k 2023-07-14T19:17:54.740 INFO:teuthology.orchestra.run.smithi139.stdout: python3-webtest noarch 2.0.33-1.el8 epel 85 k 2023-07-14T19:17:54.740 INFO:teuthology.orchestra.run.smithi139.stdout: python3-werkzeug noarch 0.12.2-4.el8 rhel-8-for-x86_64-appstream-rpms 457 k 2023-07-14T19:17:54.740 INFO:teuthology.orchestra.run.smithi139.stdout: python3-zc-lockfile noarch 2.0-2.el8 epel 23 k 2023-07-14T19:17:54.741 INFO:teuthology.orchestra.run.smithi139.stdout: socat x86_64 1.7.4.1-1.el8 rhel-8-for-x86_64-appstream-rpms 323 k 2023-07-14T19:17:54.741 INFO:teuthology.orchestra.run.smithi139.stdout: thrift x86_64 0.13.0-2.el8 epel 1.7 M 2023-07-14T19:17:54.741 INFO:teuthology.orchestra.run.smithi139.stdout: uuid x86_64 1.6.2-43.el8 rhel-8-for-x86_64-appstream-rpms 64 k 2023-07-14T19:17:54.741 INFO:teuthology.orchestra.run.smithi139.stdout: xmlstarlet x86_64 1.6.1-20.el8 lab-extras 70 k 2023-07-14T19:17:54.741 INFO:teuthology.orchestra.run.smithi139.stdout: zlib-devel x86_64 1.2.11-18.el8_5 rhel-8-for-x86_64-baseos-rpms 58 k 2023-07-14T19:17:54.741 INFO:teuthology.orchestra.run.smithi139.stdout:Enabling module streams: 2023-07-14T19:17:54.741 INFO:teuthology.orchestra.run.smithi139.stdout: postgresql 10 2023-07-14T19:17:54.742 INFO:teuthology.orchestra.run.smithi139.stdout: 2023-07-14T19:17:54.742 INFO:teuthology.orchestra.run.smithi139.stdout:Transaction Summary 2023-07-14T19:17:54.742 INFO:teuthology.orchestra.run.smithi139.stdout:=========================================================================================================================== 2023-07-14T19:17:54.742 INFO:teuthology.orchestra.run.smithi139.stdout:Install 128 Packages 2023-07-14T19:17:54.742 INFO:teuthology.orchestra.run.smithi139.stdout:Upgrade 3 Packages 2023-07-14T19:17:54.742 INFO:teuthology.orchestra.run.smithi139.stdout: 2023-07-14T19:17:54.743 INFO:teuthology.orchestra.run.smithi139.stdout:Total download size: 206 M 2023-07-14T19:17:54.743 INFO:teuthology.orchestra.run.smithi139.stdout:Downloading Packages: 2023-07-14T19:17:55.041 INFO:teuthology.orchestra.run.smithi098.stdout:(5/131): ceph-mgr-18.0.0-4940.g81c9387d.el8.x86 204 kB/s | 1.6 MB 00:07 2023-07-14T19:17:56.062 INFO:teuthology.orchestra.run.smithi139.stdout:(1/131): ceph-18.0.0-4940.g81c9387d.el8.x86_64. 13 kB/s | 6.4 kB 00:00 2023-07-14T19:17:56.299 INFO:teuthology.orchestra.run.smithi098.stdout:(6/131): ceph-base-18.0.0-4940.g81c9387d.el8.x8 210 kB/s | 5.4 MB 00:26 2023-07-14T19:17:59.028 INFO:teuthology.orchestra.run.smithi139.stdout:(2/131): ceph-fuse-18.0.0-4940.g81c9387d.el8.x8 287 kB/s | 850 kB 00:02 2023-07-14T19:17:59.278 INFO:teuthology.orchestra.run.smithi139.stdout:(3/131): ceph-immutable-object-cache-18.0.0-494 669 kB/s | 167 kB 00:00 2023-07-14T19:18:01.662 INFO:teuthology.orchestra.run.smithi139.stdout:(4/131): ceph-mds-18.0.0-4940.g81c9387d.el8.x86 986 kB/s | 2.3 MB 00:02 2023-07-14T19:18:03.095 INFO:teuthology.orchestra.run.smithi139.stdout:(5/131): ceph-mgr-18.0.0-4940.g81c9387d.el8.x86 1.1 MB/s | 1.6 MB 00:01 2023-07-14T19:18:03.590 INFO:teuthology.orchestra.run.smithi098.stdout:(7/131): ceph-mon-18.0.0-4940.g81c9387d.el8.x86 553 kB/s | 4.6 MB 00:08 2023-07-14T19:18:03.937 INFO:teuthology.orchestra.run.smithi139.stdout:(6/131): ceph-base-18.0.0-4940.g81c9387d.el8.x8 658 kB/s | 5.4 MB 00:08 2023-07-14T19:18:04.021 INFO:teuthology.orchestra.run.smithi101.stdout:(8/131): ceph-common-18.0.0-4940.g81c9387d.el8. 285 kB/s | 22 MB 01:19 2023-07-14T19:18:04.155 INFO:teuthology.orchestra.run.smithi101.stdout:(9/131): ceph-radosgw-18.0.0-4940.g81c9387d.el8 347 kB/s | 14 MB 00:39 2023-07-14T19:18:04.180 INFO:teuthology.orchestra.run.smithi101.stdout:(10/131): ceph-selinux-18.0.0-4940.g81c9387d.el 153 kB/s | 24 kB 00:00 2023-07-14T19:18:04.764 INFO:teuthology.orchestra.run.smithi101.stdout:(11/131): cephfs-mirror-18.0.0-4940.g81c9387d.e 446 kB/s | 260 kB 00:00 2023-07-14T19:18:04.939 INFO:teuthology.orchestra.run.smithi101.stdout:(12/131): libcephfs-devel-18.0.0-4940.g81c9387d 177 kB/s | 31 kB 00:00 2023-07-14T19:18:07.698 INFO:teuthology.orchestra.run.smithi101.stdout:(13/131): libcephfs2-18.0.0-4940.g81c9387d.el8. 280 kB/s | 772 kB 00:02 2023-07-14T19:18:07.981 INFO:teuthology.orchestra.run.smithi101.stdout:(14/131): libcephsqlite-18.0.0-4940.g81c9387d.e 478 kB/s | 135 kB 00:00 2023-07-14T19:18:08.320 INFO:teuthology.orchestra.run.smithi139.stdout:(7/131): ceph-mon-18.0.0-4940.g81c9387d.el8.x86 905 kB/s | 4.6 MB 00:05 2023-07-14T19:18:08.498 INFO:teuthology.orchestra.run.smithi101.stdout:(15/131): librados-devel-18.0.0-4940.g81c9387d. 269 kB/s | 139 kB 00:00 2023-07-14T19:18:09.565 INFO:teuthology.orchestra.run.smithi101.stdout:(16/131): libradosstriper1-18.0.0-4940.g81c9387 444 kB/s | 473 kB 00:01 2023-07-14T19:18:12.290 INFO:teuthology.orchestra.run.smithi101.stdout:(17/131): ceph-osd-18.0.0-4940.g81c9387d.el8.x8 342 kB/s | 18 MB 00:53 2023-07-14T19:18:12.382 INFO:teuthology.orchestra.run.smithi101.stdout:(18/131): python3-ceph-argparse-18.0.0-4940.g81 488 kB/s | 45 kB 00:00 2023-07-14T19:18:12.533 INFO:teuthology.orchestra.run.smithi101.stdout:(19/131): python3-ceph-common-18.0.0-4940.g81c9 817 kB/s | 122 kB 00:00 2023-07-14T19:18:12.774 INFO:teuthology.orchestra.run.smithi101.stdout:(20/131): python3-cephfs-18.0.0-4940.g81c9387d. 814 kB/s | 197 kB 00:00 2023-07-14T19:18:13.641 INFO:teuthology.orchestra.run.smithi101.stdout:(21/131): python3-rados-18.0.0-4940.g81c9387d.e 428 kB/s | 371 kB 00:00 2023-07-14T19:18:14.759 INFO:teuthology.orchestra.run.smithi101.stdout:(22/131): python3-rbd-18.0.0-4940.g81c9387d.el8 326 kB/s | 364 kB 00:01 2023-07-14T19:18:15.067 INFO:teuthology.orchestra.run.smithi101.stdout:(23/131): python3-rgw-18.0.0-4940.g81c9387d.el8 346 kB/s | 107 kB 00:00 2023-07-14T19:18:15.334 INFO:teuthology.orchestra.run.smithi101.stdout:(24/131): rbd-fuse-18.0.0-4940.g81c9387d.el8.x8 351 kB/s | 94 kB 00:00 2023-07-14T19:18:19.998 INFO:teuthology.orchestra.run.smithi098.stdout:(8/131): ceph-common-18.0.0-4940.g81c9387d.el8. 455 kB/s | 22 MB 00:49 2023-07-14T19:18:20.156 INFO:teuthology.orchestra.run.smithi098.stdout:(9/131): ceph-selinux-18.0.0-4940.g81c9387d.el8 154 kB/s | 24 kB 00:00 2023-07-14T19:18:23.025 INFO:teuthology.orchestra.run.smithi101.stdout:(25/131): rbd-mirror-18.0.0-4940.g81c9387d.el8. 470 kB/s | 3.5 MB 00:07 2023-07-14T19:18:23.309 INFO:teuthology.orchestra.run.smithi101.stdout:(26/131): rbd-nbd-18.0.0-4940.g81c9387d.el8.x86 631 kB/s | 179 kB 00:00 2023-07-14T19:18:23.401 INFO:teuthology.orchestra.run.smithi101.stdout:(27/131): ceph-grafana-dashboards-18.0.0-4940.g 252 kB/s | 23 kB 00:00 2023-07-14T19:18:23.626 INFO:teuthology.orchestra.run.smithi101.stdout:(28/131): ceph-mgr-cephadm-18.0.0-4940.g81c9387 575 kB/s | 129 kB 00:00 2023-07-14T19:18:26.826 INFO:teuthology.orchestra.run.smithi101.stdout:(29/131): librgw2-18.0.0-4940.g81c9387d.el8.x86 402 kB/s | 6.8 MB 00:17 2023-07-14T19:18:26.951 INFO:teuthology.orchestra.run.smithi101.stdout:(30/131): ceph-mgr-dashboard-18.0.0-4940.g81c93 528 kB/s | 1.7 MB 00:03 2023-07-14T19:18:27.468 INFO:teuthology.orchestra.run.smithi101.stdout:(31/131): ceph-mgr-modules-core-18.0.0-4940.g81 466 kB/s | 241 kB 00:00 2023-07-14T19:18:27.610 INFO:teuthology.orchestra.run.smithi101.stdout:(32/131): ceph-mgr-rook-18.0.0-4940.g81c9387d.e 335 kB/s | 47 kB 00:00 2023-07-14T19:18:27.726 INFO:teuthology.orchestra.run.smithi101.stdout:(33/131): ceph-prometheus-alerts-18.0.0-4940.g8 125 kB/s | 15 kB 00:00 2023-07-14T19:18:27.939 INFO:teuthology.orchestra.run.smithi098.stdout:(10/131): ceph-radosgw-18.0.0-4940.g81c9387d.el 569 kB/s | 14 MB 00:24 2023-07-14T19:18:28.177 INFO:teuthology.orchestra.run.smithi101.stdout:(34/131): ceph-volume-18.0.0-4940.g81c9387d.el8 573 kB/s | 258 kB 00:00 2023-07-14T19:18:28.503 INFO:teuthology.orchestra.run.smithi101.stdout:(35/131): cephadm-18.0.0-4940.g81c9387d.el8.noa 509 kB/s | 165 kB 00:00 2023-07-14T19:18:28.628 INFO:teuthology.orchestra.run.smithi101.stdout:(36/131): cephfs-top-18.0.0-4940.g81c9387d.el8. 144 kB/s | 18 kB 00:00 2023-07-14T19:18:28.806 INFO:teuthology.orchestra.run.smithi098.stdout:(11/131): cephfs-mirror-18.0.0-4940.g81c9387d.e 300 kB/s | 260 kB 00:00 2023-07-14T19:18:28.923 INFO:teuthology.orchestra.run.smithi098.stdout:(12/131): libcephfs-devel-18.0.0-4940.g81c9387d 266 kB/s | 31 kB 00:00 2023-07-14T19:18:29.426 INFO:teuthology.orchestra.run.smithi139.stdout:(8/131): ceph-radosgw-18.0.0-4940.g81c9387d.el8 657 kB/s | 14 MB 00:21 2023-07-14T19:18:29.509 INFO:teuthology.orchestra.run.smithi139.stdout:(9/131): ceph-selinux-18.0.0-4940.g81c9387d.el8 292 kB/s | 24 kB 00:00 2023-07-14T19:18:30.037 INFO:teuthology.orchestra.run.smithi101.stdout:(37/131): gperftools-libs-2.7-9.el8.x86_64.rpm 217 kB/s | 306 kB 00:01 2023-07-14T19:18:30.337 INFO:teuthology.orchestra.run.smithi101.stdout:(38/131): liboath-2.6.2-3.el8.x86_64.rpm 196 kB/s | 59 kB 00:00 2023-07-14T19:18:30.448 INFO:teuthology.orchestra.run.smithi098.stdout:(13/131): libcephfs2-18.0.0-4940.g81c9387d.el8. 507 kB/s | 772 kB 00:01 2023-07-14T19:18:30.720 INFO:teuthology.orchestra.run.smithi101.stdout:(39/131): libunwind-1.3.1-3.el8.x86_64.rpm 196 kB/s | 75 kB 00:00 2023-07-14T19:18:30.740 INFO:teuthology.orchestra.run.smithi098.stdout:(14/131): libcephsqlite-18.0.0-4940.g81c9387d.e 464 kB/s | 135 kB 00:00 2023-07-14T19:18:30.982 INFO:teuthology.orchestra.run.smithi098.stdout:(15/131): librados-devel-18.0.0-4940.g81c9387d. 576 kB/s | 139 kB 00:00 2023-07-14T19:18:31.665 INFO:teuthology.orchestra.run.smithi098.stdout:(16/131): libradosstriper1-18.0.0-4940.g81c9387 692 kB/s | 473 kB 00:00 2023-07-14T19:18:32.543 INFO:teuthology.orchestra.run.smithi139.stdout:(10/131): ceph-common-18.0.0-4940.g81c9387d.el8 614 kB/s | 22 MB 00:36 2023-07-14T19:18:32.607 INFO:teuthology.orchestra.run.smithi098.stdout:(17/131): ceph-osd-18.0.0-4940.g81c9387d.el8.x8 501 kB/s | 18 MB 00:36 2023-07-14T19:18:32.720 INFO:teuthology.orchestra.run.smithi101.stdout:(40/131): python3-asyncssh-2.7.0-2.el8.noarch.r 238 kB/s | 476 kB 00:01 2023-07-14T19:18:32.732 INFO:teuthology.orchestra.run.smithi098.stdout:(18/131): python3-ceph-argparse-18.0.0-4940.g81 358 kB/s | 45 kB 00:00 2023-07-14T19:18:32.859 INFO:teuthology.orchestra.run.smithi139.stdout:(11/131): cephfs-mirror-18.0.0-4940.g81c9387d.e 821 kB/s | 260 kB 00:00 2023-07-14T19:18:32.908 INFO:teuthology.orchestra.run.smithi098.stdout:(19/131): python3-ceph-common-18.0.0-4940.g81c9 699 kB/s | 122 kB 00:00 2023-07-14T19:18:32.960 INFO:teuthology.orchestra.run.smithi139.stdout:(12/131): libcephfs-devel-18.0.0-4940.g81c9387d 311 kB/s | 31 kB 00:00 2023-07-14T19:18:33.015 INFO:teuthology.orchestra.run.smithi101.stdout:(41/131): python3-bcrypt-3.1.6-2.el8.1.x86_64.r 148 kB/s | 44 kB 00:00 2023-07-14T19:18:33.258 INFO:teuthology.orchestra.run.smithi098.stdout:(20/131): python3-cephfs-18.0.0-4940.g81c9387d. 562 kB/s | 197 kB 00:00 2023-07-14T19:18:33.957 INFO:teuthology.orchestra.run.smithi101.stdout:(42/131): python3-beautifulsoup4-4.6.3-2.el8.1. 197 kB/s | 185 kB 00:00 2023-07-14T19:18:34.016 INFO:teuthology.orchestra.run.smithi098.stdout:(21/131): python3-rados-18.0.0-4940.g81c9387d.e 489 kB/s | 371 kB 00:00 2023-07-14T19:18:34.165 INFO:teuthology.orchestra.run.smithi101.stdout:(43/131): python3-cachetools-3.1.1-4.el8.noarch 158 kB/s | 33 kB 00:00 2023-07-14T19:18:34.324 INFO:teuthology.orchestra.run.smithi101.stdout:(44/131): python3-certifi-2018.10.15-7.el8.noar 103 kB/s | 16 kB 00:00 2023-07-14T19:18:34.750 INFO:teuthology.orchestra.run.smithi098.stdout:(22/131): python3-rbd-18.0.0-4940.g81c9387d.el8 497 kB/s | 364 kB 00:00 2023-07-14T19:18:35.009 INFO:teuthology.orchestra.run.smithi098.stdout:(23/131): python3-rgw-18.0.0-4940.g81c9387d.el8 413 kB/s | 107 kB 00:00 2023-07-14T19:18:35.051 INFO:teuthology.orchestra.run.smithi139.stdout:(13/131): libcephfs2-18.0.0-4940.g81c9387d.el8. 369 kB/s | 772 kB 00:02 2023-07-14T19:18:35.091 INFO:teuthology.orchestra.run.smithi101.stdout:(45/131): python3-cheroot-8.5.2-1.el8.noarch.rp 225 kB/s | 173 kB 00:00 2023-07-14T19:18:35.209 INFO:teuthology.orchestra.run.smithi098.stdout:(24/131): rbd-fuse-18.0.0-4940.g81c9387d.el8.x8 467 kB/s | 94 kB 00:00 2023-07-14T19:18:35.385 INFO:teuthology.orchestra.run.smithi139.stdout:(14/131): libcephsqlite-18.0.0-4940.g81c9387d.e 406 kB/s | 135 kB 00:00 2023-07-14T19:18:35.769 INFO:teuthology.orchestra.run.smithi139.stdout:(15/131): librados-devel-18.0.0-4940.g81c9387d. 363 kB/s | 139 kB 00:00 2023-07-14T19:18:36.866 INFO:teuthology.orchestra.run.smithi101.stdout:(46/131): python3-cherrypy-18.4.0-1.el8.noarch. 217 kB/s | 384 kB 00:01 2023-07-14T19:18:37.102 INFO:teuthology.orchestra.run.smithi139.stdout:(16/131): libradosstriper1-18.0.0-4940.g81c9387 355 kB/s | 473 kB 00:01 2023-07-14T19:18:37.383 INFO:teuthology.orchestra.run.smithi101.stdout:(47/131): python3-google-auth-1.1.1-10.el8.noar 157 kB/s | 81 kB 00:00 2023-07-14T19:18:37.558 INFO:teuthology.orchestra.run.smithi101.stdout:(48/131): python3-jaraco-6.2-6.el8.noarch.rpm 63 kB/s | 11 kB 00:00 2023-07-14T19:18:37.789 INFO:teuthology.orchestra.run.smithi101.stdout:(49/131): python3-jaraco-functools-2.0-4.el8.no 76 kB/s | 18 kB 00:00 2023-07-14T19:18:41.152 INFO:teuthology.orchestra.run.smithi139.stdout:(17/131): ceph-osd-18.0.0-4940.g81c9387d.el8.x8 489 kB/s | 18 MB 00:37 2023-07-14T19:18:41.718 INFO:teuthology.orchestra.run.smithi139.stdout:(18/131): python3-ceph-argparse-18.0.0-4940.g81 79 kB/s | 45 kB 00:00 2023-07-14T19:18:42.694 INFO:teuthology.orchestra.run.smithi139.stdout:(19/131): python3-ceph-common-18.0.0-4940.g81c9 126 kB/s | 122 kB 00:00 2023-07-14T19:18:43.477 INFO:teuthology.orchestra.run.smithi139.stdout:(20/131): python3-cephfs-18.0.0-4940.g81c9387d. 251 kB/s | 197 kB 00:00 2023-07-14T19:18:44.102 INFO:teuthology.orchestra.run.smithi139.stdout:(21/131): python3-rados-18.0.0-4940.g81c9387d.e 594 kB/s | 371 kB 00:00 2023-07-14T19:18:44.553 INFO:teuthology.orchestra.run.smithi139.stdout:(22/131): python3-rbd-18.0.0-4940.g81c9387d.el8 809 kB/s | 364 kB 00:00 2023-07-14T19:18:44.694 INFO:teuthology.orchestra.run.smithi139.stdout:(23/131): python3-rgw-18.0.0-4940.g81c9387d.el8 753 kB/s | 107 kB 00:00 2023-07-14T19:18:44.870 INFO:teuthology.orchestra.run.smithi139.stdout:(24/131): rbd-fuse-18.0.0-4940.g81c9387d.el8.x8 536 kB/s | 94 kB 00:00 2023-07-14T19:18:45.108 INFO:teuthology.orchestra.run.smithi098.stdout:(25/131): rbd-mirror-18.0.0-4940.g81c9387d.el8. 365 kB/s | 3.5 MB 00:09 2023-07-14T19:18:45.425 INFO:teuthology.orchestra.run.smithi098.stdout:(26/131): rbd-nbd-18.0.0-4940.g81c9387d.el8.x86 565 kB/s | 179 kB 00:00 2023-07-14T19:18:45.501 INFO:teuthology.orchestra.run.smithi098.stdout:(27/131): ceph-grafana-dashboards-18.0.0-4940.g 308 kB/s | 23 kB 00:00 2023-07-14T19:18:45.938 INFO:teuthology.orchestra.run.smithi098.stdout:(28/131): ceph-mgr-cephadm-18.0.0-4940.g81c9387 295 kB/s | 129 kB 00:00 2023-07-14T19:18:46.283 INFO:teuthology.orchestra.run.smithi101.stdout:(50/131): ceph-mgr-diskprediction-local-18.0.0- 390 kB/s | 7.4 MB 00:19 2023-07-14T19:18:47.012 INFO:teuthology.orchestra.run.smithi101.stdout:(51/131): python3-logutils-0.3.5-11.el8.noarch. 67 kB/s | 49 kB 00:00 2023-07-14T19:18:47.044 INFO:teuthology.orchestra.run.smithi139.stdout:(25/131): rbd-mirror-18.0.0-4940.g81c9387d.el8. 1.6 MB/s | 3.5 MB 00:02 2023-07-14T19:18:47.345 INFO:teuthology.orchestra.run.smithi139.stdout:(26/131): rbd-nbd-18.0.0-4940.g81c9387d.el8.x86 596 kB/s | 179 kB 00:00 2023-07-14T19:18:47.520 INFO:teuthology.orchestra.run.smithi139.stdout:(27/131): ceph-grafana-dashboards-18.0.0-4940.g 132 kB/s | 23 kB 00:00 2023-07-14T19:18:47.534 INFO:teuthology.orchestra.run.smithi101.stdout:(52/131): python3-more-itertools-7.2.0-3.el8.no 113 kB/s | 59 kB 00:00 2023-07-14T19:18:47.704 INFO:teuthology.orchestra.run.smithi139.stdout:(28/131): ceph-mgr-cephadm-18.0.0-4940.g81c9387 706 kB/s | 129 kB 00:00 2023-07-14T19:18:48.829 INFO:teuthology.orchestra.run.smithi139.stdout:(29/131): ceph-mgr-dashboard-18.0.0-4940.g81c93 1.5 MB/s | 1.7 MB 00:01 2023-07-14T19:18:49.284 INFO:teuthology.orchestra.run.smithi101.stdout:(53/131): python3-natsort-7.1.1-2.el8.noarch.rp 34 kB/s | 60 kB 00:01 2023-07-14T19:18:51.609 INFO:teuthology.orchestra.run.smithi098.stdout:(29/131): ceph-mgr-dashboard-18.0.0-4940.g81c93 310 kB/s | 1.7 MB 00:05 2023-07-14T19:18:51.801 INFO:teuthology.orchestra.run.smithi098.stdout:(30/131): librgw2-18.0.0-4940.g81c9387d.el8.x86 345 kB/s | 6.8 MB 00:20 2023-07-14T19:18:52.159 INFO:teuthology.orchestra.run.smithi098.stdout:(31/131): ceph-mgr-modules-core-18.0.0-4940.g81 672 kB/s | 241 kB 00:00 2023-07-14T19:18:52.293 INFO:teuthology.orchestra.run.smithi098.stdout:(32/131): ceph-mgr-rook-18.0.0-4940.g81c9387d.e 355 kB/s | 47 kB 00:00 2023-07-14T19:18:52.569 INFO:teuthology.orchestra.run.smithi098.stdout:(33/131): ceph-prometheus-alerts-18.0.0-4940.g8 53 kB/s | 15 kB 00:00 2023-07-14T19:18:53.161 INFO:teuthology.orchestra.run.smithi098.stdout:(34/131): ceph-volume-18.0.0-4940.g81c9387d.el8 437 kB/s | 258 kB 00:00 2023-07-14T19:18:53.627 INFO:teuthology.orchestra.run.smithi098.stdout:(35/131): cephadm-18.0.0-4940.g81c9387d.el8.noa 354 kB/s | 165 kB 00:00 2023-07-14T19:18:53.744 INFO:teuthology.orchestra.run.smithi098.stdout:(36/131): cephfs-top-18.0.0-4940.g81c9387d.el8. 154 kB/s | 18 kB 00:00 2023-07-14T19:18:53.770 INFO:teuthology.orchestra.run.smithi139.stdout:(30/131): librgw2-18.0.0-4940.g81c9387d.el8.x86 417 kB/s | 6.8 MB 00:16 2023-07-14T19:18:54.437 INFO:teuthology.orchestra.run.smithi139.stdout:(31/131): ceph-mgr-modules-core-18.0.0-4940.g81 361 kB/s | 241 kB 00:00 2023-07-14T19:18:54.679 INFO:teuthology.orchestra.run.smithi139.stdout:(32/131): ceph-mgr-rook-18.0.0-4940.g81c9387d.e 196 kB/s | 47 kB 00:00 2023-07-14T19:18:54.779 INFO:teuthology.orchestra.run.smithi139.stdout:(33/131): ceph-prometheus-alerts-18.0.0-4940.g8 145 kB/s | 15 kB 00:00 2023-07-14T19:18:55.303 INFO:teuthology.orchestra.run.smithi098.stdout:(37/131): gperftools-libs-2.7-9.el8.x86_64.rpm 196 kB/s | 306 kB 00:01 2023-07-14T19:18:55.463 INFO:teuthology.orchestra.run.smithi139.stdout:(34/131): ceph-volume-18.0.0-4940.g81c9387d.el8 378 kB/s | 258 kB 00:00 2023-07-14T19:18:55.653 INFO:teuthology.orchestra.run.smithi098.stdout:(38/131): liboath-2.6.2-3.el8.x86_64.rpm 168 kB/s | 59 kB 00:00 2023-07-14T19:18:55.838 INFO:teuthology.orchestra.run.smithi139.stdout:(35/131): cephadm-18.0.0-4940.g81c9387d.el8.noa 441 kB/s | 165 kB 00:00 2023-07-14T19:18:55.938 INFO:teuthology.orchestra.run.smithi139.stdout:(36/131): cephfs-top-18.0.0-4940.g81c9387d.el8. 180 kB/s | 18 kB 00:00 2023-07-14T19:18:55.984 INFO:teuthology.orchestra.run.smithi101.stdout:(54/131): python3-kubernetes-11.0.0-6.el8.noarc 66 kB/s | 1.2 MB 00:18 2023-07-14T19:18:56.192 INFO:teuthology.orchestra.run.smithi101.stdout:(55/131): python3-portend-2.6-1.el8.noarch.rpm 75 kB/s | 16 kB 00:00 2023-07-14T19:18:56.379 INFO:teuthology.orchestra.run.smithi098.stdout:(39/131): libunwind-1.3.1-3.el8.x86_64.rpm 103 kB/s | 75 kB 00:00 2023-07-14T19:18:57.134 INFO:teuthology.orchestra.run.smithi101.stdout:(56/131): python3-repoze-lru-0.7-6.el8.noarch.r 36 kB/s | 34 kB 00:00 2023-07-14T19:18:57.543 INFO:teuthology.orchestra.run.smithi101.stdout:(57/131): python3-pecan-1.3.2-9.el8.noarch.rpm 34 kB/s | 283 kB 00:08 2023-07-14T19:18:58.343 INFO:teuthology.orchestra.run.smithi101.stdout:(58/131): python3-rsa-4.8-1.el8.noarch.rpm 77 kB/s | 61 kB 00:00 2023-07-14T19:18:58.710 INFO:teuthology.orchestra.run.smithi101.stdout:(59/131): python3-simplegeneric-0.8.1-17.el8.no 51 kB/s | 19 kB 00:00 2023-07-14T19:18:59.568 INFO:teuthology.orchestra.run.smithi101.stdout:(60/131): python3-singledispatch-3.4.0.3-18.el8 28 kB/s | 24 kB 00:00 2023-07-14T19:19:00.285 INFO:teuthology.orchestra.run.smithi101.stdout:(61/131): python3-tempora-1.14.1-5.el8.noarch.r 41 kB/s | 29 kB 00:00 2023-07-14T19:19:00.494 INFO:teuthology.orchestra.run.smithi101.stdout:(62/131): python3-routes-2.4.1-12.el8.noarch.rp 58 kB/s | 196 kB 00:03 2023-07-14T19:19:00.620 INFO:teuthology.orchestra.run.smithi101.stdout:(63/131): python3-trustme-0.6.0-4.el8.noarch.rp 82 kB/s | 27 kB 00:00 2023-07-14T19:19:00.704 INFO:teuthology.orchestra.run.smithi139.stdout:(37/131): ceph-mgr-diskprediction-local-18.0.0- 639 kB/s | 7.4 MB 00:11 2023-07-14T19:19:01.097 INFO:teuthology.orchestra.run.smithi139.stdout:(38/131): liboath-2.6.2-3.el8.x86_64.rpm 150 kB/s | 59 kB 00:00 2023-07-14T19:19:01.522 INFO:teuthology.orchestra.run.smithi139.stdout:(39/131): libunwind-1.3.1-3.el8.x86_64.rpm 177 kB/s | 75 kB 00:00 2023-07-14T19:19:02.903 INFO:teuthology.orchestra.run.smithi101.stdout:(64/131): python3-webob-1.8.5-1.el8.1.noarch.rp 110 kB/s | 251 kB 00:02 2023-07-14T19:19:03.697 INFO:teuthology.orchestra.run.smithi139.stdout:(40/131): python3-asyncssh-2.7.0-2.el8.noarch.r 219 kB/s | 476 kB 00:02 2023-07-14T19:19:03.828 INFO:teuthology.orchestra.run.smithi101.stdout:(65/131): python3-websocket-client-0.56.0-5.el8 66 kB/s | 61 kB 00:00 2023-07-14T19:19:03.981 INFO:teuthology.orchestra.run.smithi139.stdout:(41/131): python3-bcrypt-3.1.6-2.el8.1.x86_64.r 154 kB/s | 44 kB 00:00 2023-07-14T19:19:04.595 INFO:teuthology.orchestra.run.smithi098.stdout:(40/131): ceph-mgr-diskprediction-local-18.0.0- 584 kB/s | 7.4 MB 00:12 2023-07-14T19:19:05.029 INFO:teuthology.orchestra.run.smithi101.stdout:(66/131): python3-webtest-2.0.33-1.el8.noarch.r 71 kB/s | 85 kB 00:01 2023-07-14T19:19:05.087 INFO:teuthology.orchestra.run.smithi098.stdout:(41/131): python3-bcrypt-3.1.6-2.el8.1.x86_64.r 89 kB/s | 44 kB 00:00 2023-07-14T19:19:05.579 INFO:teuthology.orchestra.run.smithi101.stdout:(67/131): python3-zc-lockfile-2.0-2.el8.noarch. 42 kB/s | 23 kB 00:00 2023-07-14T19:19:06.037 INFO:teuthology.orchestra.run.smithi098.stdout:(42/131): python3-beautifulsoup4-4.6.3-2.el8.1. 195 kB/s | 185 kB 00:00 2023-07-14T19:19:06.362 INFO:teuthology.orchestra.run.smithi098.stdout:(43/131): python3-cachetools-3.1.1-4.el8.noarch 101 kB/s | 33 kB 00:00 2023-07-14T19:19:06.546 INFO:teuthology.orchestra.run.smithi101.stdout:(68/131): python3-waitress-1.4.3-1.el8.noarch.r 42 kB/s | 255 kB 00:06 2023-07-14T19:19:06.554 INFO:teuthology.orchestra.run.smithi098.stdout:(44/131): python3-certifi-2018.10.15-7.el8.noar 85 kB/s | 16 kB 00:00 2023-07-14T19:19:06.571 INFO:teuthology.orchestra.run.smithi101.stdout:(69/131): xmlstarlet-1.6.1-20.el8.x86_64.rpm 2.7 MB/s | 70 kB 00:00 2023-07-14T19:19:06.621 INFO:teuthology.orchestra.run.smithi101.stdout:(70/131): librabbitmq-0.9.0-3.el8.x86_64.rpm 935 kB/s | 47 kB 00:00 2023-07-14T19:19:06.689 INFO:teuthology.orchestra.run.smithi101.stdout:(71/131): libselinux-devel-2.9-5.el8.x86_64.rpm 2.9 MB/s | 200 kB 00:00 2023-07-14T19:19:06.722 INFO:teuthology.orchestra.run.smithi101.stdout:(72/131): python3-jwt-1.6.1-2.el8.noarch.rpm 1.3 MB/s | 43 kB 00:00 2023-07-14T19:19:06.764 INFO:teuthology.orchestra.run.smithi101.stdout:(73/131): pcre2-utf32-10.32-2.el8.x86_64.rpm 5.2 MB/s | 220 kB 00:00 2023-07-14T19:19:06.798 INFO:teuthology.orchestra.run.smithi101.stdout:(74/131): keyutils-libs-devel-1.5.10-9.el8.x86_ 1.4 MB/s | 48 kB 00:00 2023-07-14T19:19:06.831 INFO:teuthology.orchestra.run.smithi101.stdout:(75/131): libsepol-devel-2.9-3.el8.x86_64.rpm 2.5 MB/s | 87 kB 00:00 2023-07-14T19:19:06.857 INFO:teuthology.orchestra.run.smithi101.stdout:(76/131): python3-requests-oauthlib-1.0.0-1.el8 1.7 MB/s | 43 kB 00:00 2023-07-14T19:19:06.890 INFO:teuthology.orchestra.run.smithi101.stdout:(77/131): python3-pycparser-2.14-14.el8.noarch. 3.2 MB/s | 109 kB 00:00 2023-07-14T19:19:06.933 INFO:teuthology.orchestra.run.smithi101.stdout:(78/131): pcre2-utf16-10.32-2.el8.x86_64.rpm 5.4 MB/s | 229 kB 00:00 2023-07-14T19:19:06.938 INFO:teuthology.orchestra.run.smithi139.stdout:(42/131): gperftools-libs-2.7-9.el8.x86_64.rpm 28 kB/s | 306 kB 00:10 2023-07-14T19:19:06.975 INFO:teuthology.orchestra.run.smithi101.stdout:(79/131): krb5-devel-1.18.2-14.el8.x86_64.rpm 13 MB/s | 560 kB 00:00 2023-07-14T19:19:07.000 INFO:teuthology.orchestra.run.smithi101.stdout:(80/131): libaio-devel-0.3.112-1.el8.x86_64.rpm 749 kB/s | 19 kB 00:00 2023-07-14T19:19:07.076 INFO:teuthology.orchestra.run.smithi101.stdout:(81/131): python3-oauthlib-2.1.0-1.el8.noarch.r 2.0 MB/s | 155 kB 00:00 2023-07-14T19:19:07.125 INFO:teuthology.orchestra.run.smithi101.stdout:(82/131): python3-cryptography-3.2.1-5.el8.x86_ 11 MB/s | 559 kB 00:00 2023-07-14T19:19:07.159 INFO:teuthology.orchestra.run.smithi101.stdout:(83/131): libverto-devel-0.3.0-5.el8.x86_64.rpm 535 kB/s | 18 kB 00:00 2023-07-14T19:19:07.201 INFO:teuthology.orchestra.run.smithi101.stdout:(84/131): pcre2-devel-10.32-2.el8.x86_64.rpm 14 MB/s | 605 kB 00:00 2023-07-14T19:19:07.243 INFO:teuthology.orchestra.run.smithi101.stdout:(85/131): python3-cffi-1.11.5-5.el8.x86_64.rpm 5.6 MB/s | 238 kB 00:00 2023-07-14T19:19:07.304 INFO:teuthology.orchestra.run.smithi098.stdout:(45/131): python3-asyncssh-2.7.0-2.el8.noarch.r 44 kB/s | 476 kB 00:10 2023-07-14T19:19:07.327 INFO:teuthology.orchestra.run.smithi101.stdout:(86/131): openssl-devel-1.1.1k-7.el8_6.x86_64.r 28 MB/s | 2.3 MB 00:00 2023-07-14T19:19:07.369 INFO:teuthology.orchestra.run.smithi101.stdout:(87/131): xfsprogs-devel-5.0.0-10.el8.x86_64.rp 1.7 MB/s | 72 kB 00:00 2023-07-14T19:19:07.403 INFO:teuthology.orchestra.run.smithi101.stdout:(88/131): fuse-2.9.7-15.el8.x86_64.rpm 2.4 MB/s | 83 kB 00:00 2023-07-14T19:19:07.404 INFO:teuthology.orchestra.run.smithi098.stdout:(46/131): python3-cheroot-8.5.2-1.el8.noarch.rp 203 kB/s | 173 kB 00:00 2023-07-14T19:19:07.436 INFO:teuthology.orchestra.run.smithi101.stdout:(89/131): elfutils-libelf-devel-0.186-1.el8.x86 1.8 MB/s | 60 kB 00:00 2023-07-14T19:19:07.495 INFO:teuthology.orchestra.run.smithi101.stdout:(90/131): libuuid-devel-2.32.1-35.el8.x86_64.rp 1.6 MB/s | 98 kB 00:00 2023-07-14T19:19:07.529 INFO:teuthology.orchestra.run.smithi101.stdout:(91/131): zlib-devel-1.2.11-18.el8_5.x86_64.rpm 1.7 MB/s | 58 kB 00:00 2023-07-14T19:19:07.547 INFO:teuthology.orchestra.run.smithi139.stdout:(43/131): python3-cachetools-3.1.1-4.el8.noarch 54 kB/s | 33 kB 00:00 2023-07-14T19:19:07.563 INFO:teuthology.orchestra.run.smithi101.stdout:(92/131): libcom_err-devel-1.45.6-4.el8.x86_64. 1.1 MB/s | 39 kB 00:00 2023-07-14T19:19:07.605 INFO:teuthology.orchestra.run.smithi101.stdout:(93/131): libgfortran-8.5.0-10.1.el8_6.x86_64.r 15 MB/s | 644 kB 00:00 2023-07-14T19:19:07.647 INFO:teuthology.orchestra.run.smithi101.stdout:(94/131): libquadmath-8.5.0-10.1.el8_6.x86_64.r 4.0 MB/s | 171 kB 00:00 2023-07-14T19:19:07.680 INFO:teuthology.orchestra.run.smithi101.stdout:(95/131): NetworkManager-initscripts-updown-1.3 4.0 MB/s | 138 kB 00:00 2023-07-14T19:19:07.997 INFO:teuthology.orchestra.run.smithi101.stdout:(96/131): python3-scipy-1.0.0-21.module+el8.5.0 43 MB/s | 14 MB 00:00 2023-07-14T19:19:08.048 INFO:teuthology.orchestra.run.smithi101.stdout:(97/131): python3-pyasn1-modules-0.3.7-6.el8.no 2.2 MB/s | 110 kB 00:00 2023-07-14T19:19:08.064 INFO:teuthology.orchestra.run.smithi139.stdout:(44/131): python3-certifi-2018.10.15-7.el8.noar 32 kB/s | 16 kB 00:00 2023-07-14T19:19:08.089 INFO:teuthology.orchestra.run.smithi139.stdout:(45/131): python3-beautifulsoup4-4.6.3-2.el8.1. 45 kB/s | 185 kB 00:04 2023-07-14T19:19:08.098 INFO:teuthology.orchestra.run.smithi101.stdout:(98/131): python3-pytz-2017.2-9.el8.noarch.rpm 1.1 MB/s | 54 kB 00:00 2023-07-14T19:19:08.148 INFO:teuthology.orchestra.run.smithi101.stdout:(99/131): lttng-ust-2.8.1-11.el8.x86_64.rpm 5.1 MB/s | 259 kB 00:00 2023-07-14T19:19:08.199 INFO:teuthology.orchestra.run.smithi101.stdout:(100/131): socat-1.7.4.1-1.el8.x86_64.rpm 6.3 MB/s | 323 kB 00:00 2023-07-14T19:19:08.249 INFO:teuthology.orchestra.run.smithi101.stdout:(101/131): python3-markupsafe-0.23-19.el8.x86_6 776 kB/s | 39 kB 00:00 2023-07-14T19:19:08.308 INFO:teuthology.orchestra.run.smithi101.stdout:(102/131): bison-3.0.4-10.el8.x86_64.rpm 12 MB/s | 688 kB 00:00 2023-07-14T19:19:08.358 INFO:teuthology.orchestra.run.smithi101.stdout:(103/131): python3-jinja2-2.10.1-3.el8.noarch.r 11 MB/s | 538 kB 00:00 2023-07-14T19:19:08.408 INFO:teuthology.orchestra.run.smithi101.stdout:(104/131): python3-mako-1.0.6-13.el8.noarch.rpm 3.1 MB/s | 157 kB 00:00 2023-07-14T19:19:08.459 INFO:teuthology.orchestra.run.smithi101.stdout:(105/131): oniguruma-6.8.2-2.el8.x86_64.rpm 3.6 MB/s | 187 kB 00:00 2023-07-14T19:19:08.480 INFO:teuthology.orchestra.run.smithi098.stdout:(47/131): python3-google-auth-1.1.1-10.el8.noar 76 kB/s | 81 kB 00:01 2023-07-14T19:19:08.525 INFO:teuthology.orchestra.run.smithi101.stdout:(106/131): automake-1.16.1-7.el8.noarch.rpm 10 MB/s | 713 kB 00:00 2023-07-14T19:19:08.630 INFO:teuthology.orchestra.run.smithi098.stdout:(48/131): python3-jaraco-6.2-6.el8.noarch.rpm 74 kB/s | 11 kB 00:00 2023-07-14T19:19:08.651 INFO:teuthology.orchestra.run.smithi101.stdout:(107/131): python3-babel-2.5.1-7.el8.noarch.rpm 38 MB/s | 4.8 MB 00:00 2023-07-14T19:19:08.701 INFO:teuthology.orchestra.run.smithi101.stdout:(108/131): python3-jsonpatch-1.21-2.el8.noarch. 533 kB/s | 27 kB 00:00 2023-07-14T19:19:08.751 INFO:teuthology.orchestra.run.smithi101.stdout:(109/131): flex-2.6.1-9.el8.x86_64.rpm 6.2 MB/s | 320 kB 00:00 2023-07-14T19:19:08.802 INFO:teuthology.orchestra.run.smithi101.stdout:(110/131): python3-prettytable-0.7.2-14.el8.noa 877 kB/s | 44 kB 00:00 2023-07-14T19:19:08.902 INFO:teuthology.orchestra.run.smithi101.stdout:(111/131): python3-numpy-1.14.3-10.el8.x86_64.r 37 MB/s | 3.7 MB 00:00 2023-07-14T19:19:08.953 INFO:teuthology.orchestra.run.smithi101.stdout:(112/131): python3-werkzeug-0.12.2-4.el8.noarch 8.9 MB/s | 457 kB 00:00 2023-07-14T19:19:09.003 INFO:teuthology.orchestra.run.smithi101.stdout:(113/131): python3-numpy-f2py-1.14.3-10.el8.x86 4.4 MB/s | 225 kB 00:00 2023-07-14T19:19:09.053 INFO:teuthology.orchestra.run.smithi101.stdout:(114/131): uuid-1.6.2-43.el8.x86_64.rpm 1.2 MB/s | 64 kB 00:00 2023-07-14T19:19:09.113 INFO:teuthology.orchestra.run.smithi101.stdout:(115/131): python3-pyasn1-0.3.7-6.el8.noarch.rp 2.1 MB/s | 126 kB 00:00 2023-07-14T19:19:09.163 INFO:teuthology.orchestra.run.smithi101.stdout:(116/131): python3-jsonpointer-1.10-11.el8.noar 399 kB/s | 20 kB 00:00 2023-07-14T19:19:09.213 INFO:teuthology.orchestra.run.smithi101.stdout:(117/131): python3-pyOpenSSL-19.0.0-1.el8.noarc 2.0 MB/s | 103 kB 00:00 2023-07-14T19:19:09.280 INFO:teuthology.orchestra.run.smithi101.stdout:(118/131): libtool-2.4.6-25.el8.x86_64.rpm 10 MB/s | 709 kB 00:00 2023-07-14T19:19:09.414 INFO:teuthology.orchestra.run.smithi101.stdout:(119/131): openblas-threads-0.3.15-3.el8.x86_64 36 MB/s | 4.9 MB 00:00 2023-07-14T19:19:09.456 INFO:teuthology.orchestra.run.smithi101.stdout:(120/131): libpq-13.5-1.el8.x86_64.rpm 4.7 MB/s | 198 kB 00:00 2023-07-14T19:19:09.489 INFO:teuthology.orchestra.run.smithi098.stdout:(49/131): python3-jaraco-functools-2.0-4.el8.no 20 kB/s | 18 kB 00:00 2023-07-14T19:19:09.589 INFO:teuthology.orchestra.run.smithi101.stdout:(121/131): openblas-0.3.15-3.el8.x86_64.rpm 36 MB/s | 4.7 MB 00:00 2023-07-14T19:19:09.639 INFO:teuthology.orchestra.run.smithi101.stdout:(122/131): jq-1.6-3.el8.x86_64.rpm 4.0 MB/s | 202 kB 00:00 2023-07-14T19:19:09.690 INFO:teuthology.orchestra.run.smithi101.stdout:(123/131): librdkafka-0.11.4-3.el8.x86_64.rpm 6.9 MB/s | 354 kB 00:00 2023-07-14T19:19:09.823 INFO:teuthology.orchestra.run.smithi101.stdout:(124/131): postgresql-server-10.21-2.module+el8 38 MB/s | 5.1 MB 00:00 2023-07-14T19:19:09.924 INFO:teuthology.orchestra.run.smithi101.stdout:(125/131): postgresql-contrib-10.21-2.module+el 7.9 MB/s | 808 kB 00:00 2023-07-14T19:19:09.999 INFO:teuthology.orchestra.run.smithi101.stdout:(126/131): postgresql-10.21-2.module+el8.6.0+15 20 MB/s | 1.5 MB 00:00 2023-07-14T19:19:11.297 INFO:teuthology.orchestra.run.smithi139.stdout:(46/131): python3-cheroot-8.5.2-1.el8.noarch.rp 53 kB/s | 173 kB 00:03 2023-07-14T19:19:13.964 INFO:teuthology.orchestra.run.smithi139.stdout:(47/131): python3-google-auth-1.1.1-10.el8.noar 30 kB/s | 81 kB 00:02 2023-07-14T19:19:14.022 INFO:teuthology.orchestra.run.smithi139.stdout:(48/131): python3-cherrypy-18.4.0-1.el8.noarch. 65 kB/s | 384 kB 00:05 2023-07-14T19:19:14.223 INFO:teuthology.orchestra.run.smithi139.stdout:(49/131): python3-jaraco-6.2-6.el8.noarch.rpm 43 kB/s | 11 kB 00:00 2023-07-14T19:19:14.540 INFO:teuthology.orchestra.run.smithi139.stdout:(50/131): python3-jaraco-functools-2.0-4.el8.no 34 kB/s | 18 kB 00:00 2023-07-14T19:19:14.883 INFO:teuthology.orchestra.run.smithi101.stdout:(127/131): librados2-18.0.0-4940.g81c9387d.el8. 772 kB/s | 3.7 MB 00:04 2023-07-14T19:19:15.373 INFO:teuthology.orchestra.run.smithi139.stdout:(51/131): python3-logutils-0.3.5-11.el8.noarch. 58 kB/s | 49 kB 00:00 2023-07-14T19:19:15.472 INFO:teuthology.orchestra.run.smithi098.stdout:(50/131): python3-cherrypy-18.4.0-1.el8.noarch. 47 kB/s | 384 kB 00:08 2023-07-14T19:19:16.824 INFO:teuthology.orchestra.run.smithi139.stdout:(52/131): python3-more-itertools-7.2.0-3.el8.no 40 kB/s | 59 kB 00:01 2023-07-14T19:19:17.480 INFO:teuthology.orchestra.run.smithi098.stdout:(51/131): python3-logutils-0.3.5-11.el8.noarch. 24 kB/s | 49 kB 00:02 2023-07-14T19:19:18.148 INFO:teuthology.orchestra.run.smithi139.stdout:(53/131): python3-natsort-7.1.1-2.el8.noarch.rp 45 kB/s | 60 kB 00:01 2023-07-14T19:19:18.224 INFO:teuthology.orchestra.run.smithi101.stdout:(128/131): librbd1-18.0.0-4940.g81c9387d.el8.x8 1.1 MB/s | 3.6 MB 00:03 2023-07-14T19:19:18.274 INFO:teuthology.orchestra.run.smithi101.stdout:(129/131): iproute-5.15.0-4.el8_6.1.x86_64.rpm 16 MB/s | 798 kB 00:00 2023-07-14T19:19:18.814 INFO:teuthology.orchestra.run.smithi098.stdout:(52/131): python3-more-itertools-7.2.0-3.el8.no 44 kB/s | 59 kB 00:01 2023-07-14T19:19:19.789 INFO:teuthology.orchestra.run.smithi098.stdout:(53/131): python3-natsort-7.1.1-2.el8.noarch.rp 62 kB/s | 60 kB 00:00 2023-07-14T19:19:22.539 INFO:teuthology.orchestra.run.smithi098.stdout:(54/131): python3-pecan-1.3.2-9.el8.noarch.rpm 103 kB/s | 283 kB 00:02 2023-07-14T19:19:22.706 INFO:teuthology.orchestra.run.smithi098.stdout:(55/131): python3-portend-2.6-1.el8.noarch.rpm 93 kB/s | 16 kB 00:00 2023-07-14T19:19:23.073 INFO:teuthology.orchestra.run.smithi098.stdout:(56/131): python3-repoze-lru-0.7-6.el8.noarch.r 92 kB/s | 34 kB 00:00 2023-07-14T19:19:24.449 INFO:teuthology.orchestra.run.smithi139.stdout:(54/131): python3-pecan-1.3.2-9.el8.noarch.rpm 45 kB/s | 283 kB 00:06 2023-07-14T19:19:25.606 INFO:teuthology.orchestra.run.smithi139.stdout:(55/131): python3-portend-2.6-1.el8.noarch.rpm 13 kB/s | 16 kB 00:01 2023-07-14T19:19:25.957 INFO:teuthology.orchestra.run.smithi098.stdout:(57/131): python3-routes-2.4.1-12.el8.noarch.rp 68 kB/s | 196 kB 00:02 2023-07-14T19:19:26.006 INFO:teuthology.orchestra.run.smithi139.stdout:(56/131): python3-repoze-lru-0.7-6.el8.noarch.r 84 kB/s | 34 kB 00:00 2023-07-14T19:19:26.607 INFO:teuthology.orchestra.run.smithi098.stdout:(58/131): python3-rsa-4.8-1.el8.noarch.rpm 95 kB/s | 61 kB 00:00 2023-07-14T19:19:26.907 INFO:teuthology.orchestra.run.smithi098.stdout:(59/131): python3-simplegeneric-0.8.1-17.el8.no 62 kB/s | 19 kB 00:00 2023-07-14T19:19:27.282 INFO:teuthology.orchestra.run.smithi098.stdout:(60/131): python3-singledispatch-3.4.0.3-18.el8 64 kB/s | 24 kB 00:00 2023-07-14T19:19:27.674 INFO:teuthology.orchestra.run.smithi098.stdout:(61/131): python3-tempora-1.14.1-5.el8.noarch.r 75 kB/s | 29 kB 00:00 2023-07-14T19:19:27.689 INFO:teuthology.orchestra.run.smithi139.stdout:(57/131): python3-routes-2.4.1-12.el8.noarch.rp 117 kB/s | 196 kB 00:01 2023-07-14T19:19:28.049 INFO:teuthology.orchestra.run.smithi098.stdout:(62/131): python3-trustme-0.6.0-4.el8.noarch.rp 73 kB/s | 27 kB 00:00 2023-07-14T19:19:28.165 INFO:teuthology.orchestra.run.smithi139.stdout:(58/131): python3-rsa-4.8-1.el8.noarch.rpm 129 kB/s | 61 kB 00:00 2023-07-14T19:19:28.398 INFO:teuthology.orchestra.run.smithi139.stdout:(59/131): python3-simplegeneric-0.8.1-17.el8.no 80 kB/s | 19 kB 00:00 2023-07-14T19:19:28.615 INFO:teuthology.orchestra.run.smithi139.stdout:(60/131): python3-singledispatch-3.4.0.3-18.el8 110 kB/s | 24 kB 00:00 2023-07-14T19:19:28.873 INFO:teuthology.orchestra.run.smithi139.stdout:(61/131): python3-tempora-1.14.1-5.el8.noarch.r 113 kB/s | 29 kB 00:00 2023-07-14T19:19:29.132 INFO:teuthology.orchestra.run.smithi139.stdout:(62/131): python3-trustme-0.6.0-4.el8.noarch.rp 106 kB/s | 27 kB 00:00 2023-07-14T19:19:29.691 INFO:teuthology.orchestra.run.smithi098.stdout:(63/131): python3-waitress-1.4.3-1.el8.noarch.r 156 kB/s | 255 kB 00:01 2023-07-14T19:19:29.808 INFO:teuthology.orchestra.run.smithi098.stdout:(64/131): python3-kubernetes-11.0.0-6.el8.noarc 59 kB/s | 1.2 MB 00:20 2023-07-14T19:19:30.202 INFO:teuthology.orchestra.run.smithi098.stdout:(65/131): python3-websocket-client-0.56.0-5.el8 156 kB/s | 61 kB 00:00 2023-07-14T19:19:30.333 INFO:teuthology.orchestra.run.smithi139.stdout:(63/131): python3-waitress-1.4.3-1.el8.noarch.r 213 kB/s | 255 kB 00:01 2023-07-14T19:19:30.634 INFO:teuthology.orchestra.run.smithi098.stdout:(66/131): python3-webtest-2.0.33-1.el8.noarch.r 197 kB/s | 85 kB 00:00 2023-07-14T19:19:30.849 INFO:teuthology.orchestra.run.smithi139.stdout:(64/131): python3-kubernetes-11.0.0-6.el8.noarc 72 kB/s | 1.2 MB 00:16 2023-07-14T19:19:30.851 INFO:teuthology.orchestra.run.smithi098.stdout:(67/131): python3-zc-lockfile-2.0-2.el8.noarch. 106 kB/s | 23 kB 00:00 2023-07-14T19:19:30.951 INFO:teuthology.orchestra.run.smithi098.stdout:(68/131): python3-webob-1.8.5-1.el8.1.noarch.rp 199 kB/s | 251 kB 00:01 2023-07-14T19:19:30.976 INFO:teuthology.orchestra.run.smithi098.stdout:(69/131): xmlstarlet-1.6.1-20.el8.x86_64.rpm 2.7 MB/s | 70 kB 00:00 2023-07-14T19:19:31.027 INFO:teuthology.orchestra.run.smithi098.stdout:(70/131): librabbitmq-0.9.0-3.el8.x86_64.rpm 935 kB/s | 47 kB 00:00 2023-07-14T19:19:31.069 INFO:teuthology.orchestra.run.smithi098.stdout:(71/131): libselinux-devel-2.9-5.el8.x86_64.rpm 4.7 MB/s | 200 kB 00:00 2023-07-14T19:19:31.102 INFO:teuthology.orchestra.run.smithi098.stdout:(72/131): python3-jwt-1.6.1-2.el8.noarch.rpm 1.2 MB/s | 43 kB 00:00 2023-07-14T19:19:31.145 INFO:teuthology.orchestra.run.smithi098.stdout:(73/131): pcre2-utf32-10.32-2.el8.x86_64.rpm 5.2 MB/s | 220 kB 00:00 2023-07-14T19:19:31.178 INFO:teuthology.orchestra.run.smithi098.stdout:(74/131): keyutils-libs-devel-1.5.10-9.el8.x86_ 1.4 MB/s | 48 kB 00:00 2023-07-14T19:19:31.212 INFO:teuthology.orchestra.run.smithi098.stdout:(75/131): libsepol-devel-2.9-3.el8.x86_64.rpm 2.5 MB/s | 87 kB 00:00 2023-07-14T19:19:31.245 INFO:teuthology.orchestra.run.smithi098.stdout:(76/131): python3-requests-oauthlib-1.0.0-1.el8 1.3 MB/s | 43 kB 00:00 2023-07-14T19:19:31.250 INFO:teuthology.orchestra.run.smithi139.stdout:(65/131): python3-websocket-client-0.56.0-5.el8 153 kB/s | 61 kB 00:00 2023-07-14T19:19:31.287 INFO:teuthology.orchestra.run.smithi098.stdout:(77/131): python3-pycparser-2.14-14.el8.noarch. 2.6 MB/s | 109 kB 00:00 2023-07-14T19:19:31.329 INFO:teuthology.orchestra.run.smithi098.stdout:(78/131): pcre2-utf16-10.32-2.el8.x86_64.rpm 5.4 MB/s | 229 kB 00:00 2023-07-14T19:19:31.397 INFO:teuthology.orchestra.run.smithi098.stdout:(79/131): krb5-devel-1.18.2-14.el8.x86_64.rpm 8.2 MB/s | 560 kB 00:00 2023-07-14T19:19:31.417 INFO:teuthology.orchestra.run.smithi139.stdout:(66/131): python3-webob-1.8.5-1.el8.1.noarch.rp 232 kB/s | 251 kB 00:01 2023-07-14T19:19:31.430 INFO:teuthology.orchestra.run.smithi098.stdout:(80/131): libaio-devel-0.3.112-1.el8.x86_64.rpm 561 kB/s | 19 kB 00:00 2023-07-14T19:19:31.472 INFO:teuthology.orchestra.run.smithi098.stdout:(81/131): python3-oauthlib-2.1.0-1.el8.noarch.r 3.6 MB/s | 155 kB 00:00 2023-07-14T19:19:31.514 INFO:teuthology.orchestra.run.smithi098.stdout:(82/131): python3-cryptography-3.2.1-5.el8.x86_ 13 MB/s | 559 kB 00:00 2023-07-14T19:19:31.548 INFO:teuthology.orchestra.run.smithi098.stdout:(83/131): libverto-devel-0.3.0-5.el8.x86_64.rpm 534 kB/s | 18 kB 00:00 2023-07-14T19:19:31.590 INFO:teuthology.orchestra.run.smithi098.stdout:(84/131): pcre2-devel-10.32-2.el8.x86_64.rpm 14 MB/s | 605 kB 00:00 2023-07-14T19:19:31.632 INFO:teuthology.orchestra.run.smithi098.stdout:(85/131): python3-cffi-1.11.5-5.el8.x86_64.rpm 5.6 MB/s | 238 kB 00:00 2023-07-14T19:19:31.642 INFO:teuthology.orchestra.run.smithi139.stdout:(67/131): python3-zc-lockfile-2.0-2.el8.noarch. 102 kB/s | 23 kB 00:00 2023-07-14T19:19:31.707 INFO:teuthology.orchestra.run.smithi098.stdout:(86/131): openssl-devel-1.1.1k-7.el8_6.x86_64.r 31 MB/s | 2.3 MB 00:00 2023-07-14T19:19:31.749 INFO:teuthology.orchestra.run.smithi098.stdout:(87/131): xfsprogs-devel-5.0.0-10.el8.x86_64.rp 1.7 MB/s | 72 kB 00:00 2023-07-14T19:19:31.776 INFO:teuthology.orchestra.run.smithi139.stdout:(68/131): python3-webtest-2.0.33-1.el8.noarch.r 162 kB/s | 85 kB 00:00 2023-07-14T19:19:31.783 INFO:teuthology.orchestra.run.smithi098.stdout:(88/131): fuse-2.9.7-15.el8.x86_64.rpm 2.4 MB/s | 83 kB 00:00 2023-07-14T19:19:31.809 INFO:teuthology.orchestra.run.smithi139.stdout:(69/131): xmlstarlet-1.6.1-20.el8.x86_64.rpm 2.0 MB/s | 70 kB 00:00 2023-07-14T19:19:31.817 INFO:teuthology.orchestra.run.smithi098.stdout:(89/131): elfutils-libelf-devel-0.186-1.el8.x86 1.8 MB/s | 60 kB 00:00 2023-07-14T19:19:31.850 INFO:teuthology.orchestra.run.smithi098.stdout:(90/131): libuuid-devel-2.32.1-35.el8.x86_64.rp 2.9 MB/s | 98 kB 00:00 2023-07-14T19:19:31.860 INFO:teuthology.orchestra.run.smithi139.stdout:(70/131): librabbitmq-0.9.0-3.el8.x86_64.rpm 941 kB/s | 47 kB 00:00 2023-07-14T19:19:31.884 INFO:teuthology.orchestra.run.smithi098.stdout:(91/131): zlib-devel-1.2.11-18.el8_5.x86_64.rpm 1.7 MB/s | 58 kB 00:00 2023-07-14T19:19:31.901 INFO:teuthology.orchestra.run.smithi139.stdout:(71/131): libselinux-devel-2.9-5.el8.x86_64.rpm 4.7 MB/s | 200 kB 00:00 2023-07-14T19:19:31.918 INFO:teuthology.orchestra.run.smithi098.stdout:(92/131): libcom_err-devel-1.45.6-4.el8.x86_64. 1.1 MB/s | 39 kB 00:00 2023-07-14T19:19:31.935 INFO:teuthology.orchestra.run.smithi139.stdout:(72/131): python3-jwt-1.6.1-2.el8.noarch.rpm 1.3 MB/s | 43 kB 00:00 2023-07-14T19:19:31.968 INFO:teuthology.orchestra.run.smithi098.stdout:(93/131): libgfortran-8.5.0-10.1.el8_6.x86_64.r 13 MB/s | 644 kB 00:00 2023-07-14T19:19:31.977 INFO:teuthology.orchestra.run.smithi139.stdout:(73/131): pcre2-utf32-10.32-2.el8.x86_64.rpm 5.2 MB/s | 220 kB 00:00 2023-07-14T19:19:32.019 INFO:teuthology.orchestra.run.smithi139.stdout:(74/131): keyutils-libs-devel-1.5.10-9.el8.x86_ 1.1 MB/s | 48 kB 00:00 2023-07-14T19:19:32.035 INFO:teuthology.orchestra.run.smithi098.stdout:(94/131): libquadmath-8.5.0-10.1.el8_6.x86_64.r 2.5 MB/s | 171 kB 00:00 2023-07-14T19:19:32.053 INFO:teuthology.orchestra.run.smithi139.stdout:(75/131): libsepol-devel-2.9-3.el8.x86_64.rpm 2.6 MB/s | 87 kB 00:00 2023-07-14T19:19:32.069 INFO:teuthology.orchestra.run.smithi098.stdout:(95/131): NetworkManager-initscripts-updown-1.3 4.0 MB/s | 138 kB 00:00 2023-07-14T19:19:32.086 INFO:teuthology.orchestra.run.smithi139.stdout:(76/131): python3-requests-oauthlib-1.0.0-1.el8 1.3 MB/s | 43 kB 00:00 2023-07-14T19:19:32.120 INFO:teuthology.orchestra.run.smithi139.stdout:(77/131): python3-pycparser-2.14-14.el8.noarch. 3.2 MB/s | 109 kB 00:00 2023-07-14T19:19:32.162 INFO:teuthology.orchestra.run.smithi139.stdout:(78/131): pcre2-utf16-10.32-2.el8.x86_64.rpm 5.4 MB/s | 229 kB 00:00 2023-07-14T19:19:32.204 INFO:teuthology.orchestra.run.smithi139.stdout:(79/131): krb5-devel-1.18.2-14.el8.x86_64.rpm 13 MB/s | 560 kB 00:00 2023-07-14T19:19:32.238 INFO:teuthology.orchestra.run.smithi139.stdout:(80/131): libaio-devel-0.3.112-1.el8.x86_64.rpm 560 kB/s | 19 kB 00:00 2023-07-14T19:19:32.296 INFO:teuthology.orchestra.run.smithi139.stdout:(81/131): python3-oauthlib-2.1.0-1.el8.noarch.r 2.6 MB/s | 155 kB 00:00 2023-07-14T19:19:32.338 INFO:teuthology.orchestra.run.smithi139.stdout:(82/131): python3-cryptography-3.2.1-5.el8.x86_ 13 MB/s | 559 kB 00:00 2023-07-14T19:19:32.361 INFO:teuthology.orchestra.run.smithi098.stdout:(96/131): python3-scipy-1.0.0-21.module+el8.5.0 47 MB/s | 14 MB 00:00 2023-07-14T19:19:32.372 INFO:teuthology.orchestra.run.smithi139.stdout:(83/131): libverto-devel-0.3.0-5.el8.x86_64.rpm 535 kB/s | 18 kB 00:00 2023-07-14T19:19:32.402 INFO:teuthology.orchestra.run.smithi098.stdout:(97/131): python3-pyasn1-modules-0.3.7-6.el8.no 2.6 MB/s | 110 kB 00:00 2023-07-14T19:19:32.422 INFO:teuthology.orchestra.run.smithi139.stdout:(84/131): pcre2-devel-10.32-2.el8.x86_64.rpm 12 MB/s | 605 kB 00:00 2023-07-14T19:19:32.436 INFO:teuthology.orchestra.run.smithi098.stdout:(98/131): python3-pytz-2017.2-9.el8.noarch.rpm 1.6 MB/s | 54 kB 00:00 2023-07-14T19:19:32.464 INFO:teuthology.orchestra.run.smithi139.stdout:(85/131): python3-cffi-1.11.5-5.el8.x86_64.rpm 5.6 MB/s | 238 kB 00:00 2023-07-14T19:19:32.478 INFO:teuthology.orchestra.run.smithi098.stdout:(99/131): lttng-ust-2.8.1-11.el8.x86_64.rpm 6.1 MB/s | 259 kB 00:00 2023-07-14T19:19:32.520 INFO:teuthology.orchestra.run.smithi098.stdout:(100/131): socat-1.7.4.1-1.el8.x86_64.rpm 7.6 MB/s | 323 kB 00:00 2023-07-14T19:19:32.539 INFO:teuthology.orchestra.run.smithi139.stdout:(86/131): openssl-devel-1.1.1k-7.el8_6.x86_64.r 31 MB/s | 2.3 MB 00:00 2023-07-14T19:19:32.554 INFO:teuthology.orchestra.run.smithi098.stdout:(101/131): python3-markupsafe-0.23-19.el8.x86_6 1.1 MB/s | 39 kB 00:00 2023-07-14T19:19:32.573 INFO:teuthology.orchestra.run.smithi139.stdout:(87/131): xfsprogs-devel-5.0.0-10.el8.x86_64.rp 2.1 MB/s | 72 kB 00:00 2023-07-14T19:19:32.596 INFO:teuthology.orchestra.run.smithi098.stdout:(102/131): bison-3.0.4-10.el8.x86_64.rpm 16 MB/s | 688 kB 00:00 2023-07-14T19:19:32.615 INFO:teuthology.orchestra.run.smithi139.stdout:(88/131): fuse-2.9.7-15.el8.x86_64.rpm 1.9 MB/s | 83 kB 00:00 2023-07-14T19:19:32.657 INFO:teuthology.orchestra.run.smithi139.stdout:(89/131): elfutils-libelf-devel-0.186-1.el8.x86 1.4 MB/s | 60 kB 00:00 2023-07-14T19:19:32.665 INFO:teuthology.orchestra.run.smithi101.stdout:(130/131): thrift-0.13.0-2.el8.x86_64.rpm 66 kB/s | 1.7 MB 00:27 2023-07-14T19:19:32.671 INFO:teuthology.orchestra.run.smithi098.stdout:(103/131): python3-jinja2-2.10.1-3.el8.noarch.r 7.0 MB/s | 538 kB 00:00 2023-07-14T19:19:32.691 INFO:teuthology.orchestra.run.smithi139.stdout:(90/131): libuuid-devel-2.32.1-35.el8.x86_64.rp 2.9 MB/s | 98 kB 00:00 2023-07-14T19:19:32.714 INFO:teuthology.orchestra.run.smithi098.stdout:(104/131): python3-mako-1.0.6-13.el8.noarch.rpm 3.7 MB/s | 157 kB 00:00 2023-07-14T19:19:32.724 INFO:teuthology.orchestra.run.smithi139.stdout:(91/131): zlib-devel-1.2.11-18.el8_5.x86_64.rpm 1.7 MB/s | 58 kB 00:00 2023-07-14T19:19:32.755 INFO:teuthology.orchestra.run.smithi098.stdout:(105/131): oniguruma-6.8.2-2.el8.x86_64.rpm 4.4 MB/s | 187 kB 00:00 2023-07-14T19:19:32.783 INFO:teuthology.orchestra.run.smithi139.stdout:(92/131): libcom_err-devel-1.45.6-4.el8.x86_64. 658 kB/s | 39 kB 00:00 2023-07-14T19:19:32.806 INFO:teuthology.orchestra.run.smithi098.stdout:(106/131): automake-1.16.1-7.el8.noarch.rpm 14 MB/s | 713 kB 00:00 2023-07-14T19:19:32.825 INFO:teuthology.orchestra.run.smithi139.stdout:(93/131): libgfortran-8.5.0-10.1.el8_6.x86_64.r 15 MB/s | 644 kB 00:00 2023-07-14T19:19:32.867 INFO:teuthology.orchestra.run.smithi139.stdout:(94/131): libquadmath-8.5.0-10.1.el8_6.x86_64.r 4.0 MB/s | 171 kB 00:00 2023-07-14T19:19:32.893 INFO:teuthology.orchestra.run.smithi139.stdout:(95/131): NetworkManager-initscripts-updown-1.3 5.4 MB/s | 138 kB 00:00 2023-07-14T19:19:32.939 INFO:teuthology.orchestra.run.smithi098.stdout:(107/131): python3-babel-2.5.1-7.el8.noarch.rpm 36 MB/s | 4.8 MB 00:00 2023-07-14T19:19:32.965 INFO:teuthology.orchestra.run.smithi098.stdout:(108/131): python3-jsonpatch-1.21-2.el8.noarch. 1.0 MB/s | 27 kB 00:00 2023-07-14T19:19:33.007 INFO:teuthology.orchestra.run.smithi098.stdout:(109/131): flex-2.6.1-9.el8.x86_64.rpm 7.5 MB/s | 320 kB 00:00 2023-07-14T19:19:33.032 INFO:teuthology.orchestra.run.smithi098.stdout:(110/131): python3-prettytable-0.7.2-14.el8.noa 1.7 MB/s | 44 kB 00:00 2023-07-14T19:19:33.132 INFO:teuthology.orchestra.run.smithi098.stdout:(111/131): python3-numpy-1.14.3-10.el8.x86_64.r 37 MB/s | 3.7 MB 00:00 2023-07-14T19:19:33.160 INFO:teuthology.orchestra.run.smithi139.stdout:(96/131): python3-scipy-1.0.0-21.module+el8.5.0 51 MB/s | 14 MB 00:00 2023-07-14T19:19:33.183 INFO:teuthology.orchestra.run.smithi098.stdout:(112/131): python3-werkzeug-0.12.2-4.el8.noarch 9.0 MB/s | 457 kB 00:00 2023-07-14T19:19:33.225 INFO:teuthology.orchestra.run.smithi098.stdout:(113/131): python3-numpy-f2py-1.14.3-10.el8.x86 5.3 MB/s | 225 kB 00:00 2023-07-14T19:19:33.258 INFO:teuthology.orchestra.run.smithi098.stdout:(114/131): uuid-1.6.2-43.el8.x86_64.rpm 1.9 MB/s | 64 kB 00:00 2023-07-14T19:19:33.268 INFO:teuthology.orchestra.run.smithi139.stdout:(97/131): python3-pyasn1-modules-0.3.7-6.el8.no 1.0 MB/s | 110 kB 00:00 2023-07-14T19:19:33.308 INFO:teuthology.orchestra.run.smithi098.stdout:(115/131): python3-pyasn1-0.3.7-6.el8.noarch.rp 2.5 MB/s | 126 kB 00:00 2023-07-14T19:19:33.342 INFO:teuthology.orchestra.run.smithi098.stdout:(116/131): python3-jsonpointer-1.10-11.el8.noar 600 kB/s | 20 kB 00:00 2023-07-14T19:19:33.376 INFO:teuthology.orchestra.run.smithi098.stdout:(117/131): python3-pyOpenSSL-19.0.0-1.el8.noarc 3.1 MB/s | 103 kB 00:00 2023-07-14T19:19:33.402 INFO:teuthology.orchestra.run.smithi139.stdout:(98/131): python3-pytz-2017.2-9.el8.noarch.rpm 406 kB/s | 54 kB 00:00 2023-07-14T19:19:33.426 INFO:teuthology.orchestra.run.smithi098.stdout:(118/131): libtool-2.4.6-25.el8.x86_64.rpm 14 MB/s | 709 kB 00:00 2023-07-14T19:19:33.444 INFO:teuthology.orchestra.run.smithi139.stdout:(99/131): lttng-ust-2.8.1-11.el8.x86_64.rpm 6.1 MB/s | 259 kB 00:00 2023-07-14T19:19:33.485 INFO:teuthology.orchestra.run.smithi139.stdout:(100/131): socat-1.7.4.1-1.el8.x86_64.rpm 7.6 MB/s | 323 kB 00:00 2023-07-14T19:19:33.511 INFO:teuthology.orchestra.run.smithi139.stdout:(101/131): python3-markupsafe-0.23-19.el8.x86_6 1.5 MB/s | 39 kB 00:00 2023-07-14T19:19:33.552 INFO:teuthology.orchestra.run.smithi098.stdout:(119/131): openblas-threads-0.3.15-3.el8.x86_64 39 MB/s | 4.9 MB 00:00 2023-07-14T19:19:33.561 INFO:teuthology.orchestra.run.smithi139.stdout:(102/131): bison-3.0.4-10.el8.x86_64.rpm 13 MB/s | 688 kB 00:00 2023-07-14T19:19:33.593 INFO:teuthology.orchestra.run.smithi098.stdout:(120/131): libpq-13.5-1.el8.x86_64.rpm 4.7 MB/s | 198 kB 00:00 2023-07-14T19:19:33.603 INFO:teuthology.orchestra.run.smithi139.stdout:(103/131): python3-jinja2-2.10.1-3.el8.noarch.r 13 MB/s | 538 kB 00:00 2023-07-14T19:19:33.645 INFO:teuthology.orchestra.run.smithi139.stdout:(104/131): python3-mako-1.0.6-13.el8.noarch.rpm 3.7 MB/s | 157 kB 00:00 2023-07-14T19:19:33.679 INFO:teuthology.orchestra.run.smithi139.stdout:(105/131): oniguruma-6.8.2-2.el8.x86_64.rpm 5.5 MB/s | 187 kB 00:00 2023-07-14T19:19:33.727 INFO:teuthology.orchestra.run.smithi098.stdout:(121/131): openblas-0.3.15-3.el8.x86_64.rpm 35 MB/s | 4.7 MB 00:00 2023-07-14T19:19:33.729 INFO:teuthology.orchestra.run.smithi139.stdout:(106/131): automake-1.16.1-7.el8.noarch.rpm 14 MB/s | 713 kB 00:00 2023-07-14T19:19:33.770 INFO:teuthology.orchestra.run.smithi098.stdout:(122/131): jq-1.6-3.el8.x86_64.rpm 4.8 MB/s | 202 kB 00:00 2023-07-14T19:19:33.813 INFO:teuthology.orchestra.run.smithi098.stdout:(123/131): librdkafka-0.11.4-3.el8.x86_64.rpm 8.3 MB/s | 354 kB 00:00 2023-07-14T19:19:33.863 INFO:teuthology.orchestra.run.smithi139.stdout:(107/131): python3-babel-2.5.1-7.el8.noarch.rpm 36 MB/s | 4.8 MB 00:00 2023-07-14T19:19:33.896 INFO:teuthology.orchestra.run.smithi139.stdout:(108/131): python3-jsonpatch-1.21-2.el8.noarch. 801 kB/s | 27 kB 00:00 2023-07-14T19:19:33.938 INFO:teuthology.orchestra.run.smithi139.stdout:(109/131): flex-2.6.1-9.el8.x86_64.rpm 7.5 MB/s | 320 kB 00:00 2023-07-14T19:19:33.970 INFO:teuthology.orchestra.run.smithi098.stdout:(124/131): postgresql-server-10.21-2.module+el8 32 MB/s | 5.1 MB 00:00 2023-07-14T19:19:33.972 INFO:teuthology.orchestra.run.smithi139.stdout:(110/131): python3-prettytable-0.7.2-14.el8.noa 1.3 MB/s | 44 kB 00:00 2023-07-14T19:19:34.020 INFO:teuthology.orchestra.run.smithi098.stdout:(125/131): postgresql-contrib-10.21-2.module+el 16 MB/s | 808 kB 00:00 2023-07-14T19:19:34.079 INFO:teuthology.orchestra.run.smithi098.stdout:(126/131): postgresql-10.21-2.module+el8.6.0+15 26 MB/s | 1.5 MB 00:00 2023-07-14T19:19:34.080 INFO:teuthology.orchestra.run.smithi139.stdout:(111/131): python3-numpy-1.14.3-10.el8.x86_64.r 34 MB/s | 3.7 MB 00:00 2023-07-14T19:19:34.131 INFO:teuthology.orchestra.run.smithi139.stdout:(112/131): python3-werkzeug-0.12.2-4.el8.noarch 9.0 MB/s | 457 kB 00:00 2023-07-14T19:19:34.172 INFO:teuthology.orchestra.run.smithi139.stdout:(113/131): python3-numpy-f2py-1.14.3-10.el8.x86 5.3 MB/s | 225 kB 00:00 2023-07-14T19:19:34.206 INFO:teuthology.orchestra.run.smithi139.stdout:(114/131): uuid-1.6.2-43.el8.x86_64.rpm 1.9 MB/s | 64 kB 00:00 2023-07-14T19:19:34.248 INFO:teuthology.orchestra.run.smithi139.stdout:(115/131): python3-pyasn1-0.3.7-6.el8.noarch.rp 3.0 MB/s | 126 kB 00:00 2023-07-14T19:19:34.307 INFO:teuthology.orchestra.run.smithi139.stdout:(116/131): python3-jsonpointer-1.10-11.el8.noar 344 kB/s | 20 kB 00:00 2023-07-14T19:19:34.349 INFO:teuthology.orchestra.run.smithi139.stdout:(117/131): python3-pyOpenSSL-19.0.0-1.el8.noarc 2.4 MB/s | 103 kB 00:00 2023-07-14T19:19:34.391 INFO:teuthology.orchestra.run.smithi139.stdout:(118/131): libtool-2.4.6-25.el8.x86_64.rpm 17 MB/s | 709 kB 00:00 2023-07-14T19:19:34.516 INFO:teuthology.orchestra.run.smithi139.stdout:(119/131): openblas-threads-0.3.15-3.el8.x86_64 39 MB/s | 4.9 MB 00:00 2023-07-14T19:19:34.558 INFO:teuthology.orchestra.run.smithi139.stdout:(120/131): libpq-13.5-1.el8.x86_64.rpm 4.6 MB/s | 198 kB 00:00 2023-07-14T19:19:34.676 INFO:teuthology.orchestra.run.smithi139.stdout:(121/131): openblas-0.3.15-3.el8.x86_64.rpm 41 MB/s | 4.7 MB 00:00 2023-07-14T19:19:34.717 INFO:teuthology.orchestra.run.smithi139.stdout:(122/131): jq-1.6-3.el8.x86_64.rpm 4.8 MB/s | 202 kB 00:00 2023-07-14T19:19:34.790 INFO:teuthology.orchestra.run.smithi101.stdout:(131/131): ceph-test-18.0.0-4940.g81c9387d.el8. 542 kB/s | 48 MB 01:30 2023-07-14T19:19:34.792 INFO:teuthology.orchestra.run.smithi139.stdout:(123/131): librdkafka-0.11.4-3.el8.x86_64.rpm 4.6 MB/s | 354 kB 00:00 2023-07-14T19:19:34.792 INFO:teuthology.orchestra.run.smithi101.stdout:-------------------------------------------------------------------------------- 2023-07-14T19:19:34.793 INFO:teuthology.orchestra.run.smithi101.stdout:Total 1.2 MB/s | 206 MB 02:50 2023-07-14T19:19:34.917 INFO:teuthology.orchestra.run.smithi139.stdout:(124/131): postgresql-server-10.21-2.module+el8 41 MB/s | 5.1 MB 00:00 2023-07-14T19:19:34.976 INFO:teuthology.orchestra.run.smithi139.stdout:(125/131): postgresql-contrib-10.21-2.module+el 14 MB/s | 808 kB 00:00 2023-07-14T19:19:35.035 INFO:teuthology.orchestra.run.smithi139.stdout:(126/131): postgresql-10.21-2.module+el8.6.0+15 26 MB/s | 1.5 MB 00:00 2023-07-14T19:19:35.062 INFO:teuthology.orchestra.run.smithi098.stdout:(127/131): thrift-0.13.0-2.el8.x86_64.rpm 422 kB/s | 1.7 MB 00:04 2023-07-14T19:19:35.285 INFO:teuthology.orchestra.run.smithi139.stdout:(127/131): thrift-0.13.0-2.el8.x86_64.rpm 487 kB/s | 1.7 MB 00:03 2023-07-14T19:19:35.580 INFO:teuthology.orchestra.run.smithi101.stdout:Running transaction check 2023-07-14T19:19:35.700 INFO:teuthology.orchestra.run.smithi101.stdout:Transaction check succeeded. 2023-07-14T19:19:35.700 INFO:teuthology.orchestra.run.smithi101.stdout:Running transaction test 2023-07-14T19:19:36.827 INFO:teuthology.orchestra.run.smithi101.stdout:Transaction test succeeded. 2023-07-14T19:19:36.831 INFO:teuthology.orchestra.run.smithi101.stdout:Running transaction 2023-07-14T19:19:38.751 INFO:teuthology.orchestra.run.smithi139.stdout:(128/131): ceph-test-18.0.0-4940.g81c9387d.el8. 709 kB/s | 48 MB 01:09 2023-07-14T19:19:38.802 INFO:teuthology.orchestra.run.smithi139.stdout:(129/131): iproute-5.15.0-4.el8_6.1.x86_64.rpm 16 MB/s | 798 kB 00:00 2023-07-14T19:19:41.783 INFO:teuthology.orchestra.run.smithi101.stdout: Preparing : 1/1 2023-07-14T19:19:41.971 INFO:teuthology.orchestra.run.smithi101.stdout: Installing : thrift-0.13.0-2.el8.x86_64 1/134 2023-07-14T19:19:42.204 INFO:teuthology.orchestra.run.smithi101.stdout: Installing : liboath-2.6.2-3.el8.x86_64 2/134 2023-07-14T19:19:42.423 INFO:teuthology.orchestra.run.smithi101.stdout: Installing : librdkafka-0.11.4-3.el8.x86_64 3/134 2023-07-14T19:19:42.512 INFO:teuthology.orchestra.run.smithi101.stdout: Running scriptlet: librdkafka-0.11.4-3.el8.x86_64 3/134 2023-07-14T19:19:42.731 INFO:teuthology.orchestra.run.smithi101.stdout: Installing : lttng-ust-2.8.1-11.el8.x86_64 4/134 2023-07-14T19:19:43.167 INFO:teuthology.orchestra.run.smithi101.stdout: Running scriptlet: lttng-ust-2.8.1-11.el8.x86_64 4/134 2023-07-14T19:19:43.414 INFO:teuthology.orchestra.run.smithi101.stdout: Upgrading : librados2-2:18.0.0-4940.g81c9387d.el8.x86_64 5/134 2023-07-14T19:19:43.811 INFO:teuthology.orchestra.run.smithi101.stdout: Running scriptlet: librados2-2:18.0.0-4940.g81c9387d.el8.x86_64 5/134 2023-07-14T19:19:44.031 INFO:teuthology.orchestra.run.smithi101.stdout: Upgrading : librbd1-2:18.0.0-4940.g81c9387d.el8.x86_64 6/134 2023-07-14T19:19:44.157 INFO:teuthology.orchestra.run.smithi101.stdout: Running scriptlet: librbd1-2:18.0.0-4940.g81c9387d.el8.x86_64 6/134 2023-07-14T19:19:44.356 INFO:teuthology.orchestra.run.smithi101.stdout: Installing : libcephfs2-2:18.0.0-4940.g81c9387d.el8.x86_64 7/134 2023-07-14T19:19:44.488 INFO:teuthology.orchestra.run.smithi101.stdout: Running scriptlet: libcephfs2-2:18.0.0-4940.g81c9387d.el8.x86_64 7/134 2023-07-14T19:19:44.644 INFO:teuthology.orchestra.run.smithi101.stdout: Installing : python3-rados-2:18.0.0-4940.g81c9387d.el8.x86_64 8/134 2023-07-14T19:19:44.807 INFO:teuthology.orchestra.run.smithi101.stdout: Installing : librabbitmq-0.9.0-3.el8.x86_64 9/134 2023-07-14T19:19:45.026 INFO:teuthology.orchestra.run.smithi101.stdout: Installing : libpq-13.5-1.el8.x86_64 10/134 2023-07-14T19:19:45.267 INFO:teuthology.orchestra.run.smithi101.stdout: Installing : python3-pyasn1-0.3.7-6.el8.noarch 11/134 2023-07-14T19:19:45.662 INFO:teuthology.orchestra.run.smithi101.stdout: Installing : python3-more-itertools-7.2.0-3.el8.noarch 12/134 2023-07-14T19:19:45.918 INFO:teuthology.orchestra.run.smithi101.stdout: Installing : postgresql-10.21-2.module+el8.6.0+15342+53518fac 13/134 2023-07-14T19:19:46.096 INFO:teuthology.orchestra.run.smithi098.stdout:(128/131): librbd1-18.0.0-4940.g81c9387d.el8.x8 337 kB/s | 3.6 MB 00:11 2023-07-14T19:19:46.097 INFO:teuthology.orchestra.run.smithi101.stdout: Installing : libcephsqlite-2:18.0.0-4940.g81c9387d.el8.x86_64 14/134 2023-07-14T19:19:46.154 INFO:teuthology.orchestra.run.smithi098.stdout:(129/131): iproute-5.15.0-4.el8_6.1.x86_64.rpm 14 MB/s | 798 kB 00:00 2023-07-14T19:19:46.211 INFO:teuthology.orchestra.run.smithi101.stdout: Running scriptlet: libcephsqlite-2:18.0.0-4940.g81c9387d.el8.x86_64 14/134 2023-07-14T19:19:46.356 INFO:teuthology.orchestra.run.smithi101.stdout: Installing : libradosstriper1-2:18.0.0-4940.g81c9387d.el8.x86 15/134 2023-07-14T19:19:46.451 INFO:teuthology.orchestra.run.smithi139.stdout:(130/131): librados2-18.0.0-4940.g81c9387d.el8. 330 kB/s | 3.7 MB 00:11 2023-07-14T19:19:46.538 INFO:teuthology.orchestra.run.smithi101.stdout: Running scriptlet: libradosstriper1-2:18.0.0-4940.g81c9387d.el8.x86 15/134 2023-07-14T19:19:46.751 INFO:teuthology.orchestra.run.smithi101.stdout: Installing : python3-werkzeug-0.12.2-4.el8.noarch 16/134 2023-07-14T19:19:46.779 INFO:teuthology.orchestra.run.smithi098.stdout:(130/131): librados2-18.0.0-4940.g81c9387d.el8. 297 kB/s | 3.7 MB 00:12 2023-07-14T19:19:46.793 INFO:teuthology.orchestra.run.smithi139.stdout:(131/131): librbd1-18.0.0-4940.g81c9387d.el8.x8 323 kB/s | 3.6 MB 00:11 2023-07-14T19:19:46.795 INFO:teuthology.orchestra.run.smithi139.stdout:-------------------------------------------------------------------------------- 2023-07-14T19:19:46.796 INFO:teuthology.orchestra.run.smithi139.stdout:Total 1.8 MB/s | 206 MB 01:52 2023-07-14T19:19:46.970 INFO:teuthology.orchestra.run.smithi101.stdout: Installing : python3-markupsafe-0.23-19.el8.x86_64 17/134 2023-07-14T19:19:47.148 INFO:teuthology.orchestra.run.smithi101.stdout: Installing : python3-pytz-2017.2-9.el8.noarch 18/134 2023-07-14T19:19:47.347 INFO:teuthology.orchestra.run.smithi101.stdout: Installing : libquadmath-8.5.0-10.1.el8_6.x86_64 19/134 2023-07-14T19:19:47.429 INFO:teuthology.orchestra.run.smithi098.stdout:(131/131): ceph-test-18.0.0-4940.g81c9387d.el8. 563 kB/s | 48 MB 01:27 2023-07-14T19:19:47.432 INFO:teuthology.orchestra.run.smithi098.stdout:-------------------------------------------------------------------------------- 2023-07-14T19:19:47.432 INFO:teuthology.orchestra.run.smithi098.stdout:Total 1.5 MB/s | 206 MB 02:17 2023-07-14T19:19:47.495 INFO:teuthology.orchestra.run.smithi101.stdout: Running scriptlet: libquadmath-8.5.0-10.1.el8_6.x86_64 19/134 2023-07-14T19:19:47.604 INFO:teuthology.orchestra.run.smithi139.stdout:Running transaction check 2023-07-14T19:19:47.680 INFO:teuthology.orchestra.run.smithi101.stdout: Installing : libgfortran-8.5.0-10.1.el8_6.x86_64 20/134 2023-07-14T19:19:47.724 INFO:teuthology.orchestra.run.smithi139.stdout:Transaction check succeeded. 2023-07-14T19:19:47.724 INFO:teuthology.orchestra.run.smithi139.stdout:Running transaction test 2023-07-14T19:19:48.206 INFO:teuthology.orchestra.run.smithi098.stdout:Running transaction check 2023-07-14T19:19:48.279 INFO:teuthology.orchestra.run.smithi101.stdout: Running scriptlet: libgfortran-8.5.0-10.1.el8_6.x86_64 20/134 2023-07-14T19:19:48.326 INFO:teuthology.orchestra.run.smithi098.stdout:Transaction check succeeded. 2023-07-14T19:19:48.326 INFO:teuthology.orchestra.run.smithi098.stdout:Running transaction test 2023-07-14T19:19:48.480 INFO:teuthology.orchestra.run.smithi101.stdout: Installing : openblas-threads-0.3.15-3.el8.x86_64 21/134 2023-07-14T19:19:48.837 INFO:teuthology.orchestra.run.smithi139.stdout:Transaction test succeeded. 2023-07-14T19:19:48.842 INFO:teuthology.orchestra.run.smithi139.stdout:Running transaction 2023-07-14T19:19:49.063 INFO:teuthology.orchestra.run.smithi101.stdout: Running scriptlet: openblas-threads-0.3.15-3.el8.x86_64 21/134 2023-07-14T19:19:49.272 INFO:teuthology.orchestra.run.smithi101.stdout: Installing : openblas-0.3.15-3.el8.x86_64 22/134 2023-07-14T19:19:49.444 INFO:teuthology.orchestra.run.smithi098.stdout:Transaction test succeeded. 2023-07-14T19:19:49.448 INFO:teuthology.orchestra.run.smithi098.stdout:Running transaction 2023-07-14T19:19:49.976 INFO:teuthology.orchestra.run.smithi101.stdout: Running scriptlet: openblas-0.3.15-3.el8.x86_64 22/134 2023-07-14T19:19:50.262 INFO:teuthology.orchestra.run.smithi101.stdout: Installing : python3-numpy-1:1.14.3-10.el8.x86_64 23/134 2023-07-14T19:19:50.460 INFO:teuthology.orchestra.run.smithi101.stdout: Installing : zlib-devel-1.2.11-18.el8_5.x86_64 24/134 2023-07-14T19:19:50.698 INFO:teuthology.orchestra.run.smithi101.stdout: Installing : python3-webob-1.8.5-1.el8.1.noarch 25/134 2023-07-14T19:19:50.912 INFO:teuthology.orchestra.run.smithi101.stdout: Installing : python3-ceph-common-2:18.0.0-4940.g81c9387d.el8. 26/134 2023-07-14T19:19:51.049 INFO:teuthology.orchestra.run.smithi101.stdout: Installing : python3-ceph-argparse-2:18.0.0-4940.g81c9387d.el 27/134 2023-07-14T19:19:51.268 INFO:teuthology.orchestra.run.smithi101.stdout: Installing : python3-cephfs-2:18.0.0-4940.g81c9387d.el8.x86_6 28/134 2023-07-14T19:19:51.463 INFO:teuthology.orchestra.run.smithi101.stdout: Installing : python3-numpy-f2py-1:1.14.3-10.el8.x86_64 29/134 2023-07-14T19:19:53.232 INFO:teuthology.orchestra.run.smithi101.stdout: Running scriptlet: python3-numpy-f2py-1:1.14.3-10.el8.x86_64 29/134 2023-07-14T19:19:53.883 INFO:teuthology.orchestra.run.smithi139.stdout: Preparing : 1/1 2023-07-14T19:19:54.075 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : thrift-0.13.0-2.el8.x86_64 1/134 2023-07-14T19:19:54.266 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : liboath-2.6.2-3.el8.x86_64 2/134 2023-07-14T19:19:54.288 INFO:teuthology.orchestra.run.smithi098.stdout: Preparing : 1/1 2023-07-14T19:19:54.298 INFO:teuthology.orchestra.run.smithi101.stdout: Installing : python3-scipy-1.0.0-21.module+el8.5.0+10916+41bd 30/134 2023-07-14T19:19:54.460 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : librdkafka-0.11.4-3.el8.x86_64 3/134 2023-07-14T19:19:54.480 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : thrift-0.13.0-2.el8.x86_64 1/134 2023-07-14T19:19:54.567 INFO:teuthology.orchestra.run.smithi139.stdout: Running scriptlet: librdkafka-0.11.4-3.el8.x86_64 3/134 2023-07-14T19:19:54.639 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : liboath-2.6.2-3.el8.x86_64 2/134 2023-07-14T19:19:54.651 INFO:teuthology.orchestra.run.smithi101.stdout: Installing : python3-babel-2.5.1-7.el8.noarch 31/134 2023-07-14T19:19:54.777 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : lttng-ust-2.8.1-11.el8.x86_64 4/134 2023-07-14T19:19:54.807 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : librdkafka-0.11.4-3.el8.x86_64 3/134 2023-07-14T19:19:54.918 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: librdkafka-0.11.4-3.el8.x86_64 3/134 2023-07-14T19:19:54.964 INFO:teuthology.orchestra.run.smithi101.stdout: Installing : python3-jinja2-2.10.1-3.el8.noarch 32/134 2023-07-14T19:19:55.174 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : lttng-ust-2.8.1-11.el8.x86_64 4/134 2023-07-14T19:19:55.180 INFO:teuthology.orchestra.run.smithi139.stdout: Running scriptlet: lttng-ust-2.8.1-11.el8.x86_64 4/134 2023-07-14T19:19:55.186 INFO:teuthology.orchestra.run.smithi101.stdout: Installing : python3-mako-1.0.6-13.el8.noarch 33/134 2023-07-14T19:19:55.410 INFO:teuthology.orchestra.run.smithi139.stdout: Upgrading : librados2-2:18.0.0-4940.g81c9387d.el8.x86_64 5/134 2023-07-14T19:19:55.441 INFO:teuthology.orchestra.run.smithi101.stdout: Installing : python3-rsa-4.8-1.el8.noarch 34/134 2023-07-14T19:19:55.558 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: lttng-ust-2.8.1-11.el8.x86_64 4/134 2023-07-14T19:19:55.648 INFO:teuthology.orchestra.run.smithi101.stdout: Installing : python3-pyasn1-modules-0.3.7-6.el8.noarch 35/134 2023-07-14T19:19:55.774 INFO:teuthology.orchestra.run.smithi098.stdout: Upgrading : librados2-2:18.0.0-4940.g81c9387d.el8.x86_64 5/134 2023-07-14T19:19:55.811 INFO:teuthology.orchestra.run.smithi139.stdout: Running scriptlet: librados2-2:18.0.0-4940.g81c9387d.el8.x86_64 5/134 2023-07-14T19:19:55.843 INFO:teuthology.orchestra.run.smithi101.stdout: Installing : python3-rbd-2:18.0.0-4940.g81c9387d.el8.x86_64 36/134 2023-07-14T19:19:56.011 INFO:teuthology.orchestra.run.smithi101.stdout: Installing : librados-devel-2:18.0.0-4940.g81c9387d.el8.x86_6 37/134 2023-07-14T19:19:56.019 INFO:teuthology.orchestra.run.smithi139.stdout: Upgrading : librbd1-2:18.0.0-4940.g81c9387d.el8.x86_64 6/134 2023-07-14T19:19:56.141 INFO:teuthology.orchestra.run.smithi139.stdout: Running scriptlet: librbd1-2:18.0.0-4940.g81c9387d.el8.x86_64 6/134 2023-07-14T19:19:56.172 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: librados2-2:18.0.0-4940.g81c9387d.el8.x86_64 5/134 2023-07-14T19:19:56.203 INFO:teuthology.orchestra.run.smithi101.stdout: Installing : python3-jsonpointer-1.10-11.el8.noarch 38/134 2023-07-14T19:19:56.359 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : libcephfs2-2:18.0.0-4940.g81c9387d.el8.x86_64 7/134 2023-07-14T19:19:56.363 INFO:teuthology.orchestra.run.smithi101.stdout: Installing : python3-jsonpatch-1.21-2.el8.noarch 39/134 2023-07-14T19:19:56.374 INFO:teuthology.orchestra.run.smithi098.stdout: Upgrading : librbd1-2:18.0.0-4940.g81c9387d.el8.x86_64 6/134 2023-07-14T19:19:56.447 INFO:teuthology.orchestra.run.smithi139.stdout: Running scriptlet: libcephfs2-2:18.0.0-4940.g81c9387d.el8.x86_64 7/134 2023-07-14T19:19:56.504 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: librbd1-2:18.0.0-4940.g81c9387d.el8.x86_64 6/134 2023-07-14T19:19:56.606 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : python3-rados-2:18.0.0-4940.g81c9387d.el8.x86_64 8/134 2023-07-14T19:19:56.612 INFO:teuthology.orchestra.run.smithi101.stdout: Installing : uuid-1.6.2-43.el8.x86_64 40/134 2023-07-14T19:19:56.668 INFO:teuthology.orchestra.run.smithi101.stdout: Running scriptlet: uuid-1.6.2-43.el8.x86_64 40/134 2023-07-14T19:19:56.693 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : libcephfs2-2:18.0.0-4940.g81c9387d.el8.x86_64 7/134 2023-07-14T19:19:56.760 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : librabbitmq-0.9.0-3.el8.x86_64 9/134 2023-07-14T19:19:56.776 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: libcephfs2-2:18.0.0-4940.g81c9387d.el8.x86_64 7/134 2023-07-14T19:19:56.891 INFO:teuthology.orchestra.run.smithi101.stdout: Installing : python3-prettytable-0.7.2-14.el8.noarch 41/134 2023-07-14T19:19:56.949 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-rados-2:18.0.0-4940.g81c9387d.el8.x86_64 8/134 2023-07-14T19:19:57.021 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : libpq-13.5-1.el8.x86_64 10/134 2023-07-14T19:19:57.124 INFO:teuthology.orchestra.run.smithi101.stdout: Installing : automake-1.16.1-7.el8.noarch 42/134 2023-07-14T19:19:57.140 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : librabbitmq-0.9.0-3.el8.x86_64 9/134 2023-07-14T19:19:57.221 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : python3-pyasn1-0.3.7-6.el8.noarch 11/134 2023-07-14T19:19:57.321 INFO:teuthology.orchestra.run.smithi101.stdout: Installing : oniguruma-6.8.2-2.el8.x86_64 43/134 2023-07-14T19:19:57.370 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : libpq-13.5-1.el8.x86_64 10/134 2023-07-14T19:19:57.397 INFO:teuthology.orchestra.run.smithi101.stdout: Running scriptlet: oniguruma-6.8.2-2.el8.x86_64 43/134 2023-07-14T19:19:57.568 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-pyasn1-0.3.7-6.el8.noarch 11/134 2023-07-14T19:19:57.583 INFO:teuthology.orchestra.run.smithi101.stdout: Installing : jq-1.6-3.el8.x86_64 44/134 2023-07-14T19:19:57.652 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : python3-more-itertools-7.2.0-3.el8.noarch 12/134 2023-07-14T19:19:57.791 INFO:teuthology.orchestra.run.smithi101.stdout: Installing : socat-1.7.4.1-1.el8.x86_64 45/134 2023-07-14T19:19:57.851 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : postgresql-10.21-2.module+el8.6.0+15342+53518fac 13/134 2023-07-14T19:19:57.969 INFO:teuthology.orchestra.run.smithi101.stdout: Installing : libcom_err-devel-1.45.6-4.el8.x86_64 46/134 2023-07-14T19:19:58.001 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : libcephsqlite-2:18.0.0-4940.g81c9387d.el8.x86_64 14/134 2023-07-14T19:19:58.006 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-more-itertools-7.2.0-3.el8.noarch 12/134 2023-07-14T19:19:58.113 INFO:teuthology.orchestra.run.smithi139.stdout: Running scriptlet: libcephsqlite-2:18.0.0-4940.g81c9387d.el8.x86_64 14/134 2023-07-14T19:19:58.191 INFO:teuthology.orchestra.run.smithi101.stdout: Installing : libuuid-devel-2.32.1-35.el8.x86_64 47/134 2023-07-14T19:19:58.210 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : postgresql-10.21-2.module+el8.6.0+15342+53518fac 13/134 2023-07-14T19:19:58.251 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : libradosstriper1-2:18.0.0-4940.g81c9387d.el8.x86 15/134 2023-07-14T19:19:58.365 INFO:teuthology.orchestra.run.smithi101.stdout: Installing : fuse-2.9.7-15.el8.x86_64 48/134 2023-07-14T19:19:58.382 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : libcephsqlite-2:18.0.0-4940.g81c9387d.el8.x86_64 14/134 2023-07-14T19:19:58.386 INFO:teuthology.orchestra.run.smithi139.stdout: Running scriptlet: libradosstriper1-2:18.0.0-4940.g81c9387d.el8.x86 15/134 2023-07-14T19:19:58.494 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: libcephsqlite-2:18.0.0-4940.g81c9387d.el8.x86_64 14/134 2023-07-14T19:19:58.507 INFO:teuthology.orchestra.run.smithi101.stdout: Installing : libverto-devel-0.3.0-5.el8.x86_64 49/134 2023-07-14T19:19:58.596 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : python3-werkzeug-0.12.2-4.el8.noarch 16/134 2023-07-14T19:19:58.640 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : libradosstriper1-2:18.0.0-4940.g81c9387d.el8.x86 15/134 2023-07-14T19:19:58.750 INFO:teuthology.orchestra.run.smithi101.stdout: Installing : pcre2-utf16-10.32-2.el8.x86_64 50/134 2023-07-14T19:19:58.774 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : python3-markupsafe-0.23-19.el8.x86_64 17/134 2023-07-14T19:19:58.784 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: libradosstriper1-2:18.0.0-4940.g81c9387d.el8.x86 15/134 2023-07-14T19:19:58.940 INFO:teuthology.orchestra.run.smithi101.stdout: Installing : python3-pycparser-2.14-14.el8.noarch 51/134 2023-07-14T19:19:58.982 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : python3-pytz-2017.2-9.el8.noarch 18/134 2023-07-14T19:19:59.019 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-werkzeug-0.12.2-4.el8.noarch 16/134 2023-07-14T19:19:59.109 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : libquadmath-8.5.0-10.1.el8_6.x86_64 19/134 2023-07-14T19:19:59.204 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-markupsafe-0.23-19.el8.x86_64 17/134 2023-07-14T19:19:59.251 INFO:teuthology.orchestra.run.smithi139.stdout: Running scriptlet: libquadmath-8.5.0-10.1.el8_6.x86_64 19/134 2023-07-14T19:19:59.303 INFO:teuthology.orchestra.run.smithi101.stdout: Installing : python3-cffi-1.11.5-5.el8.x86_64 52/134 2023-07-14T19:19:59.385 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-pytz-2017.2-9.el8.noarch 18/134 2023-07-14T19:19:59.434 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : libgfortran-8.5.0-10.1.el8_6.x86_64 20/134 2023-07-14T19:19:59.550 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : libquadmath-8.5.0-10.1.el8_6.x86_64 19/134 2023-07-14T19:19:59.647 INFO:teuthology.orchestra.run.smithi101.stdout: Installing : python3-cryptography-3.2.1-5.el8.x86_64 53/134 2023-07-14T19:19:59.689 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: libquadmath-8.5.0-10.1.el8_6.x86_64 19/134 2023-07-14T19:19:59.815 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : libgfortran-8.5.0-10.1.el8_6.x86_64 20/134 2023-07-14T19:19:59.912 INFO:teuthology.orchestra.run.smithi101.stdout: Installing : python3-jwt-1.6.1-2.el8.noarch 54/134 2023-07-14T19:20:00.059 INFO:teuthology.orchestra.run.smithi139.stdout: Running scriptlet: libgfortran-8.5.0-10.1.el8_6.x86_64 20/134 2023-07-14T19:20:00.124 INFO:teuthology.orchestra.run.smithi101.stdout: Installing : python3-pyOpenSSL-19.0.0-1.el8.noarch 55/134 2023-07-14T19:20:00.218 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : openblas-threads-0.3.15-3.el8.x86_64 21/134 2023-07-14T19:20:00.358 INFO:teuthology.orchestra.run.smithi101.stdout: Installing : python3-oauthlib-2.1.0-1.el8.noarch 56/134 2023-07-14T19:20:00.447 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: libgfortran-8.5.0-10.1.el8_6.x86_64 20/134 2023-07-14T19:20:00.606 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : openblas-threads-0.3.15-3.el8.x86_64 21/134 2023-07-14T19:20:00.694 INFO:teuthology.orchestra.run.smithi101.stdout: Installing : python3-requests-oauthlib-1.0.0-1.el8.noarch 57/134 2023-07-14T19:20:00.816 INFO:teuthology.orchestra.run.smithi139.stdout: Running scriptlet: openblas-threads-0.3.15-3.el8.x86_64 21/134 2023-07-14T19:20:00.886 INFO:teuthology.orchestra.run.smithi101.stdout: Installing : python3-asyncssh-2.7.0-2.el8.noarch 58/134 2023-07-14T19:20:00.976 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : openblas-0.3.15-3.el8.x86_64 22/134 2023-07-14T19:20:01.113 INFO:teuthology.orchestra.run.smithi101.stdout: Installing : python3-trustme-0.6.0-4.el8.noarch 59/134 2023-07-14T19:20:01.197 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: openblas-threads-0.3.15-3.el8.x86_64 21/134 2023-07-14T19:20:01.324 INFO:teuthology.orchestra.run.smithi101.stdout: Installing : python3-bcrypt-3.1.6-2.el8.1.x86_64 60/134 2023-07-14T19:20:01.365 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : openblas-0.3.15-3.el8.x86_64 22/134 2023-07-14T19:20:01.533 INFO:teuthology.orchestra.run.smithi101.stdout: Installing : libsepol-devel-2.9-3.el8.x86_64 61/134 2023-07-14T19:20:01.688 INFO:teuthology.orchestra.run.smithi139.stdout: Running scriptlet: openblas-0.3.15-3.el8.x86_64 22/134 2023-07-14T19:20:01.715 INFO:teuthology.orchestra.run.smithi101.stdout: Installing : keyutils-libs-devel-1.5.10-9.el8.x86_64 62/134 2023-07-14T19:20:01.965 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : python3-numpy-1:1.14.3-10.el8.x86_64 23/134 2023-07-14T19:20:02.063 INFO:teuthology.orchestra.run.smithi101.stdout: Installing : pcre2-utf32-10.32-2.el8.x86_64 63/134 2023-07-14T19:20:02.088 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: openblas-0.3.15-3.el8.x86_64 22/134 2023-07-14T19:20:02.166 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : zlib-devel-1.2.11-18.el8_5.x86_64 24/134 2023-07-14T19:20:02.321 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-numpy-1:1.14.3-10.el8.x86_64 23/134 2023-07-14T19:20:02.334 INFO:teuthology.orchestra.run.smithi101.stdout: Installing : pcre2-devel-10.32-2.el8.x86_64 64/134 2023-07-14T19:20:02.439 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : python3-webob-1.8.5-1.el8.1.noarch 25/134 2023-07-14T19:20:02.513 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : zlib-devel-1.2.11-18.el8_5.x86_64 24/134 2023-07-14T19:20:02.558 INFO:teuthology.orchestra.run.smithi101.stdout: Installing : libselinux-devel-2.9-5.el8.x86_64 65/134 2023-07-14T19:20:02.624 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : python3-ceph-common-2:18.0.0-4940.g81c9387d.el8. 26/134 2023-07-14T19:20:02.761 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : python3-ceph-argparse-2:18.0.0-4940.g81c9387d.el 27/134 2023-07-14T19:20:02.792 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-webob-1.8.5-1.el8.1.noarch 25/134 2023-07-14T19:20:02.830 INFO:teuthology.orchestra.run.smithi101.stdout: Installing : krb5-devel-1.18.2-14.el8.x86_64 66/134 2023-07-14T19:20:02.946 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-ceph-common-2:18.0.0-4940.g81c9387d.el8. 26/134 2023-07-14T19:20:03.014 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : python3-cephfs-2:18.0.0-4940.g81c9387d.el8.x86_6 28/134 2023-07-14T19:20:03.019 INFO:teuthology.orchestra.run.smithi101.stdout: Installing : xmlstarlet-1.6.1-20.el8.x86_64 67/134 2023-07-14T19:20:03.067 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-ceph-argparse-2:18.0.0-4940.g81c9387d.el 27/134 2023-07-14T19:20:03.181 INFO:teuthology.orchestra.run.smithi101.stdout: Installing : python3-zc-lockfile-2.0-2.el8.noarch 68/134 2023-07-14T19:20:03.201 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : python3-numpy-f2py-1:1.14.3-10.el8.x86_64 29/134 2023-07-14T19:20:03.302 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-cephfs-2:18.0.0-4940.g81c9387d.el8.x86_6 28/134 2023-07-14T19:20:03.463 INFO:teuthology.orchestra.run.smithi101.stdout: Installing : python3-websocket-client-0.56.0-5.el8.noarch 69/134 2023-07-14T19:20:03.564 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-numpy-f2py-1:1.14.3-10.el8.x86_64 29/134 2023-07-14T19:20:03.668 INFO:teuthology.orchestra.run.smithi101.stdout: Installing : python3-waitress-1.4.3-1.el8.noarch 70/134 2023-07-14T19:20:03.849 INFO:teuthology.orchestra.run.smithi101.stdout: Installing : python3-singledispatch-3.4.0.3-18.el8.noarch 71/134 2023-07-14T19:20:03.997 INFO:teuthology.orchestra.run.smithi101.stdout: Installing : python3-simplegeneric-0.8.1-17.el8.noarch 72/134 2023-07-14T19:20:04.186 INFO:teuthology.orchestra.run.smithi101.stdout: Installing : python3-repoze-lru-0.7-6.el8.noarch 73/134 2023-07-14T19:20:04.398 INFO:teuthology.orchestra.run.smithi101.stdout: Installing : python3-routes-2.4.1-12.el8.noarch 74/134 2023-07-14T19:20:04.571 INFO:teuthology.orchestra.run.smithi101.stdout: Installing : python3-natsort-7.1.1-2.el8.noarch 75/134 2023-07-14T19:20:04.781 INFO:teuthology.orchestra.run.smithi101.stdout: Installing : python3-logutils-0.3.5-11.el8.noarch 76/134 2023-07-14T19:20:04.942 INFO:teuthology.orchestra.run.smithi101.stdout: Installing : python3-jaraco-6.2-6.el8.noarch 77/134 2023-07-14T19:20:04.964 INFO:teuthology.orchestra.run.smithi139.stdout: Running scriptlet: python3-numpy-f2py-1:1.14.3-10.el8.x86_64 29/134 2023-07-14T19:20:05.191 INFO:teuthology.orchestra.run.smithi101.stdout: Installing : python3-jaraco-functools-2.0-4.el8.noarch 78/134 2023-07-14T19:20:05.326 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: python3-numpy-f2py-1:1.14.3-10.el8.x86_64 29/134 2023-07-14T19:20:05.448 INFO:teuthology.orchestra.run.smithi101.stdout: Installing : python3-cheroot-8.5.2-1.el8.noarch 79/134 2023-07-14T19:20:05.609 INFO:teuthology.orchestra.run.smithi101.stdout: Installing : python3-tempora-1.14.1-5.el8.noarch 80/134 2023-07-14T19:20:05.849 INFO:teuthology.orchestra.run.smithi101.stdout: Installing : python3-portend-2.6-1.el8.noarch 81/134 2023-07-14T19:20:05.965 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : python3-scipy-1.0.0-21.module+el8.5.0+10916+41bd 30/134 2023-07-14T19:20:06.069 INFO:teuthology.orchestra.run.smithi101.stdout: Installing : python3-cherrypy-18.4.0-1.el8.noarch 82/134 2023-07-14T19:20:06.259 INFO:teuthology.orchestra.run.smithi101.stdout: Installing : python3-certifi-2018.10.15-7.el8.noarch 83/134 2023-07-14T19:20:06.291 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : python3-babel-2.5.1-7.el8.noarch 31/134 2023-07-14T19:20:06.315 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-scipy-1.0.0-21.module+el8.5.0+10916+41bd 30/134 2023-07-14T19:20:06.478 INFO:teuthology.orchestra.run.smithi101.stdout: Installing : python3-cachetools-3.1.1-4.el8.noarch 84/134 2023-07-14T19:20:06.577 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : python3-jinja2-2.10.1-3.el8.noarch 32/134 2023-07-14T19:20:06.629 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-babel-2.5.1-7.el8.noarch 31/134 2023-07-14T19:20:06.837 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : python3-mako-1.0.6-13.el8.noarch 33/134 2023-07-14T19:20:06.891 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-jinja2-2.10.1-3.el8.noarch 32/134 2023-07-14T19:20:07.062 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : python3-rsa-4.8-1.el8.noarch 34/134 2023-07-14T19:20:07.107 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-mako-1.0.6-13.el8.noarch 33/134 2023-07-14T19:20:07.235 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : python3-pyasn1-modules-0.3.7-6.el8.noarch 35/134 2023-07-14T19:20:07.351 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-rsa-4.8-1.el8.noarch 34/134 2023-07-14T19:20:07.405 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : python3-rbd-2:18.0.0-4940.g81c9387d.el8.x86_64 36/134 2023-07-14T19:20:07.541 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-pyasn1-modules-0.3.7-6.el8.noarch 35/134 2023-07-14T19:20:07.548 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : librados-devel-2:18.0.0-4940.g81c9387d.el8.x86_6 37/134 2023-07-14T19:20:07.625 INFO:teuthology.orchestra.run.smithi101.stdout: Installing : python3-google-auth-1:1.1.1-10.el8.noarch 85/134 2023-07-14T19:20:07.690 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : python3-jsonpointer-1.10-11.el8.noarch 38/134 2023-07-14T19:20:07.712 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-rbd-2:18.0.0-4940.g81c9387d.el8.x86_64 36/134 2023-07-14T19:20:07.877 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : python3-jsonpatch-1.21-2.el8.noarch 39/134 2023-07-14T19:20:07.904 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : librados-devel-2:18.0.0-4940.g81c9387d.el8.x86_6 37/134 2023-07-14T19:20:07.939 INFO:teuthology.orchestra.run.smithi101.stdout: Installing : python3-kubernetes-1:11.0.0-6.el8.noarch 86/134 2023-07-14T19:20:08.041 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : uuid-1.6.2-43.el8.x86_64 40/134 2023-07-14T19:20:08.062 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-jsonpointer-1.10-11.el8.noarch 38/134 2023-07-14T19:20:08.092 INFO:teuthology.orchestra.run.smithi139.stdout: Running scriptlet: uuid-1.6.2-43.el8.x86_64 40/134 2023-07-14T19:20:08.203 INFO:teuthology.orchestra.run.smithi101.stdout: Installing : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 87/134 2023-07-14T19:20:08.213 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-jsonpatch-1.21-2.el8.noarch 39/134 2023-07-14T19:20:08.353 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : python3-prettytable-0.7.2-14.el8.noarch 41/134 2023-07-14T19:20:08.372 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : uuid-1.6.2-43.el8.x86_64 40/134 2023-07-14T19:20:08.427 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: uuid-1.6.2-43.el8.x86_64 40/134 2023-07-14T19:20:08.553 INFO:teuthology.orchestra.run.smithi101.stdout: Installing : python3-webtest-2.0.33-1.el8.noarch 88/134 2023-07-14T19:20:08.569 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : automake-1.16.1-7.el8.noarch 42/134 2023-07-14T19:20:08.686 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-prettytable-0.7.2-14.el8.noarch 41/134 2023-07-14T19:20:08.741 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : oniguruma-6.8.2-2.el8.x86_64 43/134 2023-07-14T19:20:08.829 INFO:teuthology.orchestra.run.smithi139.stdout: Running scriptlet: oniguruma-6.8.2-2.el8.x86_64 43/134 2023-07-14T19:20:08.886 INFO:teuthology.orchestra.run.smithi101.stdout: Installing : python3-pecan-1.3.2-9.el8.noarch 89/134 2023-07-14T19:20:08.900 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : automake-1.16.1-7.el8.noarch 42/134 2023-07-14T19:20:09.030 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : oniguruma-6.8.2-2.el8.x86_64 43/134 2023-07-14T19:20:09.037 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : jq-1.6-3.el8.x86_64 44/134 2023-07-14T19:20:09.114 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: oniguruma-6.8.2-2.el8.x86_64 43/134 2023-07-14T19:20:09.141 INFO:teuthology.orchestra.run.smithi101.stdout: Installing : libunwind-1.3.1-3.el8.x86_64 90/134 2023-07-14T19:20:09.187 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : socat-1.7.4.1-1.el8.x86_64 45/134 2023-07-14T19:20:09.326 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : jq-1.6-3.el8.x86_64 44/134 2023-07-14T19:20:09.390 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : libcom_err-devel-1.45.6-4.el8.x86_64 46/134 2023-07-14T19:20:09.484 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : socat-1.7.4.1-1.el8.x86_64 45/134 2023-07-14T19:20:09.559 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : libuuid-devel-2.32.1-35.el8.x86_64 47/134 2023-07-14T19:20:09.646 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : libcom_err-devel-1.45.6-4.el8.x86_64 46/134 2023-07-14T19:20:09.752 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : fuse-2.9.7-15.el8.x86_64 48/134 2023-07-14T19:20:09.849 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : libuuid-devel-2.32.1-35.el8.x86_64 47/134 2023-07-14T19:20:09.894 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : libverto-devel-0.3.0-5.el8.x86_64 49/134 2023-07-14T19:20:09.979 INFO:teuthology.orchestra.run.smithi101.stdout: Installing : gperftools-libs-1:2.7-9.el8.x86_64 91/134 2023-07-14T19:20:10.000 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : fuse-2.9.7-15.el8.x86_64 48/134 2023-07-14T19:20:10.107 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : pcre2-utf16-10.32-2.el8.x86_64 50/134 2023-07-14T19:20:10.128 INFO:teuthology.orchestra.run.smithi101.stdout: Installing : librgw2-2:18.0.0-4940.g81c9387d.el8.x86_64 92/134 2023-07-14T19:20:10.142 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : libverto-devel-0.3.0-5.el8.x86_64 49/134 2023-07-14T19:20:10.196 INFO:teuthology.orchestra.run.smithi101.stdout: Running scriptlet: librgw2-2:18.0.0-4940.g81c9387d.el8.x86_64 92/134 2023-07-14T19:20:10.340 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : python3-pycparser-2.14-14.el8.noarch 51/134 2023-07-14T19:20:10.346 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : pcre2-utf16-10.32-2.el8.x86_64 50/134 2023-07-14T19:20:10.378 INFO:teuthology.orchestra.run.smithi101.stdout: Installing : python3-rgw-2:18.0.0-4940.g81c9387d.el8.x86_64 93/134 2023-07-14T19:20:10.579 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-pycparser-2.14-14.el8.noarch 51/134 2023-07-14T19:20:10.700 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : python3-cffi-1.11.5-5.el8.x86_64 52/134 2023-07-14T19:20:10.930 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-cffi-1.11.5-5.el8.x86_64 52/134 2023-07-14T19:20:11.026 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : python3-cryptography-3.2.1-5.el8.x86_64 53/134 2023-07-14T19:20:11.249 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-cryptography-3.2.1-5.el8.x86_64 53/134 2023-07-14T19:20:11.282 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : python3-jwt-1.6.1-2.el8.noarch 54/134 2023-07-14T19:20:11.476 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-jwt-1.6.1-2.el8.noarch 54/134 2023-07-14T19:20:11.496 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : python3-pyOpenSSL-19.0.0-1.el8.noarch 55/134 2023-07-14T19:20:11.679 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : python3-oauthlib-2.1.0-1.el8.noarch 56/134 2023-07-14T19:20:11.692 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-pyOpenSSL-19.0.0-1.el8.noarch 55/134 2023-07-14T19:20:11.918 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-oauthlib-2.1.0-1.el8.noarch 56/134 2023-07-14T19:20:11.992 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : python3-requests-oauthlib-1.0.0-1.el8.noarch 57/134 2023-07-14T19:20:12.181 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : python3-asyncssh-2.7.0-2.el8.noarch 58/134 2023-07-14T19:20:12.229 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-requests-oauthlib-1.0.0-1.el8.noarch 57/134 2023-07-14T19:20:12.416 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : python3-trustme-0.6.0-4.el8.noarch 59/134 2023-07-14T19:20:12.438 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-asyncssh-2.7.0-2.el8.noarch 58/134 2023-07-14T19:20:12.589 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-trustme-0.6.0-4.el8.noarch 59/134 2023-07-14T19:20:12.645 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : python3-bcrypt-3.1.6-2.el8.1.x86_64 60/134 2023-07-14T19:20:12.793 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-bcrypt-3.1.6-2.el8.1.x86_64 60/134 2023-07-14T19:20:12.870 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : libsepol-devel-2.9-3.el8.x86_64 61/134 2023-07-14T19:20:12.950 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : libsepol-devel-2.9-3.el8.x86_64 61/134 2023-07-14T19:20:13.027 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : keyutils-libs-devel-1.5.10-9.el8.x86_64 62/134 2023-07-14T19:20:13.133 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : keyutils-libs-devel-1.5.10-9.el8.x86_64 62/134 2023-07-14T19:20:13.358 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : pcre2-utf32-10.32-2.el8.x86_64 63/134 2023-07-14T19:20:13.447 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : pcre2-utf32-10.32-2.el8.x86_64 63/134 2023-07-14T19:20:13.604 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : pcre2-devel-10.32-2.el8.x86_64 64/134 2023-07-14T19:20:13.744 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : pcre2-devel-10.32-2.el8.x86_64 64/134 2023-07-14T19:20:13.870 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : libselinux-devel-2.9-5.el8.x86_64 65/134 2023-07-14T19:20:13.976 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : libselinux-devel-2.9-5.el8.x86_64 65/134 2023-07-14T19:20:14.072 INFO:teuthology.orchestra.run.smithi101.stdout: Running scriptlet: ceph-common-2:18.0.0-4940.g81c9387d.el8.x86_64 94/134 2023-07-14T19:20:14.075 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : krb5-devel-1.18.2-14.el8.x86_64 66/134 2023-07-14T19:20:14.181 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : krb5-devel-1.18.2-14.el8.x86_64 66/134 2023-07-14T19:20:14.274 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : xmlstarlet-1.6.1-20.el8.x86_64 67/134 2023-07-14T19:20:14.277 INFO:teuthology.orchestra.run.smithi101.stdout: Installing : ceph-common-2:18.0.0-4940.g81c9387d.el8.x86_64 94/134 2023-07-14T19:20:14.354 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : xmlstarlet-1.6.1-20.el8.x86_64 67/134 2023-07-14T19:20:14.460 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : python3-zc-lockfile-2.0-2.el8.noarch 68/134 2023-07-14T19:20:14.507 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-zc-lockfile-2.0-2.el8.noarch 68/134 2023-07-14T19:20:14.719 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : python3-websocket-client-0.56.0-5.el8.noarch 69/134 2023-07-14T19:20:14.772 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-websocket-client-0.56.0-5.el8.noarch 69/134 2023-07-14T19:20:14.863 INFO:teuthology.orchestra.run.smithi101.stdout: Running scriptlet: ceph-common-2:18.0.0-4940.g81c9387d.el8.x86_64 94/134 2023-07-14T19:20:14.955 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : python3-waitress-1.4.3-1.el8.noarch 70/134 2023-07-14T19:20:15.028 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-waitress-1.4.3-1.el8.noarch 70/134 2023-07-14T19:20:15.102 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : python3-singledispatch-3.4.0.3-18.el8.noarch 71/134 2023-07-14T19:20:15.159 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-singledispatch-3.4.0.3-18.el8.noarch 71/134 2023-07-14T19:20:15.167 INFO:teuthology.orchestra.run.smithi101.stdout: Installing : ceph-base-2:18.0.0-4940.g81c9387d.el8.x86_64 95/134 2023-07-14T19:20:15.248 INFO:teuthology.orchestra.run.smithi101.stdout: Running scriptlet: ceph-base-2:18.0.0-4940.g81c9387d.el8.x86_64 95/134 2023-07-14T19:20:15.285 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : python3-simplegeneric-0.8.1-17.el8.noarch 72/134 2023-07-14T19:20:15.298 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-simplegeneric-0.8.1-17.el8.noarch 72/134 2023-07-14T19:20:15.402 INFO:teuthology.orchestra.run.smithi101.stdout: Installing : ceph-selinux-2:18.0.0-4940.g81c9387d.el8.x86_64 96/134 2023-07-14T19:20:15.492 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : python3-repoze-lru-0.7-6.el8.noarch 73/134 2023-07-14T19:20:15.495 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-repoze-lru-0.7-6.el8.noarch 73/134 2023-07-14T19:20:15.679 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : python3-routes-2.4.1-12.el8.noarch 74/134 2023-07-14T19:20:15.715 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-routes-2.4.1-12.el8.noarch 74/134 2023-07-14T19:20:15.885 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : python3-natsort-7.1.1-2.el8.noarch 75/134 2023-07-14T19:20:15.889 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-natsort-7.1.1-2.el8.noarch 75/134 2023-07-14T19:20:16.032 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-logutils-0.3.5-11.el8.noarch 76/134 2023-07-14T19:20:16.051 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : python3-logutils-0.3.5-11.el8.noarch 76/134 2023-07-14T19:20:16.201 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-jaraco-6.2-6.el8.noarch 77/134 2023-07-14T19:20:16.229 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : python3-jaraco-6.2-6.el8.noarch 77/134 2023-07-14T19:20:16.416 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-jaraco-functools-2.0-4.el8.noarch 78/134 2023-07-14T19:20:16.470 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : python3-jaraco-functools-2.0-4.el8.noarch 78/134 2023-07-14T19:20:16.649 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-cheroot-8.5.2-1.el8.noarch 79/134 2023-07-14T19:20:16.752 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : python3-cheroot-8.5.2-1.el8.noarch 79/134 2023-07-14T19:20:16.869 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-tempora-1.14.1-5.el8.noarch 80/134 2023-07-14T19:20:16.946 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : python3-tempora-1.14.1-5.el8.noarch 80/134 2023-07-14T19:20:17.204 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : python3-portend-2.6-1.el8.noarch 81/134 2023-07-14T19:20:17.375 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-portend-2.6-1.el8.noarch 81/134 2023-07-14T19:20:17.414 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : python3-cherrypy-18.4.0-1.el8.noarch 82/134 2023-07-14T19:20:17.597 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : python3-certifi-2018.10.15-7.el8.noarch 83/134 2023-07-14T19:20:17.620 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-cherrypy-18.4.0-1.el8.noarch 82/134 2023-07-14T19:20:17.770 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-certifi-2018.10.15-7.el8.noarch 83/134 2023-07-14T19:20:17.857 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : python3-cachetools-3.1.1-4.el8.noarch 84/134 2023-07-14T19:20:17.981 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-cachetools-3.1.1-4.el8.noarch 84/134 2023-07-14T19:20:18.967 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : python3-google-auth-1:1.1.1-10.el8.noarch 85/134 2023-07-14T19:20:19.121 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-google-auth-1:1.1.1-10.el8.noarch 85/134 2023-07-14T19:20:19.286 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : python3-kubernetes-1:11.0.0-6.el8.noarch 86/134 2023-07-14T19:20:19.450 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-kubernetes-1:11.0.0-6.el8.noarch 86/134 2023-07-14T19:20:19.533 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 87/134 2023-07-14T19:20:19.695 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 87/134 2023-07-14T19:20:19.883 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : python3-webtest-2.0.33-1.el8.noarch 88/134 2023-07-14T19:20:19.982 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-webtest-2.0.33-1.el8.noarch 88/134 2023-07-14T19:20:20.182 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : python3-pecan-1.3.2-9.el8.noarch 89/134 2023-07-14T19:20:20.254 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-pecan-1.3.2-9.el8.noarch 89/134 2023-07-14T19:20:20.367 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : libunwind-1.3.1-3.el8.x86_64 90/134 2023-07-14T19:20:20.422 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : libunwind-1.3.1-3.el8.x86_64 90/134 2023-07-14T19:20:21.107 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : gperftools-libs-1:2.7-9.el8.x86_64 91/134 2023-07-14T19:20:21.187 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : gperftools-libs-1:2.7-9.el8.x86_64 91/134 2023-07-14T19:20:21.272 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : librgw2-2:18.0.0-4940.g81c9387d.el8.x86_64 92/134 2023-07-14T19:20:21.334 INFO:teuthology.orchestra.run.smithi139.stdout: Running scriptlet: librgw2-2:18.0.0-4940.g81c9387d.el8.x86_64 92/134 2023-07-14T19:20:21.345 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : librgw2-2:18.0.0-4940.g81c9387d.el8.x86_64 92/134 2023-07-14T19:20:21.414 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: librgw2-2:18.0.0-4940.g81c9387d.el8.x86_64 92/134 2023-07-14T19:20:21.489 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : python3-rgw-2:18.0.0-4940.g81c9387d.el8.x86_64 93/134 2023-07-14T19:20:21.579 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-rgw-2:18.0.0-4940.g81c9387d.el8.x86_64 93/134 2023-07-14T19:20:25.129 INFO:teuthology.orchestra.run.smithi139.stdout: Running scriptlet: ceph-common-2:18.0.0-4940.g81c9387d.el8.x86_64 94/134 2023-07-14T19:20:25.229 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: ceph-common-2:18.0.0-4940.g81c9387d.el8.x86_64 94/134 2023-07-14T19:20:25.437 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : ceph-common-2:18.0.0-4940.g81c9387d.el8.x86_64 94/134 2023-07-14T19:20:25.639 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : ceph-common-2:18.0.0-4940.g81c9387d.el8.x86_64 94/134 2023-07-14T19:20:26.015 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: ceph-common-2:18.0.0-4940.g81c9387d.el8.x86_64 94/134 2023-07-14T19:20:26.223 INFO:teuthology.orchestra.run.smithi139.stdout: Running scriptlet: ceph-common-2:18.0.0-4940.g81c9387d.el8.x86_64 94/134 2023-07-14T19:20:26.295 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : ceph-base-2:18.0.0-4940.g81c9387d.el8.x86_64 95/134 2023-07-14T19:20:26.374 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: ceph-base-2:18.0.0-4940.g81c9387d.el8.x86_64 95/134 2023-07-14T19:20:26.498 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : ceph-base-2:18.0.0-4940.g81c9387d.el8.x86_64 95/134 2023-07-14T19:20:26.545 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : ceph-selinux-2:18.0.0-4940.g81c9387d.el8.x86_64 96/134 2023-07-14T19:20:26.613 INFO:teuthology.orchestra.run.smithi139.stdout: Running scriptlet: ceph-base-2:18.0.0-4940.g81c9387d.el8.x86_64 95/134 2023-07-14T19:20:26.754 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : ceph-selinux-2:18.0.0-4940.g81c9387d.el8.x86_64 96/134 2023-07-14T19:20:34.279 INFO:teuthology.orchestra.run.smithi101.stdout: Running scriptlet: ceph-selinux-2:18.0.0-4940.g81c9387d.el8.x86_64 96/134 2023-07-14T19:20:34.279 INFO:teuthology.orchestra.run.smithi101.stdout:skipping the directory /sys 2023-07-14T19:20:34.280 INFO:teuthology.orchestra.run.smithi101.stdout:skipping the directory /proc 2023-07-14T19:20:34.280 INFO:teuthology.orchestra.run.smithi101.stdout:skipping the directory /dev 2023-07-14T19:20:34.280 INFO:teuthology.orchestra.run.smithi101.stdout:skipping the directory /run 2023-07-14T19:20:34.281 INFO:teuthology.orchestra.run.smithi101.stdout:skipping the directory /mnt 2023-07-14T19:20:34.281 INFO:teuthology.orchestra.run.smithi101.stdout:skipping the directory /var/tmp 2023-07-14T19:20:34.281 INFO:teuthology.orchestra.run.smithi101.stdout:skipping the directory /home 2023-07-14T19:20:34.281 INFO:teuthology.orchestra.run.smithi101.stdout:skipping the directory /tmp 2023-07-14T19:20:34.282 INFO:teuthology.orchestra.run.smithi101.stdout:skipping the directory /dev 2023-07-14T19:20:34.282 INFO:teuthology.orchestra.run.smithi101.stdout: 2023-07-14T19:20:34.642 INFO:teuthology.orchestra.run.smithi101.stdout: Installing : ceph-volume-2:18.0.0-4940.g81c9387d.el8.noarch 97/134 2023-07-14T19:20:36.228 INFO:teuthology.orchestra.run.smithi101.stdout: Running scriptlet: ceph-volume-2:18.0.0-4940.g81c9387d.el8.noarch 97/134 2023-07-14T19:20:36.422 INFO:teuthology.orchestra.run.smithi101.stdout: Installing : ceph-osd-2:18.0.0-4940.g81c9387d.el8.x86_64 98/134 2023-07-14T19:20:36.640 INFO:teuthology.orchestra.run.smithi101.stdout: Running scriptlet: ceph-osd-2:18.0.0-4940.g81c9387d.el8.x86_64 98/134 2023-07-14T19:20:36.799 INFO:teuthology.orchestra.run.smithi101.stdout: Installing : ceph-mds-2:18.0.0-4940.g81c9387d.el8.x86_64 99/134 2023-07-14T19:20:37.252 INFO:teuthology.orchestra.run.smithi101.stdout: Running scriptlet: ceph-mds-2:18.0.0-4940.g81c9387d.el8.x86_64 99/134 2023-07-14T19:20:37.441 INFO:teuthology.orchestra.run.smithi101.stdout: Installing : ceph-mon-2:18.0.0-4940.g81c9387d.el8.x86_64 100/134 2023-07-14T19:20:37.458 INFO:teuthology.orchestra.run.smithi101.stdout: Running scriptlet: ceph-mon-2:18.0.0-4940.g81c9387d.el8.x86_64 100/134 2023-07-14T19:20:39.117 INFO:teuthology.orchestra.run.smithi101.stdout: Running scriptlet: cephadm-2:18.0.0-4940.g81c9387d.el8.noarch 101/134 2023-07-14T19:20:39.282 INFO:teuthology.orchestra.run.smithi101.stdout: Installing : cephadm-2:18.0.0-4940.g81c9387d.el8.noarch 101/134 2023-07-14T19:20:39.448 INFO:teuthology.orchestra.run.smithi101.stdout: Installing : ceph-prometheus-alerts-2:18.0.0-4940.g81c9387d.e 102/134 2023-07-14T19:20:39.657 INFO:teuthology.orchestra.run.smithi101.stdout: Installing : ceph-grafana-dashboards-2:18.0.0-4940.g81c9387d. 103/134 2023-07-14T19:20:39.848 INFO:teuthology.orchestra.run.smithi101.stdout: Installing : ceph-mgr-cephadm-2:18.0.0-4940.g81c9387d.el8.noa 104/134 2023-07-14T19:20:40.064 INFO:teuthology.orchestra.run.smithi101.stdout: Running scriptlet: ceph-mgr-cephadm-2:18.0.0-4940.g81c9387d.el8.noa 104/134 2023-07-14T19:20:40.363 INFO:teuthology.orchestra.run.smithi101.stdout: Installing : ceph-mgr-dashboard-2:18.0.0-4940.g81c9387d.el8.n 105/134 2023-07-14T19:20:41.318 INFO:teuthology.orchestra.run.smithi101.stdout: Running scriptlet: ceph-mgr-dashboard-2:18.0.0-4940.g81c9387d.el8.n 105/134 2023-07-14T19:20:41.515 INFO:teuthology.orchestra.run.smithi101.stdout: Installing : ceph-mgr-diskprediction-local-2:18.0.0-4940.g81c 106/134 2023-07-14T19:20:41.606 INFO:teuthology.orchestra.run.smithi101.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:18.0.0-4940.g81c 106/134 2023-07-14T19:20:41.957 INFO:teuthology.orchestra.run.smithi101.stdout: Installing : ceph-mgr-modules-core-2:18.0.0-4940.g81c9387d.el 107/134 2023-07-14T19:20:42.166 INFO:teuthology.orchestra.run.smithi101.stdout: Installing : ceph-mgr-2:18.0.0-4940.g81c9387d.el8.x86_64 108/134 2023-07-14T19:20:42.200 INFO:teuthology.orchestra.run.smithi101.stdout: Running scriptlet: ceph-mgr-2:18.0.0-4940.g81c9387d.el8.x86_64 108/134 2023-07-14T19:20:42.382 INFO:teuthology.orchestra.run.smithi101.stdout: Installing : ceph-mgr-rook-2:18.0.0-4940.g81c9387d.el8.noarch 109/134 2023-07-14T19:20:42.392 INFO:teuthology.orchestra.run.smithi101.stdout: Running scriptlet: ceph-mgr-rook-2:18.0.0-4940.g81c9387d.el8.noarch 109/134 2023-07-14T19:20:43.700 INFO:teuthology.orchestra.run.smithi101.stdout: Installing : ceph-2:18.0.0-4940.g81c9387d.el8.x86_64 110/134 2023-07-14T19:20:43.933 INFO:teuthology.orchestra.run.smithi101.stdout: Installing : ceph-radosgw-2:18.0.0-4940.g81c9387d.el8.x86_64 111/134 2023-07-14T19:20:43.967 INFO:teuthology.orchestra.run.smithi101.stdout: Running scriptlet: ceph-radosgw-2:18.0.0-4940.g81c9387d.el8.x86_64 111/134 2023-07-14T19:20:44.107 INFO:teuthology.orchestra.run.smithi101.stdout: Installing : ceph-immutable-object-cache-2:18.0.0-4940.g81c93 112/134 2023-07-14T19:20:44.148 INFO:teuthology.orchestra.run.smithi101.stdout: Running scriptlet: ceph-immutable-object-cache-2:18.0.0-4940.g81c93 112/134 2023-07-14T19:20:44.332 INFO:teuthology.orchestra.run.smithi101.stdout: Installing : cephfs-mirror-2:18.0.0-4940.g81c9387d.el8.x86_64 113/134 2023-07-14T19:20:44.683 INFO:teuthology.orchestra.run.smithi101.stdout: Running scriptlet: cephfs-mirror-2:18.0.0-4940.g81c9387d.el8.x86_64 113/134 2023-07-14T19:20:44.848 INFO:teuthology.orchestra.run.smithi101.stdout: Installing : rbd-mirror-2:18.0.0-4940.g81c9387d.el8.x86_64 114/134 2023-07-14T19:20:45.417 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: ceph-selinux-2:18.0.0-4940.g81c9387d.el8.x86_64 96/134 2023-07-14T19:20:45.418 INFO:teuthology.orchestra.run.smithi098.stdout:skipping the directory /sys 2023-07-14T19:20:45.418 INFO:teuthology.orchestra.run.smithi098.stdout:skipping the directory /proc 2023-07-14T19:20:45.418 INFO:teuthology.orchestra.run.smithi098.stdout:skipping the directory /dev 2023-07-14T19:20:45.419 INFO:teuthology.orchestra.run.smithi098.stdout:skipping the directory /run 2023-07-14T19:20:45.419 INFO:teuthology.orchestra.run.smithi098.stdout:skipping the directory /mnt 2023-07-14T19:20:45.419 INFO:teuthology.orchestra.run.smithi098.stdout:skipping the directory /var/tmp 2023-07-14T19:20:45.419 INFO:teuthology.orchestra.run.smithi098.stdout:skipping the directory /home 2023-07-14T19:20:45.419 INFO:teuthology.orchestra.run.smithi098.stdout:skipping the directory /tmp 2023-07-14T19:20:45.420 INFO:teuthology.orchestra.run.smithi098.stdout:skipping the directory /dev 2023-07-14T19:20:45.420 INFO:teuthology.orchestra.run.smithi098.stdout: 2023-07-14T19:20:45.763 INFO:teuthology.orchestra.run.smithi139.stdout: Running scriptlet: ceph-selinux-2:18.0.0-4940.g81c9387d.el8.x86_64 96/134 2023-07-14T19:20:45.764 INFO:teuthology.orchestra.run.smithi139.stdout:skipping the directory /sys 2023-07-14T19:20:45.764 INFO:teuthology.orchestra.run.smithi139.stdout:skipping the directory /proc 2023-07-14T19:20:45.764 INFO:teuthology.orchestra.run.smithi139.stdout:skipping the directory /dev 2023-07-14T19:20:45.765 INFO:teuthology.orchestra.run.smithi139.stdout:skipping the directory /run 2023-07-14T19:20:45.765 INFO:teuthology.orchestra.run.smithi139.stdout:skipping the directory /mnt 2023-07-14T19:20:45.765 INFO:teuthology.orchestra.run.smithi139.stdout:skipping the directory /var/tmp 2023-07-14T19:20:45.765 INFO:teuthology.orchestra.run.smithi139.stdout:skipping the directory /home 2023-07-14T19:20:45.766 INFO:teuthology.orchestra.run.smithi139.stdout:skipping the directory /tmp 2023-07-14T19:20:45.766 INFO:teuthology.orchestra.run.smithi139.stdout:skipping the directory /dev 2023-07-14T19:20:45.766 INFO:teuthology.orchestra.run.smithi139.stdout: 2023-07-14T19:20:45.792 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : ceph-volume-2:18.0.0-4940.g81c9387d.el8.noarch 97/134 2023-07-14T19:20:46.138 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : ceph-volume-2:18.0.0-4940.g81c9387d.el8.noarch 97/134 2023-07-14T19:20:47.368 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: ceph-volume-2:18.0.0-4940.g81c9387d.el8.noarch 97/134 2023-07-14T19:20:47.592 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : ceph-osd-2:18.0.0-4940.g81c9387d.el8.x86_64 98/134 2023-07-14T19:20:47.725 INFO:teuthology.orchestra.run.smithi139.stdout: Running scriptlet: ceph-volume-2:18.0.0-4940.g81c9387d.el8.noarch 97/134 2023-07-14T19:20:47.811 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: ceph-osd-2:18.0.0-4940.g81c9387d.el8.x86_64 98/134 2023-07-14T19:20:47.904 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : ceph-osd-2:18.0.0-4940.g81c9387d.el8.x86_64 98/134 2023-07-14T19:20:47.975 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : ceph-mds-2:18.0.0-4940.g81c9387d.el8.x86_64 99/134 2023-07-14T19:20:48.121 INFO:teuthology.orchestra.run.smithi139.stdout: Running scriptlet: ceph-osd-2:18.0.0-4940.g81c9387d.el8.x86_64 98/134 2023-07-14T19:20:48.284 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : ceph-mds-2:18.0.0-4940.g81c9387d.el8.x86_64 99/134 2023-07-14T19:20:48.427 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: ceph-mds-2:18.0.0-4940.g81c9387d.el8.x86_64 99/134 2023-07-14T19:20:48.592 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : ceph-mon-2:18.0.0-4940.g81c9387d.el8.x86_64 100/134 2023-07-14T19:20:48.608 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: ceph-mon-2:18.0.0-4940.g81c9387d.el8.x86_64 100/134 2023-07-14T19:20:48.738 INFO:teuthology.orchestra.run.smithi139.stdout: Running scriptlet: ceph-mds-2:18.0.0-4940.g81c9387d.el8.x86_64 99/134 2023-07-14T19:20:48.910 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : ceph-mon-2:18.0.0-4940.g81c9387d.el8.x86_64 100/134 2023-07-14T19:20:48.926 INFO:teuthology.orchestra.run.smithi139.stdout: Running scriptlet: ceph-mon-2:18.0.0-4940.g81c9387d.el8.x86_64 100/134 2023-07-14T19:20:49.402 INFO:teuthology.orchestra.run.smithi101.stdout: Running scriptlet: rbd-mirror-2:18.0.0-4940.g81c9387d.el8.x86_64 114/134 2023-07-14T19:20:50.281 INFO:teuthology.orchestra.run.smithi101.stdout: Installing : ceph-test-2:18.0.0-4940.g81c9387d.el8.x86_64 115/134 2023-07-14T19:20:50.415 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: cephadm-2:18.0.0-4940.g81c9387d.el8.noarch 101/134 2023-07-14T19:20:50.577 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : cephadm-2:18.0.0-4940.g81c9387d.el8.noarch 101/134 2023-07-14T19:20:50.580 INFO:teuthology.orchestra.run.smithi139.stdout: Running scriptlet: cephadm-2:18.0.0-4940.g81c9387d.el8.noarch 101/134 2023-07-14T19:20:50.620 INFO:teuthology.orchestra.run.smithi101.stdout: Installing : openssl-devel-1:1.1.1k-7.el8_6.x86_64 116/134 2023-07-14T19:20:50.698 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : ceph-prometheus-alerts-2:18.0.0-4940.g81c9387d.e 102/134 2023-07-14T19:20:50.746 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : cephadm-2:18.0.0-4940.g81c9387d.el8.noarch 101/134 2023-07-14T19:20:50.825 INFO:teuthology.orchestra.run.smithi101.stdout: Installing : ceph-fuse-2:18.0.0-4940.g81c9387d.el8.x86_64 117/134 2023-07-14T19:20:50.865 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : ceph-prometheus-alerts-2:18.0.0-4940.g81c9387d.e 102/134 2023-07-14T19:20:50.898 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : ceph-grafana-dashboards-2:18.0.0-4940.g81c9387d. 103/134 2023-07-14T19:20:51.079 INFO:teuthology.orchestra.run.smithi101.stdout: Installing : xfsprogs-devel-5.0.0-10.el8.x86_64 118/134 2023-07-14T19:20:51.091 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : ceph-mgr-cephadm-2:18.0.0-4940.g81c9387d.el8.noa 104/134 2023-07-14T19:20:51.099 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : ceph-grafana-dashboards-2:18.0.0-4940.g81c9387d. 103/134 2023-07-14T19:20:51.275 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : ceph-mgr-cephadm-2:18.0.0-4940.g81c9387d.el8.noa 104/134 2023-07-14T19:20:51.280 INFO:teuthology.orchestra.run.smithi101.stdout: Installing : libtool-2.4.6-25.el8.x86_64 119/134 2023-07-14T19:20:51.306 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: ceph-mgr-cephadm-2:18.0.0-4940.g81c9387d.el8.noa 104/134 2023-07-14T19:20:51.474 INFO:teuthology.orchestra.run.smithi101.stdout: Running scriptlet: libtool-2.4.6-25.el8.x86_64 119/134 2023-07-14T19:20:51.494 INFO:teuthology.orchestra.run.smithi139.stdout: Running scriptlet: ceph-mgr-cephadm-2:18.0.0-4940.g81c9387d.el8.noa 104/134 2023-07-14T19:20:51.625 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : ceph-mgr-dashboard-2:18.0.0-4940.g81c9387d.el8.n 105/134 2023-07-14T19:20:51.751 INFO:teuthology.orchestra.run.smithi101.stdout: Installing : postgresql-contrib-10.21-2.module+el8.6.0+15342+ 120/134 2023-07-14T19:20:51.818 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : ceph-mgr-dashboard-2:18.0.0-4940.g81c9387d.el8.n 105/134 2023-07-14T19:20:51.926 INFO:teuthology.orchestra.run.smithi101.stdout: Installing : libcephfs-devel-2:18.0.0-4940.g81c9387d.el8.x86_ 121/134 2023-07-14T19:20:52.099 INFO:teuthology.orchestra.run.smithi101.stdout: Installing : elfutils-libelf-devel-0.186-1.el8.x86_64 122/134 2023-07-14T19:20:52.580 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: ceph-mgr-dashboard-2:18.0.0-4940.g81c9387d.el8.n 105/134 2023-07-14T19:20:52.777 INFO:teuthology.orchestra.run.smithi139.stdout: Running scriptlet: ceph-mgr-dashboard-2:18.0.0-4940.g81c9387d.el8.n 105/134 2023-07-14T19:20:52.791 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : ceph-mgr-diskprediction-local-2:18.0.0-4940.g81c 106/134 2023-07-14T19:20:52.882 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:18.0.0-4940.g81c 106/134 2023-07-14T19:20:52.933 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : ceph-mgr-diskprediction-local-2:18.0.0-4940.g81c 106/134 2023-07-14T19:20:53.025 INFO:teuthology.orchestra.run.smithi139.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:18.0.0-4940.g81c 106/134 2023-07-14T19:20:53.211 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : ceph-mgr-modules-core-2:18.0.0-4940.g81c9387d.el 107/134 2023-07-14T19:20:53.404 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : ceph-mgr-modules-core-2:18.0.0-4940.g81c9387d.el 107/134 2023-07-14T19:20:53.447 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : ceph-mgr-2:18.0.0-4940.g81c9387d.el8.x86_64 108/134 2023-07-14T19:20:53.481 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: ceph-mgr-2:18.0.0-4940.g81c9387d.el8.x86_64 108/134 2023-07-14T19:20:53.616 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : ceph-mgr-rook-2:18.0.0-4940.g81c9387d.el8.noarch 109/134 2023-07-14T19:20:53.626 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: ceph-mgr-rook-2:18.0.0-4940.g81c9387d.el8.noarch 109/134 2023-07-14T19:20:53.652 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : ceph-mgr-2:18.0.0-4940.g81c9387d.el8.x86_64 108/134 2023-07-14T19:20:53.687 INFO:teuthology.orchestra.run.smithi139.stdout: Running scriptlet: ceph-mgr-2:18.0.0-4940.g81c9387d.el8.x86_64 108/134 2023-07-14T19:20:53.850 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : ceph-mgr-rook-2:18.0.0-4940.g81c9387d.el8.noarch 109/134 2023-07-14T19:20:53.861 INFO:teuthology.orchestra.run.smithi139.stdout: Running scriptlet: ceph-mgr-rook-2:18.0.0-4940.g81c9387d.el8.noarch 109/134 2023-07-14T19:20:54.311 INFO:teuthology.orchestra.run.smithi101.stdout: Running scriptlet: postgresql-server-10.21-2.module+el8.6.0+15342+5 123/134 2023-07-14T19:20:54.605 INFO:teuthology.orchestra.run.smithi101.stdout: Installing : postgresql-server-10.21-2.module+el8.6.0+15342+5 123/134 2023-07-14T19:20:54.623 INFO:teuthology.orchestra.run.smithi101.stdout: Running scriptlet: postgresql-server-10.21-2.module+el8.6.0+15342+5 123/134 2023-07-14T19:20:54.795 INFO:teuthology.orchestra.run.smithi101.stdout: Installing : cephfs-top-2:18.0.0-4940.g81c9387d.el8.noarch 124/134 2023-07-14T19:20:54.932 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : ceph-2:18.0.0-4940.g81c9387d.el8.x86_64 110/134 2023-07-14T19:20:55.088 INFO:teuthology.orchestra.run.smithi101.stdout: Installing : rbd-fuse-2:18.0.0-4940.g81c9387d.el8.x86_64 125/134 2023-07-14T19:20:55.103 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : ceph-radosgw-2:18.0.0-4940.g81c9387d.el8.x86_64 111/134 2023-07-14T19:20:55.136 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: ceph-radosgw-2:18.0.0-4940.g81c9387d.el8.x86_64 111/134 2023-07-14T19:20:55.171 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : ceph-2:18.0.0-4940.g81c9387d.el8.x86_64 110/134 2023-07-14T19:20:55.308 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : ceph-immutable-object-cache-2:18.0.0-4940.g81c93 112/134 2023-07-14T19:20:55.349 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: ceph-immutable-object-cache-2:18.0.0-4940.g81c93 112/134 2023-07-14T19:20:55.356 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : ceph-radosgw-2:18.0.0-4940.g81c9387d.el8.x86_64 111/134 2023-07-14T19:20:55.389 INFO:teuthology.orchestra.run.smithi139.stdout: Running scriptlet: ceph-radosgw-2:18.0.0-4940.g81c9387d.el8.x86_64 111/134 2023-07-14T19:20:55.391 INFO:teuthology.orchestra.run.smithi101.stdout: Installing : rbd-nbd-2:18.0.0-4940.g81c9387d.el8.x86_64 126/134 2023-07-14T19:20:55.491 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : cephfs-mirror-2:18.0.0-4940.g81c9387d.el8.x86_64 113/134 2023-07-14T19:20:55.550 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : ceph-immutable-object-cache-2:18.0.0-4940.g81c93 112/134 2023-07-14T19:20:55.591 INFO:teuthology.orchestra.run.smithi139.stdout: Running scriptlet: ceph-immutable-object-cache-2:18.0.0-4940.g81c93 112/134 2023-07-14T19:20:55.750 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : cephfs-mirror-2:18.0.0-4940.g81c9387d.el8.x86_64 113/134 2023-07-14T19:20:55.791 INFO:teuthology.orchestra.run.smithi101.stdout: Upgrading : iproute-5.15.0-4.el8_6.1.x86_64 127/134 2023-07-14T19:20:55.842 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: cephfs-mirror-2:18.0.0-4940.g81c9387d.el8.x86_64 113/134 2023-07-14T19:20:55.971 INFO:teuthology.orchestra.run.smithi101.stdout: Installing : flex-2.6.1-9.el8.x86_64 128/134 2023-07-14T19:20:55.974 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : rbd-mirror-2:18.0.0-4940.g81c9387d.el8.x86_64 114/134 2023-07-14T19:20:56.053 INFO:teuthology.orchestra.run.smithi101.stdout: Running scriptlet: flex-2.6.1-9.el8.x86_64 128/134 2023-07-14T19:20:56.102 INFO:teuthology.orchestra.run.smithi139.stdout: Running scriptlet: cephfs-mirror-2:18.0.0-4940.g81c9387d.el8.x86_64 113/134 2023-07-14T19:20:56.233 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : rbd-mirror-2:18.0.0-4940.g81c9387d.el8.x86_64 114/134 2023-07-14T19:20:56.272 INFO:teuthology.orchestra.run.smithi101.stdout: Installing : bison-3.0.4-10.el8.x86_64 129/134 2023-07-14T19:20:56.284 INFO:teuthology.orchestra.run.smithi101.stdout: Running scriptlet: bison-3.0.4-10.el8.x86_64 129/134 2023-07-14T19:20:56.438 INFO:teuthology.orchestra.run.smithi101.stdout: Installing : NetworkManager-initscripts-updown-1:1.36.0-9.el8 130/134 2023-07-14T19:20:56.446 INFO:teuthology.orchestra.run.smithi101.stdout: Running scriptlet: NetworkManager-initscripts-updown-1:1.36.0-9.el8 130/134 2023-07-14T19:20:56.615 INFO:teuthology.orchestra.run.smithi101.stdout: Installing : libaio-devel-0.3.112-1.el8.x86_64 131/134 2023-07-14T19:20:56.616 INFO:teuthology.orchestra.run.smithi101.stdout: Cleanup : librbd1-1:12.2.7-9.el8.x86_64 132/134 2023-07-14T19:20:56.800 INFO:teuthology.orchestra.run.smithi101.stdout: Running scriptlet: librbd1-1:12.2.7-9.el8.x86_64 132/134 2023-07-14T19:20:56.801 INFO:teuthology.orchestra.run.smithi101.stdout: Cleanup : librados2-1:12.2.7-9.el8.x86_64 133/134 2023-07-14T19:20:57.071 INFO:teuthology.orchestra.run.smithi101.stdout: Running scriptlet: librados2-1:12.2.7-9.el8.x86_64 133/134 2023-07-14T19:20:57.254 INFO:teuthology.orchestra.run.smithi101.stdout: Cleanup : iproute-5.15.0-4.el8.x86_64 134/134 2023-07-14T19:20:57.793 INFO:teuthology.orchestra.run.smithi101.stdout: Running scriptlet: iproute-5.15.0-4.el8.x86_64 134/134 2023-07-14T19:20:57.793 INFO:teuthology.orchestra.run.smithi101.stdout: Verifying : ceph-2:18.0.0-4940.g81c9387d.el8.x86_64 1/134 2023-07-14T19:20:57.793 INFO:teuthology.orchestra.run.smithi101.stdout: Verifying : ceph-base-2:18.0.0-4940.g81c9387d.el8.x86_64 2/134 2023-07-14T19:20:57.793 INFO:teuthology.orchestra.run.smithi101.stdout: Verifying : ceph-common-2:18.0.0-4940.g81c9387d.el8.x86_64 3/134 2023-07-14T19:20:57.793 INFO:teuthology.orchestra.run.smithi101.stdout: Verifying : ceph-fuse-2:18.0.0-4940.g81c9387d.el8.x86_64 4/134 2023-07-14T19:20:57.794 INFO:teuthology.orchestra.run.smithi101.stdout: Verifying : ceph-immutable-object-cache-2:18.0.0-4940.g81c93 5/134 2023-07-14T19:20:57.794 INFO:teuthology.orchestra.run.smithi101.stdout: Verifying : ceph-mds-2:18.0.0-4940.g81c9387d.el8.x86_64 6/134 2023-07-14T19:20:57.794 INFO:teuthology.orchestra.run.smithi101.stdout: Verifying : ceph-mgr-2:18.0.0-4940.g81c9387d.el8.x86_64 7/134 2023-07-14T19:20:57.794 INFO:teuthology.orchestra.run.smithi101.stdout: Verifying : ceph-mon-2:18.0.0-4940.g81c9387d.el8.x86_64 8/134 2023-07-14T19:20:57.794 INFO:teuthology.orchestra.run.smithi101.stdout: Verifying : ceph-osd-2:18.0.0-4940.g81c9387d.el8.x86_64 9/134 2023-07-14T19:20:57.794 INFO:teuthology.orchestra.run.smithi101.stdout: Verifying : ceph-radosgw-2:18.0.0-4940.g81c9387d.el8.x86_64 10/134 2023-07-14T19:20:57.794 INFO:teuthology.orchestra.run.smithi101.stdout: Verifying : ceph-selinux-2:18.0.0-4940.g81c9387d.el8.x86_64 11/134 2023-07-14T19:20:57.794 INFO:teuthology.orchestra.run.smithi101.stdout: Verifying : ceph-test-2:18.0.0-4940.g81c9387d.el8.x86_64 12/134 2023-07-14T19:20:57.795 INFO:teuthology.orchestra.run.smithi101.stdout: Verifying : cephfs-mirror-2:18.0.0-4940.g81c9387d.el8.x86_64 13/134 2023-07-14T19:20:57.795 INFO:teuthology.orchestra.run.smithi101.stdout: Verifying : libcephfs-devel-2:18.0.0-4940.g81c9387d.el8.x86_ 14/134 2023-07-14T19:20:57.795 INFO:teuthology.orchestra.run.smithi101.stdout: Verifying : libcephfs2-2:18.0.0-4940.g81c9387d.el8.x86_64 15/134 2023-07-14T19:20:57.795 INFO:teuthology.orchestra.run.smithi101.stdout: Verifying : libcephsqlite-2:18.0.0-4940.g81c9387d.el8.x86_64 16/134 2023-07-14T19:20:57.795 INFO:teuthology.orchestra.run.smithi101.stdout: Verifying : librados-devel-2:18.0.0-4940.g81c9387d.el8.x86_6 17/134 2023-07-14T19:20:57.795 INFO:teuthology.orchestra.run.smithi101.stdout: Verifying : libradosstriper1-2:18.0.0-4940.g81c9387d.el8.x86 18/134 2023-07-14T19:20:57.796 INFO:teuthology.orchestra.run.smithi101.stdout: Verifying : librgw2-2:18.0.0-4940.g81c9387d.el8.x86_64 19/134 2023-07-14T19:20:57.796 INFO:teuthology.orchestra.run.smithi101.stdout: Verifying : python3-ceph-argparse-2:18.0.0-4940.g81c9387d.el 20/134 2023-07-14T19:20:57.796 INFO:teuthology.orchestra.run.smithi101.stdout: Verifying : python3-ceph-common-2:18.0.0-4940.g81c9387d.el8. 21/134 2023-07-14T19:20:57.796 INFO:teuthology.orchestra.run.smithi101.stdout: Verifying : python3-cephfs-2:18.0.0-4940.g81c9387d.el8.x86_6 22/134 2023-07-14T19:20:57.796 INFO:teuthology.orchestra.run.smithi101.stdout: Verifying : python3-rados-2:18.0.0-4940.g81c9387d.el8.x86_64 23/134 2023-07-14T19:20:57.797 INFO:teuthology.orchestra.run.smithi101.stdout: Verifying : python3-rbd-2:18.0.0-4940.g81c9387d.el8.x86_64 24/134 2023-07-14T19:20:57.797 INFO:teuthology.orchestra.run.smithi101.stdout: Verifying : python3-rgw-2:18.0.0-4940.g81c9387d.el8.x86_64 25/134 2023-07-14T19:20:57.797 INFO:teuthology.orchestra.run.smithi101.stdout: Verifying : rbd-fuse-2:18.0.0-4940.g81c9387d.el8.x86_64 26/134 2023-07-14T19:20:57.797 INFO:teuthology.orchestra.run.smithi101.stdout: Verifying : rbd-mirror-2:18.0.0-4940.g81c9387d.el8.x86_64 27/134 2023-07-14T19:20:57.797 INFO:teuthology.orchestra.run.smithi101.stdout: Verifying : rbd-nbd-2:18.0.0-4940.g81c9387d.el8.x86_64 28/134 2023-07-14T19:20:57.797 INFO:teuthology.orchestra.run.smithi101.stdout: Verifying : ceph-grafana-dashboards-2:18.0.0-4940.g81c9387d. 29/134 2023-07-14T19:20:57.804 INFO:teuthology.orchestra.run.smithi101.stdout: Verifying : ceph-mgr-cephadm-2:18.0.0-4940.g81c9387d.el8.noa 30/134 2023-07-14T19:20:57.804 INFO:teuthology.orchestra.run.smithi101.stdout: Verifying : ceph-mgr-dashboard-2:18.0.0-4940.g81c9387d.el8.n 31/134 2023-07-14T19:20:57.805 INFO:teuthology.orchestra.run.smithi101.stdout: Verifying : ceph-mgr-diskprediction-local-2:18.0.0-4940.g81c 32/134 2023-07-14T19:20:57.805 INFO:teuthology.orchestra.run.smithi101.stdout: Verifying : ceph-mgr-modules-core-2:18.0.0-4940.g81c9387d.el 33/134 2023-07-14T19:20:57.805 INFO:teuthology.orchestra.run.smithi101.stdout: Verifying : ceph-mgr-rook-2:18.0.0-4940.g81c9387d.el8.noarch 34/134 2023-07-14T19:20:57.805 INFO:teuthology.orchestra.run.smithi101.stdout: Verifying : ceph-prometheus-alerts-2:18.0.0-4940.g81c9387d.e 35/134 2023-07-14T19:20:57.805 INFO:teuthology.orchestra.run.smithi101.stdout: Verifying : ceph-volume-2:18.0.0-4940.g81c9387d.el8.noarch 36/134 2023-07-14T19:20:57.805 INFO:teuthology.orchestra.run.smithi101.stdout: Verifying : cephadm-2:18.0.0-4940.g81c9387d.el8.noarch 37/134 2023-07-14T19:20:57.806 INFO:teuthology.orchestra.run.smithi101.stdout: Verifying : cephfs-top-2:18.0.0-4940.g81c9387d.el8.noarch 38/134 2023-07-14T19:20:57.806 INFO:teuthology.orchestra.run.smithi101.stdout: Verifying : gperftools-libs-1:2.7-9.el8.x86_64 39/134 2023-07-14T19:20:57.806 INFO:teuthology.orchestra.run.smithi101.stdout: Verifying : liboath-2.6.2-3.el8.x86_64 40/134 2023-07-14T19:20:57.806 INFO:teuthology.orchestra.run.smithi101.stdout: Verifying : libunwind-1.3.1-3.el8.x86_64 41/134 2023-07-14T19:20:57.806 INFO:teuthology.orchestra.run.smithi101.stdout: Verifying : python3-asyncssh-2.7.0-2.el8.noarch 42/134 2023-07-14T19:20:57.807 INFO:teuthology.orchestra.run.smithi101.stdout: Verifying : python3-bcrypt-3.1.6-2.el8.1.x86_64 43/134 2023-07-14T19:20:57.807 INFO:teuthology.orchestra.run.smithi101.stdout: Verifying : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 44/134 2023-07-14T19:20:57.807 INFO:teuthology.orchestra.run.smithi101.stdout: Verifying : python3-cachetools-3.1.1-4.el8.noarch 45/134 2023-07-14T19:20:57.807 INFO:teuthology.orchestra.run.smithi101.stdout: Verifying : python3-certifi-2018.10.15-7.el8.noarch 46/134 2023-07-14T19:20:57.808 INFO:teuthology.orchestra.run.smithi101.stdout: Verifying : python3-cheroot-8.5.2-1.el8.noarch 47/134 2023-07-14T19:20:57.808 INFO:teuthology.orchestra.run.smithi101.stdout: Verifying : python3-cherrypy-18.4.0-1.el8.noarch 48/134 2023-07-14T19:20:57.808 INFO:teuthology.orchestra.run.smithi101.stdout: Verifying : python3-google-auth-1:1.1.1-10.el8.noarch 49/134 2023-07-14T19:20:57.808 INFO:teuthology.orchestra.run.smithi101.stdout: Verifying : python3-jaraco-6.2-6.el8.noarch 50/134 2023-07-14T19:20:57.808 INFO:teuthology.orchestra.run.smithi101.stdout: Verifying : python3-jaraco-functools-2.0-4.el8.noarch 51/134 2023-07-14T19:20:57.808 INFO:teuthology.orchestra.run.smithi101.stdout: Verifying : python3-kubernetes-1:11.0.0-6.el8.noarch 52/134 2023-07-14T19:20:57.809 INFO:teuthology.orchestra.run.smithi101.stdout: Verifying : python3-logutils-0.3.5-11.el8.noarch 53/134 2023-07-14T19:20:57.809 INFO:teuthology.orchestra.run.smithi101.stdout: Verifying : python3-more-itertools-7.2.0-3.el8.noarch 54/134 2023-07-14T19:20:57.809 INFO:teuthology.orchestra.run.smithi101.stdout: Verifying : python3-natsort-7.1.1-2.el8.noarch 55/134 2023-07-14T19:20:57.809 INFO:teuthology.orchestra.run.smithi101.stdout: Verifying : python3-pecan-1.3.2-9.el8.noarch 56/134 2023-07-14T19:20:57.809 INFO:teuthology.orchestra.run.smithi101.stdout: Verifying : python3-portend-2.6-1.el8.noarch 57/134 2023-07-14T19:20:57.809 INFO:teuthology.orchestra.run.smithi101.stdout: Verifying : python3-repoze-lru-0.7-6.el8.noarch 58/134 2023-07-14T19:20:57.810 INFO:teuthology.orchestra.run.smithi101.stdout: Verifying : python3-routes-2.4.1-12.el8.noarch 59/134 2023-07-14T19:20:57.810 INFO:teuthology.orchestra.run.smithi101.stdout: Verifying : python3-rsa-4.8-1.el8.noarch 60/134 2023-07-14T19:20:57.810 INFO:teuthology.orchestra.run.smithi101.stdout: Verifying : python3-simplegeneric-0.8.1-17.el8.noarch 61/134 2023-07-14T19:20:57.810 INFO:teuthology.orchestra.run.smithi101.stdout: Verifying : python3-singledispatch-3.4.0.3-18.el8.noarch 62/134 2023-07-14T19:20:57.810 INFO:teuthology.orchestra.run.smithi101.stdout: Verifying : python3-tempora-1.14.1-5.el8.noarch 63/134 2023-07-14T19:20:57.810 INFO:teuthology.orchestra.run.smithi101.stdout: Verifying : python3-trustme-0.6.0-4.el8.noarch 64/134 2023-07-14T19:20:57.810 INFO:teuthology.orchestra.run.smithi101.stdout: Verifying : python3-waitress-1.4.3-1.el8.noarch 65/134 2023-07-14T19:20:57.810 INFO:teuthology.orchestra.run.smithi101.stdout: Verifying : python3-webob-1.8.5-1.el8.1.noarch 66/134 2023-07-14T19:20:57.811 INFO:teuthology.orchestra.run.smithi101.stdout: Verifying : python3-websocket-client-0.56.0-5.el8.noarch 67/134 2023-07-14T19:20:57.811 INFO:teuthology.orchestra.run.smithi101.stdout: Verifying : python3-webtest-2.0.33-1.el8.noarch 68/134 2023-07-14T19:20:57.811 INFO:teuthology.orchestra.run.smithi101.stdout: Verifying : python3-zc-lockfile-2.0-2.el8.noarch 69/134 2023-07-14T19:20:57.811 INFO:teuthology.orchestra.run.smithi101.stdout: Verifying : thrift-0.13.0-2.el8.x86_64 70/134 2023-07-14T19:20:57.811 INFO:teuthology.orchestra.run.smithi101.stdout: Verifying : xmlstarlet-1.6.1-20.el8.x86_64 71/134 2023-07-14T19:20:57.811 INFO:teuthology.orchestra.run.smithi101.stdout: Verifying : librabbitmq-0.9.0-3.el8.x86_64 72/134 2023-07-14T19:20:57.811 INFO:teuthology.orchestra.run.smithi101.stdout: Verifying : libselinux-devel-2.9-5.el8.x86_64 73/134 2023-07-14T19:20:57.812 INFO:teuthology.orchestra.run.smithi101.stdout: Verifying : python3-jwt-1.6.1-2.el8.noarch 74/134 2023-07-14T19:20:57.812 INFO:teuthology.orchestra.run.smithi101.stdout: Verifying : pcre2-utf32-10.32-2.el8.x86_64 75/134 2023-07-14T19:20:57.812 INFO:teuthology.orchestra.run.smithi101.stdout: Verifying : keyutils-libs-devel-1.5.10-9.el8.x86_64 76/134 2023-07-14T19:20:57.812 INFO:teuthology.orchestra.run.smithi101.stdout: Verifying : libsepol-devel-2.9-3.el8.x86_64 77/134 2023-07-14T19:20:57.812 INFO:teuthology.orchestra.run.smithi101.stdout: Verifying : python3-requests-oauthlib-1.0.0-1.el8.noarch 78/134 2023-07-14T19:20:57.812 INFO:teuthology.orchestra.run.smithi101.stdout: Verifying : python3-pycparser-2.14-14.el8.noarch 79/134 2023-07-14T19:20:57.812 INFO:teuthology.orchestra.run.smithi101.stdout: Verifying : pcre2-utf16-10.32-2.el8.x86_64 80/134 2023-07-14T19:20:57.813 INFO:teuthology.orchestra.run.smithi101.stdout: Verifying : krb5-devel-1.18.2-14.el8.x86_64 81/134 2023-07-14T19:20:57.813 INFO:teuthology.orchestra.run.smithi101.stdout: Verifying : libaio-devel-0.3.112-1.el8.x86_64 82/134 2023-07-14T19:20:57.813 INFO:teuthology.orchestra.run.smithi101.stdout: Verifying : python3-oauthlib-2.1.0-1.el8.noarch 83/134 2023-07-14T19:20:57.813 INFO:teuthology.orchestra.run.smithi101.stdout: Verifying : python3-cryptography-3.2.1-5.el8.x86_64 84/134 2023-07-14T19:20:57.813 INFO:teuthology.orchestra.run.smithi101.stdout: Verifying : libverto-devel-0.3.0-5.el8.x86_64 85/134 2023-07-14T19:20:57.813 INFO:teuthology.orchestra.run.smithi101.stdout: Verifying : pcre2-devel-10.32-2.el8.x86_64 86/134 2023-07-14T19:20:57.813 INFO:teuthology.orchestra.run.smithi101.stdout: Verifying : python3-cffi-1.11.5-5.el8.x86_64 87/134 2023-07-14T19:20:57.813 INFO:teuthology.orchestra.run.smithi101.stdout: Verifying : openssl-devel-1:1.1.1k-7.el8_6.x86_64 88/134 2023-07-14T19:20:57.814 INFO:teuthology.orchestra.run.smithi101.stdout: Verifying : xfsprogs-devel-5.0.0-10.el8.x86_64 89/134 2023-07-14T19:20:57.814 INFO:teuthology.orchestra.run.smithi101.stdout: Verifying : fuse-2.9.7-15.el8.x86_64 90/134 2023-07-14T19:20:57.814 INFO:teuthology.orchestra.run.smithi101.stdout: Verifying : elfutils-libelf-devel-0.186-1.el8.x86_64 91/134 2023-07-14T19:20:57.814 INFO:teuthology.orchestra.run.smithi101.stdout: Verifying : libuuid-devel-2.32.1-35.el8.x86_64 92/134 2023-07-14T19:20:57.814 INFO:teuthology.orchestra.run.smithi101.stdout: Verifying : zlib-devel-1.2.11-18.el8_5.x86_64 93/134 2023-07-14T19:20:57.814 INFO:teuthology.orchestra.run.smithi101.stdout: Verifying : libcom_err-devel-1.45.6-4.el8.x86_64 94/134 2023-07-14T19:20:57.814 INFO:teuthology.orchestra.run.smithi101.stdout: Verifying : libgfortran-8.5.0-10.1.el8_6.x86_64 95/134 2023-07-14T19:20:57.814 INFO:teuthology.orchestra.run.smithi101.stdout: Verifying : libquadmath-8.5.0-10.1.el8_6.x86_64 96/134 2023-07-14T19:20:57.815 INFO:teuthology.orchestra.run.smithi101.stdout: Verifying : NetworkManager-initscripts-updown-1:1.36.0-9.el8 97/134 2023-07-14T19:20:57.815 INFO:teuthology.orchestra.run.smithi101.stdout: Verifying : python3-scipy-1.0.0-21.module+el8.5.0+10916+41bd 98/134 2023-07-14T19:20:57.815 INFO:teuthology.orchestra.run.smithi101.stdout: Verifying : python3-pyasn1-modules-0.3.7-6.el8.noarch 99/134 2023-07-14T19:20:57.815 INFO:teuthology.orchestra.run.smithi101.stdout: Verifying : python3-pytz-2017.2-9.el8.noarch 100/134 2023-07-14T19:20:57.815 INFO:teuthology.orchestra.run.smithi101.stdout: Verifying : lttng-ust-2.8.1-11.el8.x86_64 101/134 2023-07-14T19:20:57.815 INFO:teuthology.orchestra.run.smithi101.stdout: Verifying : socat-1.7.4.1-1.el8.x86_64 102/134 2023-07-14T19:20:57.815 INFO:teuthology.orchestra.run.smithi101.stdout: Verifying : python3-markupsafe-0.23-19.el8.x86_64 103/134 2023-07-14T19:20:57.816 INFO:teuthology.orchestra.run.smithi101.stdout: Verifying : bison-3.0.4-10.el8.x86_64 104/134 2023-07-14T19:20:57.816 INFO:teuthology.orchestra.run.smithi101.stdout: Verifying : python3-jinja2-2.10.1-3.el8.noarch 105/134 2023-07-14T19:20:57.816 INFO:teuthology.orchestra.run.smithi101.stdout: Verifying : python3-mako-1.0.6-13.el8.noarch 106/134 2023-07-14T19:20:57.816 INFO:teuthology.orchestra.run.smithi101.stdout: Verifying : oniguruma-6.8.2-2.el8.x86_64 107/134 2023-07-14T19:20:57.816 INFO:teuthology.orchestra.run.smithi101.stdout: Verifying : automake-1.16.1-7.el8.noarch 108/134 2023-07-14T19:20:57.816 INFO:teuthology.orchestra.run.smithi101.stdout: Verifying : python3-babel-2.5.1-7.el8.noarch 109/134 2023-07-14T19:20:57.816 INFO:teuthology.orchestra.run.smithi101.stdout: Verifying : python3-jsonpatch-1.21-2.el8.noarch 110/134 2023-07-14T19:20:57.816 INFO:teuthology.orchestra.run.smithi101.stdout: Verifying : flex-2.6.1-9.el8.x86_64 111/134 2023-07-14T19:20:57.817 INFO:teuthology.orchestra.run.smithi101.stdout: Verifying : python3-prettytable-0.7.2-14.el8.noarch 112/134 2023-07-14T19:20:57.817 INFO:teuthology.orchestra.run.smithi101.stdout: Verifying : python3-numpy-1:1.14.3-10.el8.x86_64 113/134 2023-07-14T19:20:57.817 INFO:teuthology.orchestra.run.smithi101.stdout: Verifying : python3-werkzeug-0.12.2-4.el8.noarch 114/134 2023-07-14T19:20:57.817 INFO:teuthology.orchestra.run.smithi101.stdout: Verifying : python3-numpy-f2py-1:1.14.3-10.el8.x86_64 115/134 2023-07-14T19:20:57.817 INFO:teuthology.orchestra.run.smithi101.stdout: Verifying : uuid-1.6.2-43.el8.x86_64 116/134 2023-07-14T19:20:57.817 INFO:teuthology.orchestra.run.smithi101.stdout: Verifying : python3-pyasn1-0.3.7-6.el8.noarch 117/134 2023-07-14T19:20:57.817 INFO:teuthology.orchestra.run.smithi101.stdout: Verifying : python3-jsonpointer-1.10-11.el8.noarch 118/134 2023-07-14T19:20:57.818 INFO:teuthology.orchestra.run.smithi101.stdout: Verifying : python3-pyOpenSSL-19.0.0-1.el8.noarch 119/134 2023-07-14T19:20:57.818 INFO:teuthology.orchestra.run.smithi101.stdout: Verifying : libtool-2.4.6-25.el8.x86_64 120/134 2023-07-14T19:20:57.818 INFO:teuthology.orchestra.run.smithi101.stdout: Verifying : openblas-threads-0.3.15-3.el8.x86_64 121/134 2023-07-14T19:20:57.818 INFO:teuthology.orchestra.run.smithi101.stdout: Verifying : libpq-13.5-1.el8.x86_64 122/134 2023-07-14T19:20:57.818 INFO:teuthology.orchestra.run.smithi101.stdout: Verifying : openblas-0.3.15-3.el8.x86_64 123/134 2023-07-14T19:20:57.818 INFO:teuthology.orchestra.run.smithi101.stdout: Verifying : jq-1.6-3.el8.x86_64 124/134 2023-07-14T19:20:57.818 INFO:teuthology.orchestra.run.smithi101.stdout: Verifying : librdkafka-0.11.4-3.el8.x86_64 125/134 2023-07-14T19:20:57.818 INFO:teuthology.orchestra.run.smithi101.stdout: Verifying : postgresql-server-10.21-2.module+el8.6.0+15342+5 126/134 2023-07-14T19:20:57.819 INFO:teuthology.orchestra.run.smithi101.stdout: Verifying : postgresql-contrib-10.21-2.module+el8.6.0+15342+ 127/134 2023-07-14T19:20:57.819 INFO:teuthology.orchestra.run.smithi101.stdout: Verifying : postgresql-10.21-2.module+el8.6.0+15342+53518fac 128/134 2023-07-14T19:20:57.819 INFO:teuthology.orchestra.run.smithi101.stdout: Verifying : librados2-2:18.0.0-4940.g81c9387d.el8.x86_64 129/134 2023-07-14T19:20:57.819 INFO:teuthology.orchestra.run.smithi101.stdout: Verifying : librados2-1:12.2.7-9.el8.x86_64 130/134 2023-07-14T19:20:57.819 INFO:teuthology.orchestra.run.smithi101.stdout: Verifying : librbd1-2:18.0.0-4940.g81c9387d.el8.x86_64 131/134 2023-07-14T19:20:57.819 INFO:teuthology.orchestra.run.smithi101.stdout: Verifying : librbd1-1:12.2.7-9.el8.x86_64 132/134 2023-07-14T19:20:57.819 INFO:teuthology.orchestra.run.smithi101.stdout: Verifying : iproute-5.15.0-4.el8_6.1.x86_64 133/134 2023-07-14T19:21:00.319 INFO:teuthology.orchestra.run.smithi101.stdout: Verifying : iproute-5.15.0-4.el8.x86_64 134/134 2023-07-14T19:21:00.320 INFO:teuthology.orchestra.run.smithi101.stdout:Last metadata expiration check: 0:04:28 ago on Fri 14 Jul 2023 07:16:32 PM UTC. 2023-07-14T19:21:00.556 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: rbd-mirror-2:18.0.0-4940.g81c9387d.el8.x86_64 114/134 2023-07-14T19:21:00.773 INFO:teuthology.orchestra.run.smithi139.stdout: Running scriptlet: rbd-mirror-2:18.0.0-4940.g81c9387d.el8.x86_64 114/134 2023-07-14T19:21:01.403 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : ceph-test-2:18.0.0-4940.g81c9387d.el8.x86_64 115/134 2023-07-14T19:21:01.679 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : ceph-test-2:18.0.0-4940.g81c9387d.el8.x86_64 115/134 2023-07-14T19:21:01.730 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : openssl-devel-1:1.1.1k-7.el8_6.x86_64 116/134 2023-07-14T19:21:01.910 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : ceph-fuse-2:18.0.0-4940.g81c9387d.el8.x86_64 117/134 2023-07-14T19:21:02.022 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : openssl-devel-1:1.1.1k-7.el8_6.x86_64 116/134 2023-07-14T19:21:02.175 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : xfsprogs-devel-5.0.0-10.el8.x86_64 118/134 2023-07-14T19:21:02.193 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : ceph-fuse-2:18.0.0-4940.g81c9387d.el8.x86_64 117/134 2023-07-14T19:21:02.356 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : libtool-2.4.6-25.el8.x86_64 119/134 2023-07-14T19:21:02.468 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : xfsprogs-devel-5.0.0-10.el8.x86_64 118/134 2023-07-14T19:21:02.565 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: libtool-2.4.6-25.el8.x86_64 119/134 2023-07-14T19:21:02.623 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : libtool-2.4.6-25.el8.x86_64 119/134 2023-07-14T19:21:02.786 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : postgresql-contrib-10.21-2.module+el8.6.0+15342+ 120/134 2023-07-14T19:21:02.822 INFO:teuthology.orchestra.run.smithi139.stdout: Running scriptlet: libtool-2.4.6-25.el8.x86_64 119/134 2023-07-14T19:21:02.832 INFO:teuthology.orchestra.run.smithi101.stdout:Installed products updated. 2023-07-14T19:21:03.010 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : libcephfs-devel-2:18.0.0-4940.g81c9387d.el8.x86_ 121/134 2023-07-14T19:21:03.041 INFO:teuthology.orchestra.run.smithi101.stdout: 2023-07-14T19:21:03.042 INFO:teuthology.orchestra.run.smithi101.stdout:Upgraded: 2023-07-14T19:21:03.042 INFO:teuthology.orchestra.run.smithi101.stdout: iproute-5.15.0-4.el8_6.1.x86_64 2023-07-14T19:21:03.042 INFO:teuthology.orchestra.run.smithi101.stdout: librados2-2:18.0.0-4940.g81c9387d.el8.x86_64 2023-07-14T19:21:03.042 INFO:teuthology.orchestra.run.smithi101.stdout: librbd1-2:18.0.0-4940.g81c9387d.el8.x86_64 2023-07-14T19:21:03.042 INFO:teuthology.orchestra.run.smithi101.stdout:Installed: 2023-07-14T19:21:03.042 INFO:teuthology.orchestra.run.smithi101.stdout: NetworkManager-initscripts-updown-1:1.36.0-9.el8_6.noarch 2023-07-14T19:21:03.043 INFO:teuthology.orchestra.run.smithi101.stdout: automake-1.16.1-7.el8.noarch 2023-07-14T19:21:03.043 INFO:teuthology.orchestra.run.smithi101.stdout: bison-3.0.4-10.el8.x86_64 2023-07-14T19:21:03.043 INFO:teuthology.orchestra.run.smithi101.stdout: ceph-2:18.0.0-4940.g81c9387d.el8.x86_64 2023-07-14T19:21:03.043 INFO:teuthology.orchestra.run.smithi101.stdout: ceph-base-2:18.0.0-4940.g81c9387d.el8.x86_64 2023-07-14T19:21:03.043 INFO:teuthology.orchestra.run.smithi101.stdout: ceph-common-2:18.0.0-4940.g81c9387d.el8.x86_64 2023-07-14T19:21:03.043 INFO:teuthology.orchestra.run.smithi101.stdout: ceph-fuse-2:18.0.0-4940.g81c9387d.el8.x86_64 2023-07-14T19:21:03.043 INFO:teuthology.orchestra.run.smithi101.stdout: ceph-grafana-dashboards-2:18.0.0-4940.g81c9387d.el8.noarch 2023-07-14T19:21:03.044 INFO:teuthology.orchestra.run.smithi101.stdout: ceph-immutable-object-cache-2:18.0.0-4940.g81c9387d.el8.x86_64 2023-07-14T19:21:03.044 INFO:teuthology.orchestra.run.smithi101.stdout: ceph-mds-2:18.0.0-4940.g81c9387d.el8.x86_64 2023-07-14T19:21:03.044 INFO:teuthology.orchestra.run.smithi101.stdout: ceph-mgr-2:18.0.0-4940.g81c9387d.el8.x86_64 2023-07-14T19:21:03.044 INFO:teuthology.orchestra.run.smithi101.stdout: ceph-mgr-cephadm-2:18.0.0-4940.g81c9387d.el8.noarch 2023-07-14T19:21:03.044 INFO:teuthology.orchestra.run.smithi101.stdout: ceph-mgr-dashboard-2:18.0.0-4940.g81c9387d.el8.noarch 2023-07-14T19:21:03.044 INFO:teuthology.orchestra.run.smithi101.stdout: ceph-mgr-diskprediction-local-2:18.0.0-4940.g81c9387d.el8.noarch 2023-07-14T19:21:03.045 INFO:teuthology.orchestra.run.smithi101.stdout: ceph-mgr-modules-core-2:18.0.0-4940.g81c9387d.el8.noarch 2023-07-14T19:21:03.045 INFO:teuthology.orchestra.run.smithi101.stdout: ceph-mgr-rook-2:18.0.0-4940.g81c9387d.el8.noarch 2023-07-14T19:21:03.045 INFO:teuthology.orchestra.run.smithi101.stdout: ceph-mon-2:18.0.0-4940.g81c9387d.el8.x86_64 2023-07-14T19:21:03.045 INFO:teuthology.orchestra.run.smithi101.stdout: ceph-osd-2:18.0.0-4940.g81c9387d.el8.x86_64 2023-07-14T19:21:03.045 INFO:teuthology.orchestra.run.smithi101.stdout: ceph-prometheus-alerts-2:18.0.0-4940.g81c9387d.el8.noarch 2023-07-14T19:21:03.046 INFO:teuthology.orchestra.run.smithi101.stdout: ceph-radosgw-2:18.0.0-4940.g81c9387d.el8.x86_64 2023-07-14T19:21:03.046 INFO:teuthology.orchestra.run.smithi101.stdout: ceph-selinux-2:18.0.0-4940.g81c9387d.el8.x86_64 2023-07-14T19:21:03.046 INFO:teuthology.orchestra.run.smithi101.stdout: ceph-test-2:18.0.0-4940.g81c9387d.el8.x86_64 2023-07-14T19:21:03.046 INFO:teuthology.orchestra.run.smithi101.stdout: ceph-volume-2:18.0.0-4940.g81c9387d.el8.noarch 2023-07-14T19:21:03.046 INFO:teuthology.orchestra.run.smithi101.stdout: cephadm-2:18.0.0-4940.g81c9387d.el8.noarch 2023-07-14T19:21:03.046 INFO:teuthology.orchestra.run.smithi101.stdout: cephfs-mirror-2:18.0.0-4940.g81c9387d.el8.x86_64 2023-07-14T19:21:03.047 INFO:teuthology.orchestra.run.smithi101.stdout: cephfs-top-2:18.0.0-4940.g81c9387d.el8.noarch 2023-07-14T19:21:03.047 INFO:teuthology.orchestra.run.smithi101.stdout: elfutils-libelf-devel-0.186-1.el8.x86_64 2023-07-14T19:21:03.047 INFO:teuthology.orchestra.run.smithi101.stdout: flex-2.6.1-9.el8.x86_64 2023-07-14T19:21:03.047 INFO:teuthology.orchestra.run.smithi101.stdout: fuse-2.9.7-15.el8.x86_64 2023-07-14T19:21:03.047 INFO:teuthology.orchestra.run.smithi101.stdout: gperftools-libs-1:2.7-9.el8.x86_64 2023-07-14T19:21:03.048 INFO:teuthology.orchestra.run.smithi101.stdout: jq-1.6-3.el8.x86_64 2023-07-14T19:21:03.048 INFO:teuthology.orchestra.run.smithi101.stdout: keyutils-libs-devel-1.5.10-9.el8.x86_64 2023-07-14T19:21:03.049 INFO:teuthology.orchestra.run.smithi101.stdout: krb5-devel-1.18.2-14.el8.x86_64 2023-07-14T19:21:03.049 INFO:teuthology.orchestra.run.smithi101.stdout: libaio-devel-0.3.112-1.el8.x86_64 2023-07-14T19:21:03.049 INFO:teuthology.orchestra.run.smithi101.stdout: libcephfs-devel-2:18.0.0-4940.g81c9387d.el8.x86_64 2023-07-14T19:21:03.049 INFO:teuthology.orchestra.run.smithi101.stdout: libcephfs2-2:18.0.0-4940.g81c9387d.el8.x86_64 2023-07-14T19:21:03.049 INFO:teuthology.orchestra.run.smithi101.stdout: libcephsqlite-2:18.0.0-4940.g81c9387d.el8.x86_64 2023-07-14T19:21:03.049 INFO:teuthology.orchestra.run.smithi101.stdout: libcom_err-devel-1.45.6-4.el8.x86_64 2023-07-14T19:21:03.050 INFO:teuthology.orchestra.run.smithi101.stdout: libgfortran-8.5.0-10.1.el8_6.x86_64 2023-07-14T19:21:03.050 INFO:teuthology.orchestra.run.smithi101.stdout: liboath-2.6.2-3.el8.x86_64 2023-07-14T19:21:03.050 INFO:teuthology.orchestra.run.smithi101.stdout: libpq-13.5-1.el8.x86_64 2023-07-14T19:21:03.050 INFO:teuthology.orchestra.run.smithi101.stdout: libquadmath-8.5.0-10.1.el8_6.x86_64 2023-07-14T19:21:03.050 INFO:teuthology.orchestra.run.smithi101.stdout: librabbitmq-0.9.0-3.el8.x86_64 2023-07-14T19:21:03.050 INFO:teuthology.orchestra.run.smithi101.stdout: librados-devel-2:18.0.0-4940.g81c9387d.el8.x86_64 2023-07-14T19:21:03.051 INFO:teuthology.orchestra.run.smithi101.stdout: libradosstriper1-2:18.0.0-4940.g81c9387d.el8.x86_64 2023-07-14T19:21:03.051 INFO:teuthology.orchestra.run.smithi101.stdout: librdkafka-0.11.4-3.el8.x86_64 2023-07-14T19:21:03.051 INFO:teuthology.orchestra.run.smithi101.stdout: librgw2-2:18.0.0-4940.g81c9387d.el8.x86_64 2023-07-14T19:21:03.051 INFO:teuthology.orchestra.run.smithi101.stdout: libselinux-devel-2.9-5.el8.x86_64 2023-07-14T19:21:03.052 INFO:teuthology.orchestra.run.smithi101.stdout: libsepol-devel-2.9-3.el8.x86_64 2023-07-14T19:21:03.052 INFO:teuthology.orchestra.run.smithi101.stdout: libtool-2.4.6-25.el8.x86_64 2023-07-14T19:21:03.052 INFO:teuthology.orchestra.run.smithi101.stdout: libunwind-1.3.1-3.el8.x86_64 2023-07-14T19:21:03.052 INFO:teuthology.orchestra.run.smithi101.stdout: libuuid-devel-2.32.1-35.el8.x86_64 2023-07-14T19:21:03.052 INFO:teuthology.orchestra.run.smithi101.stdout: libverto-devel-0.3.0-5.el8.x86_64 2023-07-14T19:21:03.052 INFO:teuthology.orchestra.run.smithi101.stdout: lttng-ust-2.8.1-11.el8.x86_64 2023-07-14T19:21:03.052 INFO:teuthology.orchestra.run.smithi101.stdout: oniguruma-6.8.2-2.el8.x86_64 2023-07-14T19:21:03.053 INFO:teuthology.orchestra.run.smithi101.stdout: openblas-0.3.15-3.el8.x86_64 2023-07-14T19:21:03.053 INFO:teuthology.orchestra.run.smithi101.stdout: openblas-threads-0.3.15-3.el8.x86_64 2023-07-14T19:21:03.053 INFO:teuthology.orchestra.run.smithi101.stdout: openssl-devel-1:1.1.1k-7.el8_6.x86_64 2023-07-14T19:21:03.053 INFO:teuthology.orchestra.run.smithi101.stdout: pcre2-devel-10.32-2.el8.x86_64 2023-07-14T19:21:03.053 INFO:teuthology.orchestra.run.smithi101.stdout: pcre2-utf16-10.32-2.el8.x86_64 2023-07-14T19:21:03.053 INFO:teuthology.orchestra.run.smithi101.stdout: pcre2-utf32-10.32-2.el8.x86_64 2023-07-14T19:21:03.053 INFO:teuthology.orchestra.run.smithi101.stdout: postgresql-10.21-2.module+el8.6.0+15342+53518fac.x86_64 2023-07-14T19:21:03.053 INFO:teuthology.orchestra.run.smithi101.stdout: postgresql-contrib-10.21-2.module+el8.6.0+15342+53518fac.x86_64 2023-07-14T19:21:03.054 INFO:teuthology.orchestra.run.smithi101.stdout: postgresql-server-10.21-2.module+el8.6.0+15342+53518fac.x86_64 2023-07-14T19:21:03.054 INFO:teuthology.orchestra.run.smithi101.stdout: python3-asyncssh-2.7.0-2.el8.noarch 2023-07-14T19:21:03.054 INFO:teuthology.orchestra.run.smithi101.stdout: python3-babel-2.5.1-7.el8.noarch 2023-07-14T19:21:03.054 INFO:teuthology.orchestra.run.smithi101.stdout: python3-bcrypt-3.1.6-2.el8.1.x86_64 2023-07-14T19:21:03.054 INFO:teuthology.orchestra.run.smithi101.stdout: python3-beautifulsoup4-4.6.3-2.el8.1.noarch 2023-07-14T19:21:03.054 INFO:teuthology.orchestra.run.smithi101.stdout: python3-cachetools-3.1.1-4.el8.noarch 2023-07-14T19:21:03.054 INFO:teuthology.orchestra.run.smithi101.stdout: python3-ceph-argparse-2:18.0.0-4940.g81c9387d.el8.x86_64 2023-07-14T19:21:03.054 INFO:teuthology.orchestra.run.smithi101.stdout: python3-ceph-common-2:18.0.0-4940.g81c9387d.el8.x86_64 2023-07-14T19:21:03.055 INFO:teuthology.orchestra.run.smithi101.stdout: python3-cephfs-2:18.0.0-4940.g81c9387d.el8.x86_64 2023-07-14T19:21:03.055 INFO:teuthology.orchestra.run.smithi101.stdout: python3-certifi-2018.10.15-7.el8.noarch 2023-07-14T19:21:03.055 INFO:teuthology.orchestra.run.smithi101.stdout: python3-cffi-1.11.5-5.el8.x86_64 2023-07-14T19:21:03.055 INFO:teuthology.orchestra.run.smithi101.stdout: python3-cheroot-8.5.2-1.el8.noarch 2023-07-14T19:21:03.055 INFO:teuthology.orchestra.run.smithi101.stdout: python3-cherrypy-18.4.0-1.el8.noarch 2023-07-14T19:21:03.055 INFO:teuthology.orchestra.run.smithi101.stdout: python3-cryptography-3.2.1-5.el8.x86_64 2023-07-14T19:21:03.055 INFO:teuthology.orchestra.run.smithi101.stdout: python3-google-auth-1:1.1.1-10.el8.noarch 2023-07-14T19:21:03.056 INFO:teuthology.orchestra.run.smithi101.stdout: python3-jaraco-6.2-6.el8.noarch 2023-07-14T19:21:03.056 INFO:teuthology.orchestra.run.smithi101.stdout: python3-jaraco-functools-2.0-4.el8.noarch 2023-07-14T19:21:03.056 INFO:teuthology.orchestra.run.smithi101.stdout: python3-jinja2-2.10.1-3.el8.noarch 2023-07-14T19:21:03.056 INFO:teuthology.orchestra.run.smithi101.stdout: python3-jsonpatch-1.21-2.el8.noarch 2023-07-14T19:21:03.056 INFO:teuthology.orchestra.run.smithi101.stdout: python3-jsonpointer-1.10-11.el8.noarch 2023-07-14T19:21:03.056 INFO:teuthology.orchestra.run.smithi101.stdout: python3-jwt-1.6.1-2.el8.noarch 2023-07-14T19:21:03.056 INFO:teuthology.orchestra.run.smithi101.stdout: python3-kubernetes-1:11.0.0-6.el8.noarch 2023-07-14T19:21:03.056 INFO:teuthology.orchestra.run.smithi101.stdout: python3-logutils-0.3.5-11.el8.noarch 2023-07-14T19:21:03.057 INFO:teuthology.orchestra.run.smithi101.stdout: python3-mako-1.0.6-13.el8.noarch 2023-07-14T19:21:03.057 INFO:teuthology.orchestra.run.smithi101.stdout: python3-markupsafe-0.23-19.el8.x86_64 2023-07-14T19:21:03.057 INFO:teuthology.orchestra.run.smithi101.stdout: python3-more-itertools-7.2.0-3.el8.noarch 2023-07-14T19:21:03.057 INFO:teuthology.orchestra.run.smithi101.stdout: python3-natsort-7.1.1-2.el8.noarch 2023-07-14T19:21:03.057 INFO:teuthology.orchestra.run.smithi101.stdout: python3-numpy-1:1.14.3-10.el8.x86_64 2023-07-14T19:21:03.057 INFO:teuthology.orchestra.run.smithi101.stdout: python3-numpy-f2py-1:1.14.3-10.el8.x86_64 2023-07-14T19:21:03.057 INFO:teuthology.orchestra.run.smithi101.stdout: python3-oauthlib-2.1.0-1.el8.noarch 2023-07-14T19:21:03.058 INFO:teuthology.orchestra.run.smithi101.stdout: python3-pecan-1.3.2-9.el8.noarch 2023-07-14T19:21:03.058 INFO:teuthology.orchestra.run.smithi101.stdout: python3-portend-2.6-1.el8.noarch 2023-07-14T19:21:03.058 INFO:teuthology.orchestra.run.smithi101.stdout: python3-prettytable-0.7.2-14.el8.noarch 2023-07-14T19:21:03.058 INFO:teuthology.orchestra.run.smithi101.stdout: python3-pyOpenSSL-19.0.0-1.el8.noarch 2023-07-14T19:21:03.058 INFO:teuthology.orchestra.run.smithi101.stdout: python3-pyasn1-0.3.7-6.el8.noarch 2023-07-14T19:21:03.058 INFO:teuthology.orchestra.run.smithi101.stdout: python3-pyasn1-modules-0.3.7-6.el8.noarch 2023-07-14T19:21:03.058 INFO:teuthology.orchestra.run.smithi101.stdout: python3-pycparser-2.14-14.el8.noarch 2023-07-14T19:21:03.058 INFO:teuthology.orchestra.run.smithi101.stdout: python3-pytz-2017.2-9.el8.noarch 2023-07-14T19:21:03.059 INFO:teuthology.orchestra.run.smithi101.stdout: python3-rados-2:18.0.0-4940.g81c9387d.el8.x86_64 2023-07-14T19:21:03.059 INFO:teuthology.orchestra.run.smithi101.stdout: python3-rbd-2:18.0.0-4940.g81c9387d.el8.x86_64 2023-07-14T19:21:03.059 INFO:teuthology.orchestra.run.smithi101.stdout: python3-repoze-lru-0.7-6.el8.noarch 2023-07-14T19:21:03.059 INFO:teuthology.orchestra.run.smithi101.stdout: python3-requests-oauthlib-1.0.0-1.el8.noarch 2023-07-14T19:21:03.059 INFO:teuthology.orchestra.run.smithi101.stdout: python3-rgw-2:18.0.0-4940.g81c9387d.el8.x86_64 2023-07-14T19:21:03.059 INFO:teuthology.orchestra.run.smithi101.stdout: python3-routes-2.4.1-12.el8.noarch 2023-07-14T19:21:03.059 INFO:teuthology.orchestra.run.smithi101.stdout: python3-rsa-4.8-1.el8.noarch 2023-07-14T19:21:03.060 INFO:teuthology.orchestra.run.smithi101.stdout: python3-scipy-1.0.0-21.module+el8.5.0+10916+41bd434d.x86_64 2023-07-14T19:21:03.060 INFO:teuthology.orchestra.run.smithi101.stdout: python3-simplegeneric-0.8.1-17.el8.noarch 2023-07-14T19:21:03.060 INFO:teuthology.orchestra.run.smithi101.stdout: python3-singledispatch-3.4.0.3-18.el8.noarch 2023-07-14T19:21:03.060 INFO:teuthology.orchestra.run.smithi101.stdout: python3-tempora-1.14.1-5.el8.noarch 2023-07-14T19:21:03.060 INFO:teuthology.orchestra.run.smithi101.stdout: python3-trustme-0.6.0-4.el8.noarch 2023-07-14T19:21:03.060 INFO:teuthology.orchestra.run.smithi101.stdout: python3-waitress-1.4.3-1.el8.noarch 2023-07-14T19:21:03.060 INFO:teuthology.orchestra.run.smithi101.stdout: python3-webob-1.8.5-1.el8.1.noarch 2023-07-14T19:21:03.060 INFO:teuthology.orchestra.run.smithi101.stdout: python3-websocket-client-0.56.0-5.el8.noarch 2023-07-14T19:21:03.061 INFO:teuthology.orchestra.run.smithi101.stdout: python3-webtest-2.0.33-1.el8.noarch 2023-07-14T19:21:03.061 INFO:teuthology.orchestra.run.smithi101.stdout: python3-werkzeug-0.12.2-4.el8.noarch 2023-07-14T19:21:03.061 INFO:teuthology.orchestra.run.smithi101.stdout: python3-zc-lockfile-2.0-2.el8.noarch 2023-07-14T19:21:03.061 INFO:teuthology.orchestra.run.smithi101.stdout: rbd-fuse-2:18.0.0-4940.g81c9387d.el8.x86_64 2023-07-14T19:21:03.061 INFO:teuthology.orchestra.run.smithi101.stdout: rbd-mirror-2:18.0.0-4940.g81c9387d.el8.x86_64 2023-07-14T19:21:03.061 INFO:teuthology.orchestra.run.smithi101.stdout: rbd-nbd-2:18.0.0-4940.g81c9387d.el8.x86_64 2023-07-14T19:21:03.061 INFO:teuthology.orchestra.run.smithi101.stdout: socat-1.7.4.1-1.el8.x86_64 2023-07-14T19:21:03.062 INFO:teuthology.orchestra.run.smithi101.stdout: thrift-0.13.0-2.el8.x86_64 2023-07-14T19:21:03.062 INFO:teuthology.orchestra.run.smithi101.stdout: uuid-1.6.2-43.el8.x86_64 2023-07-14T19:21:03.062 INFO:teuthology.orchestra.run.smithi101.stdout: xfsprogs-devel-5.0.0-10.el8.x86_64 2023-07-14T19:21:03.062 INFO:teuthology.orchestra.run.smithi101.stdout: xmlstarlet-1.6.1-20.el8.x86_64 2023-07-14T19:21:03.063 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : postgresql-contrib-10.21-2.module+el8.6.0+15342+ 120/134 2023-07-14T19:21:03.063 INFO:teuthology.orchestra.run.smithi101.stdout: zlib-devel-1.2.11-18.el8_5.x86_64 2023-07-14T19:21:03.063 INFO:teuthology.orchestra.run.smithi101.stdout: 2023-07-14T19:21:03.063 INFO:teuthology.orchestra.run.smithi101.stdout:Complete! 2023-07-14T19:21:03.183 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : elfutils-libelf-devel-0.186-1.el8.x86_64 122/134 2023-07-14T19:21:03.239 DEBUG:teuthology.parallel:result is None 2023-07-14T19:21:03.244 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : libcephfs-devel-2:18.0.0-4940.g81c9387d.el8.x86_ 121/134 2023-07-14T19:21:03.432 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : elfutils-libelf-devel-0.186-1.el8.x86_64 122/134 2023-07-14T19:21:05.307 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: postgresql-server-10.21-2.module+el8.6.0+15342+5 123/134 2023-07-14T19:21:05.565 INFO:teuthology.orchestra.run.smithi139.stdout: Running scriptlet: postgresql-server-10.21-2.module+el8.6.0+15342+5 123/134 2023-07-14T19:21:05.565 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : postgresql-server-10.21-2.module+el8.6.0+15342+5 123/134 2023-07-14T19:21:05.582 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: postgresql-server-10.21-2.module+el8.6.0+15342+5 123/134 2023-07-14T19:21:05.732 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : cephfs-top-2:18.0.0-4940.g81c9387d.el8.noarch 124/134 2023-07-14T19:21:05.848 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : postgresql-server-10.21-2.module+el8.6.0+15342+5 123/134 2023-07-14T19:21:05.870 INFO:teuthology.orchestra.run.smithi139.stdout: Running scriptlet: postgresql-server-10.21-2.module+el8.6.0+15342+5 123/134 2023-07-14T19:21:06.014 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : rbd-fuse-2:18.0.0-4940.g81c9387d.el8.x86_64 125/134 2023-07-14T19:21:06.016 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : cephfs-top-2:18.0.0-4940.g81c9387d.el8.noarch 124/134 2023-07-14T19:21:06.305 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : rbd-fuse-2:18.0.0-4940.g81c9387d.el8.x86_64 125/134 2023-07-14T19:21:06.332 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : rbd-nbd-2:18.0.0-4940.g81c9387d.el8.x86_64 126/134 2023-07-14T19:21:06.617 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : rbd-nbd-2:18.0.0-4940.g81c9387d.el8.x86_64 126/134 2023-07-14T19:21:06.668 INFO:teuthology.orchestra.run.smithi098.stdout: Upgrading : iproute-5.15.0-4.el8_6.1.x86_64 127/134 2023-07-14T19:21:06.874 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : flex-2.6.1-9.el8.x86_64 128/134 2023-07-14T19:21:06.956 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: flex-2.6.1-9.el8.x86_64 128/134 2023-07-14T19:21:06.978 INFO:teuthology.orchestra.run.smithi139.stdout: Upgrading : iproute-5.15.0-4.el8_6.1.x86_64 127/134 2023-07-14T19:21:07.130 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : flex-2.6.1-9.el8.x86_64 128/134 2023-07-14T19:21:07.156 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : bison-3.0.4-10.el8.x86_64 129/134 2023-07-14T19:21:07.168 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: bison-3.0.4-10.el8.x86_64 129/134 2023-07-14T19:21:07.214 INFO:teuthology.orchestra.run.smithi139.stdout: Running scriptlet: flex-2.6.1-9.el8.x86_64 128/134 2023-07-14T19:21:07.323 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : NetworkManager-initscripts-updown-1:1.36.0-9.el8 130/134 2023-07-14T19:21:07.330 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: NetworkManager-initscripts-updown-1:1.36.0-9.el8 130/134 2023-07-14T19:21:07.439 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : bison-3.0.4-10.el8.x86_64 129/134 2023-07-14T19:21:07.451 INFO:teuthology.orchestra.run.smithi139.stdout: Running scriptlet: bison-3.0.4-10.el8.x86_64 129/134 2023-07-14T19:21:07.491 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : libaio-devel-0.3.112-1.el8.x86_64 131/134 2023-07-14T19:21:07.491 INFO:teuthology.orchestra.run.smithi098.stdout: Cleanup : librbd1-1:12.2.7-9.el8.x86_64 132/134 2023-07-14T19:21:07.622 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : NetworkManager-initscripts-updown-1:1.36.0-9.el8 130/134 2023-07-14T19:21:07.630 INFO:teuthology.orchestra.run.smithi139.stdout: Running scriptlet: NetworkManager-initscripts-updown-1:1.36.0-9.el8 130/134 2023-07-14T19:21:07.675 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: librbd1-1:12.2.7-9.el8.x86_64 132/134 2023-07-14T19:21:07.676 INFO:teuthology.orchestra.run.smithi098.stdout: Cleanup : librados2-1:12.2.7-9.el8.x86_64 133/134 2023-07-14T19:21:07.766 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : libaio-devel-0.3.112-1.el8.x86_64 131/134 2023-07-14T19:21:07.766 INFO:teuthology.orchestra.run.smithi139.stdout: Cleanup : librbd1-1:12.2.7-9.el8.x86_64 132/134 2023-07-14T19:21:07.912 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: librados2-1:12.2.7-9.el8.x86_64 133/134 2023-07-14T19:21:07.984 INFO:teuthology.orchestra.run.smithi139.stdout: Running scriptlet: librbd1-1:12.2.7-9.el8.x86_64 132/134 2023-07-14T19:21:07.984 INFO:teuthology.orchestra.run.smithi139.stdout: Cleanup : librados2-1:12.2.7-9.el8.x86_64 133/134 2023-07-14T19:21:08.138 INFO:teuthology.orchestra.run.smithi098.stdout: Cleanup : iproute-5.15.0-4.el8.x86_64 134/134 2023-07-14T19:21:08.217 INFO:teuthology.orchestra.run.smithi139.stdout: Running scriptlet: librados2-1:12.2.7-9.el8.x86_64 133/134 2023-07-14T19:21:08.438 INFO:teuthology.orchestra.run.smithi139.stdout: Cleanup : iproute-5.15.0-4.el8.x86_64 134/134 2023-07-14T19:21:08.635 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: iproute-5.15.0-4.el8.x86_64 134/134 2023-07-14T19:21:08.636 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : ceph-2:18.0.0-4940.g81c9387d.el8.x86_64 1/134 2023-07-14T19:21:08.636 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : ceph-base-2:18.0.0-4940.g81c9387d.el8.x86_64 2/134 2023-07-14T19:21:08.636 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : ceph-common-2:18.0.0-4940.g81c9387d.el8.x86_64 3/134 2023-07-14T19:21:08.636 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : ceph-fuse-2:18.0.0-4940.g81c9387d.el8.x86_64 4/134 2023-07-14T19:21:08.637 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : ceph-immutable-object-cache-2:18.0.0-4940.g81c93 5/134 2023-07-14T19:21:08.637 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : ceph-mds-2:18.0.0-4940.g81c9387d.el8.x86_64 6/134 2023-07-14T19:21:08.637 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : ceph-mgr-2:18.0.0-4940.g81c9387d.el8.x86_64 7/134 2023-07-14T19:21:08.637 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : ceph-mon-2:18.0.0-4940.g81c9387d.el8.x86_64 8/134 2023-07-14T19:21:08.637 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : ceph-osd-2:18.0.0-4940.g81c9387d.el8.x86_64 9/134 2023-07-14T19:21:08.637 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : ceph-radosgw-2:18.0.0-4940.g81c9387d.el8.x86_64 10/134 2023-07-14T19:21:08.637 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : ceph-selinux-2:18.0.0-4940.g81c9387d.el8.x86_64 11/134 2023-07-14T19:21:08.638 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : ceph-test-2:18.0.0-4940.g81c9387d.el8.x86_64 12/134 2023-07-14T19:21:08.638 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : cephfs-mirror-2:18.0.0-4940.g81c9387d.el8.x86_64 13/134 2023-07-14T19:21:08.638 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : libcephfs-devel-2:18.0.0-4940.g81c9387d.el8.x86_ 14/134 2023-07-14T19:21:08.638 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : libcephfs2-2:18.0.0-4940.g81c9387d.el8.x86_64 15/134 2023-07-14T19:21:08.638 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : libcephsqlite-2:18.0.0-4940.g81c9387d.el8.x86_64 16/134 2023-07-14T19:21:08.638 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : librados-devel-2:18.0.0-4940.g81c9387d.el8.x86_6 17/134 2023-07-14T19:21:08.638 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : libradosstriper1-2:18.0.0-4940.g81c9387d.el8.x86 18/134 2023-07-14T19:21:08.638 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : librgw2-2:18.0.0-4940.g81c9387d.el8.x86_64 19/134 2023-07-14T19:21:08.639 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-ceph-argparse-2:18.0.0-4940.g81c9387d.el 20/134 2023-07-14T19:21:08.640 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-ceph-common-2:18.0.0-4940.g81c9387d.el8. 21/134 2023-07-14T19:21:08.640 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-cephfs-2:18.0.0-4940.g81c9387d.el8.x86_6 22/134 2023-07-14T19:21:08.640 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-rados-2:18.0.0-4940.g81c9387d.el8.x86_64 23/134 2023-07-14T19:21:08.640 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-rbd-2:18.0.0-4940.g81c9387d.el8.x86_64 24/134 2023-07-14T19:21:08.640 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-rgw-2:18.0.0-4940.g81c9387d.el8.x86_64 25/134 2023-07-14T19:21:08.640 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : rbd-fuse-2:18.0.0-4940.g81c9387d.el8.x86_64 26/134 2023-07-14T19:21:08.641 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : rbd-mirror-2:18.0.0-4940.g81c9387d.el8.x86_64 27/134 2023-07-14T19:21:08.641 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : rbd-nbd-2:18.0.0-4940.g81c9387d.el8.x86_64 28/134 2023-07-14T19:21:08.641 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : ceph-grafana-dashboards-2:18.0.0-4940.g81c9387d. 29/134 2023-07-14T19:21:08.650 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : ceph-mgr-cephadm-2:18.0.0-4940.g81c9387d.el8.noa 30/134 2023-07-14T19:21:08.651 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : ceph-mgr-dashboard-2:18.0.0-4940.g81c9387d.el8.n 31/134 2023-07-14T19:21:08.651 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : ceph-mgr-diskprediction-local-2:18.0.0-4940.g81c 32/134 2023-07-14T19:21:08.651 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : ceph-mgr-modules-core-2:18.0.0-4940.g81c9387d.el 33/134 2023-07-14T19:21:08.651 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : ceph-mgr-rook-2:18.0.0-4940.g81c9387d.el8.noarch 34/134 2023-07-14T19:21:08.651 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : ceph-prometheus-alerts-2:18.0.0-4940.g81c9387d.e 35/134 2023-07-14T19:21:08.652 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : ceph-volume-2:18.0.0-4940.g81c9387d.el8.noarch 36/134 2023-07-14T19:21:08.652 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : cephadm-2:18.0.0-4940.g81c9387d.el8.noarch 37/134 2023-07-14T19:21:08.653 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : cephfs-top-2:18.0.0-4940.g81c9387d.el8.noarch 38/134 2023-07-14T19:21:08.653 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : gperftools-libs-1:2.7-9.el8.x86_64 39/134 2023-07-14T19:21:08.653 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : liboath-2.6.2-3.el8.x86_64 40/134 2023-07-14T19:21:08.653 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : libunwind-1.3.1-3.el8.x86_64 41/134 2023-07-14T19:21:08.654 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-asyncssh-2.7.0-2.el8.noarch 42/134 2023-07-14T19:21:08.654 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-bcrypt-3.1.6-2.el8.1.x86_64 43/134 2023-07-14T19:21:08.654 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 44/134 2023-07-14T19:21:08.654 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-cachetools-3.1.1-4.el8.noarch 45/134 2023-07-14T19:21:08.655 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-certifi-2018.10.15-7.el8.noarch 46/134 2023-07-14T19:21:08.655 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-cheroot-8.5.2-1.el8.noarch 47/134 2023-07-14T19:21:08.655 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-cherrypy-18.4.0-1.el8.noarch 48/134 2023-07-14T19:21:08.655 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-google-auth-1:1.1.1-10.el8.noarch 49/134 2023-07-14T19:21:08.656 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-jaraco-6.2-6.el8.noarch 50/134 2023-07-14T19:21:08.656 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-jaraco-functools-2.0-4.el8.noarch 51/134 2023-07-14T19:21:08.656 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-kubernetes-1:11.0.0-6.el8.noarch 52/134 2023-07-14T19:21:08.656 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-logutils-0.3.5-11.el8.noarch 53/134 2023-07-14T19:21:08.656 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-more-itertools-7.2.0-3.el8.noarch 54/134 2023-07-14T19:21:08.657 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-natsort-7.1.1-2.el8.noarch 55/134 2023-07-14T19:21:08.657 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-pecan-1.3.2-9.el8.noarch 56/134 2023-07-14T19:21:08.657 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-portend-2.6-1.el8.noarch 57/134 2023-07-14T19:21:08.657 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-repoze-lru-0.7-6.el8.noarch 58/134 2023-07-14T19:21:08.658 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-routes-2.4.1-12.el8.noarch 59/134 2023-07-14T19:21:08.658 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-rsa-4.8-1.el8.noarch 60/134 2023-07-14T19:21:08.658 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-simplegeneric-0.8.1-17.el8.noarch 61/134 2023-07-14T19:21:08.658 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-singledispatch-3.4.0.3-18.el8.noarch 62/134 2023-07-14T19:21:08.659 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-tempora-1.14.1-5.el8.noarch 63/134 2023-07-14T19:21:08.659 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-trustme-0.6.0-4.el8.noarch 64/134 2023-07-14T19:21:08.659 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-waitress-1.4.3-1.el8.noarch 65/134 2023-07-14T19:21:08.660 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-webob-1.8.5-1.el8.1.noarch 66/134 2023-07-14T19:21:08.660 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-websocket-client-0.56.0-5.el8.noarch 67/134 2023-07-14T19:21:08.660 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-webtest-2.0.33-1.el8.noarch 68/134 2023-07-14T19:21:08.660 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-zc-lockfile-2.0-2.el8.noarch 69/134 2023-07-14T19:21:08.661 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : thrift-0.13.0-2.el8.x86_64 70/134 2023-07-14T19:21:08.661 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : xmlstarlet-1.6.1-20.el8.x86_64 71/134 2023-07-14T19:21:08.661 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : librabbitmq-0.9.0-3.el8.x86_64 72/134 2023-07-14T19:21:08.661 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : libselinux-devel-2.9-5.el8.x86_64 73/134 2023-07-14T19:21:08.661 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-jwt-1.6.1-2.el8.noarch 74/134 2023-07-14T19:21:08.661 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : pcre2-utf32-10.32-2.el8.x86_64 75/134 2023-07-14T19:21:08.661 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : keyutils-libs-devel-1.5.10-9.el8.x86_64 76/134 2023-07-14T19:21:08.662 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : libsepol-devel-2.9-3.el8.x86_64 77/134 2023-07-14T19:21:08.662 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-requests-oauthlib-1.0.0-1.el8.noarch 78/134 2023-07-14T19:21:08.662 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-pycparser-2.14-14.el8.noarch 79/134 2023-07-14T19:21:08.662 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : pcre2-utf16-10.32-2.el8.x86_64 80/134 2023-07-14T19:21:08.662 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : krb5-devel-1.18.2-14.el8.x86_64 81/134 2023-07-14T19:21:08.662 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : libaio-devel-0.3.112-1.el8.x86_64 82/134 2023-07-14T19:21:08.662 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-oauthlib-2.1.0-1.el8.noarch 83/134 2023-07-14T19:21:08.662 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-cryptography-3.2.1-5.el8.x86_64 84/134 2023-07-14T19:21:08.663 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : libverto-devel-0.3.0-5.el8.x86_64 85/134 2023-07-14T19:21:08.663 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : pcre2-devel-10.32-2.el8.x86_64 86/134 2023-07-14T19:21:08.663 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-cffi-1.11.5-5.el8.x86_64 87/134 2023-07-14T19:21:08.663 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : openssl-devel-1:1.1.1k-7.el8_6.x86_64 88/134 2023-07-14T19:21:08.663 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : xfsprogs-devel-5.0.0-10.el8.x86_64 89/134 2023-07-14T19:21:08.663 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : fuse-2.9.7-15.el8.x86_64 90/134 2023-07-14T19:21:08.663 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : elfutils-libelf-devel-0.186-1.el8.x86_64 91/134 2023-07-14T19:21:08.664 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : libuuid-devel-2.32.1-35.el8.x86_64 92/134 2023-07-14T19:21:08.664 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : zlib-devel-1.2.11-18.el8_5.x86_64 93/134 2023-07-14T19:21:08.664 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : libcom_err-devel-1.45.6-4.el8.x86_64 94/134 2023-07-14T19:21:08.664 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : libgfortran-8.5.0-10.1.el8_6.x86_64 95/134 2023-07-14T19:21:08.664 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : libquadmath-8.5.0-10.1.el8_6.x86_64 96/134 2023-07-14T19:21:08.664 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : NetworkManager-initscripts-updown-1:1.36.0-9.el8 97/134 2023-07-14T19:21:08.664 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-scipy-1.0.0-21.module+el8.5.0+10916+41bd 98/134 2023-07-14T19:21:08.665 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-pyasn1-modules-0.3.7-6.el8.noarch 99/134 2023-07-14T19:21:08.665 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-pytz-2017.2-9.el8.noarch 100/134 2023-07-14T19:21:08.665 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : lttng-ust-2.8.1-11.el8.x86_64 101/134 2023-07-14T19:21:08.665 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : socat-1.7.4.1-1.el8.x86_64 102/134 2023-07-14T19:21:08.665 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-markupsafe-0.23-19.el8.x86_64 103/134 2023-07-14T19:21:08.665 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : bison-3.0.4-10.el8.x86_64 104/134 2023-07-14T19:21:08.665 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-jinja2-2.10.1-3.el8.noarch 105/134 2023-07-14T19:21:08.665 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-mako-1.0.6-13.el8.noarch 106/134 2023-07-14T19:21:08.666 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : oniguruma-6.8.2-2.el8.x86_64 107/134 2023-07-14T19:21:08.666 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : automake-1.16.1-7.el8.noarch 108/134 2023-07-14T19:21:08.666 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-babel-2.5.1-7.el8.noarch 109/134 2023-07-14T19:21:08.666 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-jsonpatch-1.21-2.el8.noarch 110/134 2023-07-14T19:21:08.666 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : flex-2.6.1-9.el8.x86_64 111/134 2023-07-14T19:21:08.666 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-prettytable-0.7.2-14.el8.noarch 112/134 2023-07-14T19:21:08.666 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-numpy-1:1.14.3-10.el8.x86_64 113/134 2023-07-14T19:21:08.667 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-werkzeug-0.12.2-4.el8.noarch 114/134 2023-07-14T19:21:08.667 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-numpy-f2py-1:1.14.3-10.el8.x86_64 115/134 2023-07-14T19:21:08.667 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : uuid-1.6.2-43.el8.x86_64 116/134 2023-07-14T19:21:08.667 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-pyasn1-0.3.7-6.el8.noarch 117/134 2023-07-14T19:21:08.667 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-jsonpointer-1.10-11.el8.noarch 118/134 2023-07-14T19:21:08.667 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-pyOpenSSL-19.0.0-1.el8.noarch 119/134 2023-07-14T19:21:08.667 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : libtool-2.4.6-25.el8.x86_64 120/134 2023-07-14T19:21:08.667 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : openblas-threads-0.3.15-3.el8.x86_64 121/134 2023-07-14T19:21:08.668 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : libpq-13.5-1.el8.x86_64 122/134 2023-07-14T19:21:08.668 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : openblas-0.3.15-3.el8.x86_64 123/134 2023-07-14T19:21:08.668 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : jq-1.6-3.el8.x86_64 124/134 2023-07-14T19:21:08.668 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : librdkafka-0.11.4-3.el8.x86_64 125/134 2023-07-14T19:21:08.668 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : postgresql-server-10.21-2.module+el8.6.0+15342+5 126/134 2023-07-14T19:21:08.668 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : postgresql-contrib-10.21-2.module+el8.6.0+15342+ 127/134 2023-07-14T19:21:08.668 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : postgresql-10.21-2.module+el8.6.0+15342+53518fac 128/134 2023-07-14T19:21:08.669 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : librados2-2:18.0.0-4940.g81c9387d.el8.x86_64 129/134 2023-07-14T19:21:08.669 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : librados2-1:12.2.7-9.el8.x86_64 130/134 2023-07-14T19:21:08.669 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : librbd1-2:18.0.0-4940.g81c9387d.el8.x86_64 131/134 2023-07-14T19:21:08.669 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : librbd1-1:12.2.7-9.el8.x86_64 132/134 2023-07-14T19:21:08.669 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : iproute-5.15.0-4.el8_6.1.x86_64 133/134 2023-07-14T19:21:08.954 INFO:teuthology.orchestra.run.smithi139.stdout: Running scriptlet: iproute-5.15.0-4.el8.x86_64 134/134 2023-07-14T19:21:08.954 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : ceph-2:18.0.0-4940.g81c9387d.el8.x86_64 1/134 2023-07-14T19:21:08.954 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : ceph-base-2:18.0.0-4940.g81c9387d.el8.x86_64 2/134 2023-07-14T19:21:08.954 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : ceph-common-2:18.0.0-4940.g81c9387d.el8.x86_64 3/134 2023-07-14T19:21:08.955 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : ceph-fuse-2:18.0.0-4940.g81c9387d.el8.x86_64 4/134 2023-07-14T19:21:08.955 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : ceph-immutable-object-cache-2:18.0.0-4940.g81c93 5/134 2023-07-14T19:21:08.955 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : ceph-mds-2:18.0.0-4940.g81c9387d.el8.x86_64 6/134 2023-07-14T19:21:08.955 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : ceph-mgr-2:18.0.0-4940.g81c9387d.el8.x86_64 7/134 2023-07-14T19:21:08.955 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : ceph-mon-2:18.0.0-4940.g81c9387d.el8.x86_64 8/134 2023-07-14T19:21:08.955 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : ceph-osd-2:18.0.0-4940.g81c9387d.el8.x86_64 9/134 2023-07-14T19:21:08.955 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : ceph-radosgw-2:18.0.0-4940.g81c9387d.el8.x86_64 10/134 2023-07-14T19:21:08.956 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : ceph-selinux-2:18.0.0-4940.g81c9387d.el8.x86_64 11/134 2023-07-14T19:21:08.956 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : ceph-test-2:18.0.0-4940.g81c9387d.el8.x86_64 12/134 2023-07-14T19:21:08.956 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : cephfs-mirror-2:18.0.0-4940.g81c9387d.el8.x86_64 13/134 2023-07-14T19:21:08.956 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : libcephfs-devel-2:18.0.0-4940.g81c9387d.el8.x86_ 14/134 2023-07-14T19:21:08.957 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : libcephfs2-2:18.0.0-4940.g81c9387d.el8.x86_64 15/134 2023-07-14T19:21:08.957 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : libcephsqlite-2:18.0.0-4940.g81c9387d.el8.x86_64 16/134 2023-07-14T19:21:08.957 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : librados-devel-2:18.0.0-4940.g81c9387d.el8.x86_6 17/134 2023-07-14T19:21:08.957 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : libradosstriper1-2:18.0.0-4940.g81c9387d.el8.x86 18/134 2023-07-14T19:21:08.957 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : librgw2-2:18.0.0-4940.g81c9387d.el8.x86_64 19/134 2023-07-14T19:21:08.957 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : python3-ceph-argparse-2:18.0.0-4940.g81c9387d.el 20/134 2023-07-14T19:21:08.958 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : python3-ceph-common-2:18.0.0-4940.g81c9387d.el8. 21/134 2023-07-14T19:21:08.958 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : python3-cephfs-2:18.0.0-4940.g81c9387d.el8.x86_6 22/134 2023-07-14T19:21:08.958 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : python3-rados-2:18.0.0-4940.g81c9387d.el8.x86_64 23/134 2023-07-14T19:21:08.958 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : python3-rbd-2:18.0.0-4940.g81c9387d.el8.x86_64 24/134 2023-07-14T19:21:08.958 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : python3-rgw-2:18.0.0-4940.g81c9387d.el8.x86_64 25/134 2023-07-14T19:21:08.958 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : rbd-fuse-2:18.0.0-4940.g81c9387d.el8.x86_64 26/134 2023-07-14T19:21:08.958 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : rbd-mirror-2:18.0.0-4940.g81c9387d.el8.x86_64 27/134 2023-07-14T19:21:08.958 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : rbd-nbd-2:18.0.0-4940.g81c9387d.el8.x86_64 28/134 2023-07-14T19:21:08.959 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : ceph-grafana-dashboards-2:18.0.0-4940.g81c9387d. 29/134 2023-07-14T19:21:08.959 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : ceph-mgr-cephadm-2:18.0.0-4940.g81c9387d.el8.noa 30/134 2023-07-14T19:21:08.959 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : ceph-mgr-dashboard-2:18.0.0-4940.g81c9387d.el8.n 31/134 2023-07-14T19:21:08.959 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : ceph-mgr-diskprediction-local-2:18.0.0-4940.g81c 32/134 2023-07-14T19:21:08.959 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : ceph-mgr-modules-core-2:18.0.0-4940.g81c9387d.el 33/134 2023-07-14T19:21:08.959 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : ceph-mgr-rook-2:18.0.0-4940.g81c9387d.el8.noarch 34/134 2023-07-14T19:21:08.959 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : ceph-prometheus-alerts-2:18.0.0-4940.g81c9387d.e 35/134 2023-07-14T19:21:08.959 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : ceph-volume-2:18.0.0-4940.g81c9387d.el8.noarch 36/134 2023-07-14T19:21:08.960 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : cephadm-2:18.0.0-4940.g81c9387d.el8.noarch 37/134 2023-07-14T19:21:08.960 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : cephfs-top-2:18.0.0-4940.g81c9387d.el8.noarch 38/134 2023-07-14T19:21:08.960 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : gperftools-libs-1:2.7-9.el8.x86_64 39/134 2023-07-14T19:21:08.960 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : liboath-2.6.2-3.el8.x86_64 40/134 2023-07-14T19:21:08.960 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : libunwind-1.3.1-3.el8.x86_64 41/134 2023-07-14T19:21:08.967 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : python3-asyncssh-2.7.0-2.el8.noarch 42/134 2023-07-14T19:21:08.967 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : python3-bcrypt-3.1.6-2.el8.1.x86_64 43/134 2023-07-14T19:21:08.967 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 44/134 2023-07-14T19:21:08.968 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : python3-cachetools-3.1.1-4.el8.noarch 45/134 2023-07-14T19:21:08.968 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : python3-certifi-2018.10.15-7.el8.noarch 46/134 2023-07-14T19:21:08.968 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : python3-cheroot-8.5.2-1.el8.noarch 47/134 2023-07-14T19:21:08.969 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : python3-cherrypy-18.4.0-1.el8.noarch 48/134 2023-07-14T19:21:08.969 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : python3-google-auth-1:1.1.1-10.el8.noarch 49/134 2023-07-14T19:21:08.969 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : python3-jaraco-6.2-6.el8.noarch 50/134 2023-07-14T19:21:08.969 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : python3-jaraco-functools-2.0-4.el8.noarch 51/134 2023-07-14T19:21:08.969 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : python3-kubernetes-1:11.0.0-6.el8.noarch 52/134 2023-07-14T19:21:08.970 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : python3-logutils-0.3.5-11.el8.noarch 53/134 2023-07-14T19:21:08.970 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : python3-more-itertools-7.2.0-3.el8.noarch 54/134 2023-07-14T19:21:08.970 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : python3-natsort-7.1.1-2.el8.noarch 55/134 2023-07-14T19:21:08.970 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : python3-pecan-1.3.2-9.el8.noarch 56/134 2023-07-14T19:21:08.971 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : python3-portend-2.6-1.el8.noarch 57/134 2023-07-14T19:21:08.971 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : python3-repoze-lru-0.7-6.el8.noarch 58/134 2023-07-14T19:21:08.971 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : python3-routes-2.4.1-12.el8.noarch 59/134 2023-07-14T19:21:08.971 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : python3-rsa-4.8-1.el8.noarch 60/134 2023-07-14T19:21:08.972 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : python3-simplegeneric-0.8.1-17.el8.noarch 61/134 2023-07-14T19:21:08.972 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : python3-singledispatch-3.4.0.3-18.el8.noarch 62/134 2023-07-14T19:21:08.972 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : python3-tempora-1.14.1-5.el8.noarch 63/134 2023-07-14T19:21:08.972 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : python3-trustme-0.6.0-4.el8.noarch 64/134 2023-07-14T19:21:08.973 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : python3-waitress-1.4.3-1.el8.noarch 65/134 2023-07-14T19:21:08.973 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : python3-webob-1.8.5-1.el8.1.noarch 66/134 2023-07-14T19:21:08.973 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : python3-websocket-client-0.56.0-5.el8.noarch 67/134 2023-07-14T19:21:08.974 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : python3-webtest-2.0.33-1.el8.noarch 68/134 2023-07-14T19:21:08.974 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : python3-zc-lockfile-2.0-2.el8.noarch 69/134 2023-07-14T19:21:08.974 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : thrift-0.13.0-2.el8.x86_64 70/134 2023-07-14T19:21:08.974 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : xmlstarlet-1.6.1-20.el8.x86_64 71/134 2023-07-14T19:21:08.974 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : librabbitmq-0.9.0-3.el8.x86_64 72/134 2023-07-14T19:21:08.974 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : libselinux-devel-2.9-5.el8.x86_64 73/134 2023-07-14T19:21:08.975 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : python3-jwt-1.6.1-2.el8.noarch 74/134 2023-07-14T19:21:08.975 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : pcre2-utf32-10.32-2.el8.x86_64 75/134 2023-07-14T19:21:08.975 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : keyutils-libs-devel-1.5.10-9.el8.x86_64 76/134 2023-07-14T19:21:08.975 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : libsepol-devel-2.9-3.el8.x86_64 77/134 2023-07-14T19:21:08.976 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : python3-requests-oauthlib-1.0.0-1.el8.noarch 78/134 2023-07-14T19:21:08.976 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : python3-pycparser-2.14-14.el8.noarch 79/134 2023-07-14T19:21:08.976 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : pcre2-utf16-10.32-2.el8.x86_64 80/134 2023-07-14T19:21:08.976 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : krb5-devel-1.18.2-14.el8.x86_64 81/134 2023-07-14T19:21:08.976 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : libaio-devel-0.3.112-1.el8.x86_64 82/134 2023-07-14T19:21:08.976 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : python3-oauthlib-2.1.0-1.el8.noarch 83/134 2023-07-14T19:21:08.977 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : python3-cryptography-3.2.1-5.el8.x86_64 84/134 2023-07-14T19:21:08.977 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : libverto-devel-0.3.0-5.el8.x86_64 85/134 2023-07-14T19:21:08.977 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : pcre2-devel-10.32-2.el8.x86_64 86/134 2023-07-14T19:21:08.977 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : python3-cffi-1.11.5-5.el8.x86_64 87/134 2023-07-14T19:21:08.977 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : openssl-devel-1:1.1.1k-7.el8_6.x86_64 88/134 2023-07-14T19:21:08.978 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : xfsprogs-devel-5.0.0-10.el8.x86_64 89/134 2023-07-14T19:21:08.978 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : fuse-2.9.7-15.el8.x86_64 90/134 2023-07-14T19:21:08.978 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : elfutils-libelf-devel-0.186-1.el8.x86_64 91/134 2023-07-14T19:21:08.979 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : libuuid-devel-2.32.1-35.el8.x86_64 92/134 2023-07-14T19:21:08.979 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : zlib-devel-1.2.11-18.el8_5.x86_64 93/134 2023-07-14T19:21:08.979 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : libcom_err-devel-1.45.6-4.el8.x86_64 94/134 2023-07-14T19:21:08.980 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : libgfortran-8.5.0-10.1.el8_6.x86_64 95/134 2023-07-14T19:21:08.980 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : libquadmath-8.5.0-10.1.el8_6.x86_64 96/134 2023-07-14T19:21:08.980 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : NetworkManager-initscripts-updown-1:1.36.0-9.el8 97/134 2023-07-14T19:21:08.980 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : python3-scipy-1.0.0-21.module+el8.5.0+10916+41bd 98/134 2023-07-14T19:21:08.981 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : python3-pyasn1-modules-0.3.7-6.el8.noarch 99/134 2023-07-14T19:21:08.981 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : python3-pytz-2017.2-9.el8.noarch 100/134 2023-07-14T19:21:08.981 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : lttng-ust-2.8.1-11.el8.x86_64 101/134 2023-07-14T19:21:08.982 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : socat-1.7.4.1-1.el8.x86_64 102/134 2023-07-14T19:21:08.982 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : python3-markupsafe-0.23-19.el8.x86_64 103/134 2023-07-14T19:21:08.982 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : bison-3.0.4-10.el8.x86_64 104/134 2023-07-14T19:21:08.982 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : python3-jinja2-2.10.1-3.el8.noarch 105/134 2023-07-14T19:21:08.982 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : python3-mako-1.0.6-13.el8.noarch 106/134 2023-07-14T19:21:08.983 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : oniguruma-6.8.2-2.el8.x86_64 107/134 2023-07-14T19:21:08.983 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : automake-1.16.1-7.el8.noarch 108/134 2023-07-14T19:21:08.983 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : python3-babel-2.5.1-7.el8.noarch 109/134 2023-07-14T19:21:08.983 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : python3-jsonpatch-1.21-2.el8.noarch 110/134 2023-07-14T19:21:08.984 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : flex-2.6.1-9.el8.x86_64 111/134 2023-07-14T19:21:08.984 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : python3-prettytable-0.7.2-14.el8.noarch 112/134 2023-07-14T19:21:08.984 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : python3-numpy-1:1.14.3-10.el8.x86_64 113/134 2023-07-14T19:21:08.984 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : python3-werkzeug-0.12.2-4.el8.noarch 114/134 2023-07-14T19:21:08.984 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : python3-numpy-f2py-1:1.14.3-10.el8.x86_64 115/134 2023-07-14T19:21:08.984 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : uuid-1.6.2-43.el8.x86_64 116/134 2023-07-14T19:21:08.985 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : python3-pyasn1-0.3.7-6.el8.noarch 117/134 2023-07-14T19:21:08.985 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : python3-jsonpointer-1.10-11.el8.noarch 118/134 2023-07-14T19:21:08.985 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : python3-pyOpenSSL-19.0.0-1.el8.noarch 119/134 2023-07-14T19:21:08.985 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : libtool-2.4.6-25.el8.x86_64 120/134 2023-07-14T19:21:08.985 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : openblas-threads-0.3.15-3.el8.x86_64 121/134 2023-07-14T19:21:08.985 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : libpq-13.5-1.el8.x86_64 122/134 2023-07-14T19:21:08.985 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : openblas-0.3.15-3.el8.x86_64 123/134 2023-07-14T19:21:08.985 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : jq-1.6-3.el8.x86_64 124/134 2023-07-14T19:21:08.986 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : librdkafka-0.11.4-3.el8.x86_64 125/134 2023-07-14T19:21:08.986 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : postgresql-server-10.21-2.module+el8.6.0+15342+5 126/134 2023-07-14T19:21:08.986 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : postgresql-contrib-10.21-2.module+el8.6.0+15342+ 127/134 2023-07-14T19:21:08.986 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : postgresql-10.21-2.module+el8.6.0+15342+53518fac 128/134 2023-07-14T19:21:08.986 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : librados2-2:18.0.0-4940.g81c9387d.el8.x86_64 129/134 2023-07-14T19:21:08.986 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : librados2-1:12.2.7-9.el8.x86_64 130/134 2023-07-14T19:21:08.986 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : librbd1-2:18.0.0-4940.g81c9387d.el8.x86_64 131/134 2023-07-14T19:21:08.986 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : librbd1-1:12.2.7-9.el8.x86_64 132/134 2023-07-14T19:21:08.987 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : iproute-5.15.0-4.el8_6.1.x86_64 133/134 2023-07-14T19:21:11.125 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : iproute-5.15.0-4.el8.x86_64 134/134 2023-07-14T19:21:11.126 INFO:teuthology.orchestra.run.smithi098.stdout:Last metadata expiration check: 0:03:53 ago on Fri 14 Jul 2023 07:17:18 PM UTC. 2023-07-14T19:21:11.499 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : iproute-5.15.0-4.el8.x86_64 134/134 2023-07-14T19:21:11.499 INFO:teuthology.orchestra.run.smithi139.stdout:Last metadata expiration check: 0:03:29 ago on Fri 14 Jul 2023 07:17:42 PM UTC. 2023-07-14T19:21:14.152 INFO:teuthology.orchestra.run.smithi098.stdout:Installed products updated. 2023-07-14T19:21:14.285 INFO:teuthology.orchestra.run.smithi139.stdout:Installed products updated. 2023-07-14T19:21:14.364 INFO:teuthology.orchestra.run.smithi098.stdout: 2023-07-14T19:21:14.364 INFO:teuthology.orchestra.run.smithi098.stdout:Upgraded: 2023-07-14T19:21:14.364 INFO:teuthology.orchestra.run.smithi098.stdout: iproute-5.15.0-4.el8_6.1.x86_64 2023-07-14T19:21:14.364 INFO:teuthology.orchestra.run.smithi098.stdout: librados2-2:18.0.0-4940.g81c9387d.el8.x86_64 2023-07-14T19:21:14.365 INFO:teuthology.orchestra.run.smithi098.stdout: librbd1-2:18.0.0-4940.g81c9387d.el8.x86_64 2023-07-14T19:21:14.365 INFO:teuthology.orchestra.run.smithi098.stdout:Installed: 2023-07-14T19:21:14.365 INFO:teuthology.orchestra.run.smithi098.stdout: NetworkManager-initscripts-updown-1:1.36.0-9.el8_6.noarch 2023-07-14T19:21:14.365 INFO:teuthology.orchestra.run.smithi098.stdout: automake-1.16.1-7.el8.noarch 2023-07-14T19:21:14.365 INFO:teuthology.orchestra.run.smithi098.stdout: bison-3.0.4-10.el8.x86_64 2023-07-14T19:21:14.365 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-2:18.0.0-4940.g81c9387d.el8.x86_64 2023-07-14T19:21:14.365 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-base-2:18.0.0-4940.g81c9387d.el8.x86_64 2023-07-14T19:21:14.366 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-common-2:18.0.0-4940.g81c9387d.el8.x86_64 2023-07-14T19:21:14.366 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-fuse-2:18.0.0-4940.g81c9387d.el8.x86_64 2023-07-14T19:21:14.366 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-grafana-dashboards-2:18.0.0-4940.g81c9387d.el8.noarch 2023-07-14T19:21:14.366 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-immutable-object-cache-2:18.0.0-4940.g81c9387d.el8.x86_64 2023-07-14T19:21:14.366 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-mds-2:18.0.0-4940.g81c9387d.el8.x86_64 2023-07-14T19:21:14.366 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-mgr-2:18.0.0-4940.g81c9387d.el8.x86_64 2023-07-14T19:21:14.366 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-mgr-cephadm-2:18.0.0-4940.g81c9387d.el8.noarch 2023-07-14T19:21:14.367 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-mgr-dashboard-2:18.0.0-4940.g81c9387d.el8.noarch 2023-07-14T19:21:14.367 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-mgr-diskprediction-local-2:18.0.0-4940.g81c9387d.el8.noarch 2023-07-14T19:21:14.367 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-mgr-modules-core-2:18.0.0-4940.g81c9387d.el8.noarch 2023-07-14T19:21:14.367 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-mgr-rook-2:18.0.0-4940.g81c9387d.el8.noarch 2023-07-14T19:21:14.367 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-mon-2:18.0.0-4940.g81c9387d.el8.x86_64 2023-07-14T19:21:14.367 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-osd-2:18.0.0-4940.g81c9387d.el8.x86_64 2023-07-14T19:21:14.367 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-prometheus-alerts-2:18.0.0-4940.g81c9387d.el8.noarch 2023-07-14T19:21:14.367 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-radosgw-2:18.0.0-4940.g81c9387d.el8.x86_64 2023-07-14T19:21:14.368 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-selinux-2:18.0.0-4940.g81c9387d.el8.x86_64 2023-07-14T19:21:14.368 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-test-2:18.0.0-4940.g81c9387d.el8.x86_64 2023-07-14T19:21:14.368 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-volume-2:18.0.0-4940.g81c9387d.el8.noarch 2023-07-14T19:21:14.368 INFO:teuthology.orchestra.run.smithi098.stdout: cephadm-2:18.0.0-4940.g81c9387d.el8.noarch 2023-07-14T19:21:14.368 INFO:teuthology.orchestra.run.smithi098.stdout: cephfs-mirror-2:18.0.0-4940.g81c9387d.el8.x86_64 2023-07-14T19:21:14.368 INFO:teuthology.orchestra.run.smithi098.stdout: cephfs-top-2:18.0.0-4940.g81c9387d.el8.noarch 2023-07-14T19:21:14.368 INFO:teuthology.orchestra.run.smithi098.stdout: elfutils-libelf-devel-0.186-1.el8.x86_64 2023-07-14T19:21:14.369 INFO:teuthology.orchestra.run.smithi098.stdout: flex-2.6.1-9.el8.x86_64 2023-07-14T19:21:14.369 INFO:teuthology.orchestra.run.smithi098.stdout: fuse-2.9.7-15.el8.x86_64 2023-07-14T19:21:14.369 INFO:teuthology.orchestra.run.smithi098.stdout: gperftools-libs-1:2.7-9.el8.x86_64 2023-07-14T19:21:14.369 INFO:teuthology.orchestra.run.smithi098.stdout: jq-1.6-3.el8.x86_64 2023-07-14T19:21:14.369 INFO:teuthology.orchestra.run.smithi098.stdout: keyutils-libs-devel-1.5.10-9.el8.x86_64 2023-07-14T19:21:14.369 INFO:teuthology.orchestra.run.smithi098.stdout: krb5-devel-1.18.2-14.el8.x86_64 2023-07-14T19:21:14.369 INFO:teuthology.orchestra.run.smithi098.stdout: libaio-devel-0.3.112-1.el8.x86_64 2023-07-14T19:21:14.369 INFO:teuthology.orchestra.run.smithi098.stdout: libcephfs-devel-2:18.0.0-4940.g81c9387d.el8.x86_64 2023-07-14T19:21:14.370 INFO:teuthology.orchestra.run.smithi098.stdout: libcephfs2-2:18.0.0-4940.g81c9387d.el8.x86_64 2023-07-14T19:21:14.370 INFO:teuthology.orchestra.run.smithi098.stdout: libcephsqlite-2:18.0.0-4940.g81c9387d.el8.x86_64 2023-07-14T19:21:14.370 INFO:teuthology.orchestra.run.smithi098.stdout: libcom_err-devel-1.45.6-4.el8.x86_64 2023-07-14T19:21:14.370 INFO:teuthology.orchestra.run.smithi098.stdout: libgfortran-8.5.0-10.1.el8_6.x86_64 2023-07-14T19:21:14.370 INFO:teuthology.orchestra.run.smithi098.stdout: liboath-2.6.2-3.el8.x86_64 2023-07-14T19:21:14.370 INFO:teuthology.orchestra.run.smithi098.stdout: libpq-13.5-1.el8.x86_64 2023-07-14T19:21:14.370 INFO:teuthology.orchestra.run.smithi098.stdout: libquadmath-8.5.0-10.1.el8_6.x86_64 2023-07-14T19:21:14.371 INFO:teuthology.orchestra.run.smithi098.stdout: librabbitmq-0.9.0-3.el8.x86_64 2023-07-14T19:21:14.371 INFO:teuthology.orchestra.run.smithi098.stdout: librados-devel-2:18.0.0-4940.g81c9387d.el8.x86_64 2023-07-14T19:21:14.371 INFO:teuthology.orchestra.run.smithi098.stdout: libradosstriper1-2:18.0.0-4940.g81c9387d.el8.x86_64 2023-07-14T19:21:14.371 INFO:teuthology.orchestra.run.smithi098.stdout: librdkafka-0.11.4-3.el8.x86_64 2023-07-14T19:21:14.371 INFO:teuthology.orchestra.run.smithi098.stdout: librgw2-2:18.0.0-4940.g81c9387d.el8.x86_64 2023-07-14T19:21:14.371 INFO:teuthology.orchestra.run.smithi098.stdout: libselinux-devel-2.9-5.el8.x86_64 2023-07-14T19:21:14.371 INFO:teuthology.orchestra.run.smithi098.stdout: libsepol-devel-2.9-3.el8.x86_64 2023-07-14T19:21:14.372 INFO:teuthology.orchestra.run.smithi098.stdout: libtool-2.4.6-25.el8.x86_64 2023-07-14T19:21:14.372 INFO:teuthology.orchestra.run.smithi098.stdout: libunwind-1.3.1-3.el8.x86_64 2023-07-14T19:21:14.372 INFO:teuthology.orchestra.run.smithi098.stdout: libuuid-devel-2.32.1-35.el8.x86_64 2023-07-14T19:21:14.372 INFO:teuthology.orchestra.run.smithi098.stdout: libverto-devel-0.3.0-5.el8.x86_64 2023-07-14T19:21:14.372 INFO:teuthology.orchestra.run.smithi098.stdout: lttng-ust-2.8.1-11.el8.x86_64 2023-07-14T19:21:14.372 INFO:teuthology.orchestra.run.smithi098.stdout: oniguruma-6.8.2-2.el8.x86_64 2023-07-14T19:21:14.372 INFO:teuthology.orchestra.run.smithi098.stdout: openblas-0.3.15-3.el8.x86_64 2023-07-14T19:21:14.373 INFO:teuthology.orchestra.run.smithi098.stdout: openblas-threads-0.3.15-3.el8.x86_64 2023-07-14T19:21:14.373 INFO:teuthology.orchestra.run.smithi098.stdout: openssl-devel-1:1.1.1k-7.el8_6.x86_64 2023-07-14T19:21:14.373 INFO:teuthology.orchestra.run.smithi098.stdout: pcre2-devel-10.32-2.el8.x86_64 2023-07-14T19:21:14.373 INFO:teuthology.orchestra.run.smithi098.stdout: pcre2-utf16-10.32-2.el8.x86_64 2023-07-14T19:21:14.373 INFO:teuthology.orchestra.run.smithi098.stdout: pcre2-utf32-10.32-2.el8.x86_64 2023-07-14T19:21:14.373 INFO:teuthology.orchestra.run.smithi098.stdout: postgresql-10.21-2.module+el8.6.0+15342+53518fac.x86_64 2023-07-14T19:21:14.373 INFO:teuthology.orchestra.run.smithi098.stdout: postgresql-contrib-10.21-2.module+el8.6.0+15342+53518fac.x86_64 2023-07-14T19:21:14.373 INFO:teuthology.orchestra.run.smithi098.stdout: postgresql-server-10.21-2.module+el8.6.0+15342+53518fac.x86_64 2023-07-14T19:21:14.374 INFO:teuthology.orchestra.run.smithi098.stdout: python3-asyncssh-2.7.0-2.el8.noarch 2023-07-14T19:21:14.374 INFO:teuthology.orchestra.run.smithi098.stdout: python3-babel-2.5.1-7.el8.noarch 2023-07-14T19:21:14.374 INFO:teuthology.orchestra.run.smithi098.stdout: python3-bcrypt-3.1.6-2.el8.1.x86_64 2023-07-14T19:21:14.374 INFO:teuthology.orchestra.run.smithi098.stdout: python3-beautifulsoup4-4.6.3-2.el8.1.noarch 2023-07-14T19:21:14.374 INFO:teuthology.orchestra.run.smithi098.stdout: python3-cachetools-3.1.1-4.el8.noarch 2023-07-14T19:21:14.374 INFO:teuthology.orchestra.run.smithi098.stdout: python3-ceph-argparse-2:18.0.0-4940.g81c9387d.el8.x86_64 2023-07-14T19:21:14.374 INFO:teuthology.orchestra.run.smithi098.stdout: python3-ceph-common-2:18.0.0-4940.g81c9387d.el8.x86_64 2023-07-14T19:21:14.374 INFO:teuthology.orchestra.run.smithi098.stdout: python3-cephfs-2:18.0.0-4940.g81c9387d.el8.x86_64 2023-07-14T19:21:14.375 INFO:teuthology.orchestra.run.smithi098.stdout: python3-certifi-2018.10.15-7.el8.noarch 2023-07-14T19:21:14.375 INFO:teuthology.orchestra.run.smithi098.stdout: python3-cffi-1.11.5-5.el8.x86_64 2023-07-14T19:21:14.375 INFO:teuthology.orchestra.run.smithi098.stdout: python3-cheroot-8.5.2-1.el8.noarch 2023-07-14T19:21:14.375 INFO:teuthology.orchestra.run.smithi098.stdout: python3-cherrypy-18.4.0-1.el8.noarch 2023-07-14T19:21:14.375 INFO:teuthology.orchestra.run.smithi098.stdout: python3-cryptography-3.2.1-5.el8.x86_64 2023-07-14T19:21:14.375 INFO:teuthology.orchestra.run.smithi098.stdout: python3-google-auth-1:1.1.1-10.el8.noarch 2023-07-14T19:21:14.375 INFO:teuthology.orchestra.run.smithi098.stdout: python3-jaraco-6.2-6.el8.noarch 2023-07-14T19:21:14.375 INFO:teuthology.orchestra.run.smithi098.stdout: python3-jaraco-functools-2.0-4.el8.noarch 2023-07-14T19:21:14.376 INFO:teuthology.orchestra.run.smithi098.stdout: python3-jinja2-2.10.1-3.el8.noarch 2023-07-14T19:21:14.376 INFO:teuthology.orchestra.run.smithi098.stdout: python3-jsonpatch-1.21-2.el8.noarch 2023-07-14T19:21:14.376 INFO:teuthology.orchestra.run.smithi098.stdout: python3-jsonpointer-1.10-11.el8.noarch 2023-07-14T19:21:14.376 INFO:teuthology.orchestra.run.smithi098.stdout: python3-jwt-1.6.1-2.el8.noarch 2023-07-14T19:21:14.376 INFO:teuthology.orchestra.run.smithi098.stdout: python3-kubernetes-1:11.0.0-6.el8.noarch 2023-07-14T19:21:14.376 INFO:teuthology.orchestra.run.smithi098.stdout: python3-logutils-0.3.5-11.el8.noarch 2023-07-14T19:21:14.376 INFO:teuthology.orchestra.run.smithi098.stdout: python3-mako-1.0.6-13.el8.noarch 2023-07-14T19:21:14.377 INFO:teuthology.orchestra.run.smithi098.stdout: python3-markupsafe-0.23-19.el8.x86_64 2023-07-14T19:21:14.377 INFO:teuthology.orchestra.run.smithi098.stdout: python3-more-itertools-7.2.0-3.el8.noarch 2023-07-14T19:21:14.377 INFO:teuthology.orchestra.run.smithi098.stdout: python3-natsort-7.1.1-2.el8.noarch 2023-07-14T19:21:14.377 INFO:teuthology.orchestra.run.smithi098.stdout: python3-numpy-1:1.14.3-10.el8.x86_64 2023-07-14T19:21:14.377 INFO:teuthology.orchestra.run.smithi098.stdout: python3-numpy-f2py-1:1.14.3-10.el8.x86_64 2023-07-14T19:21:14.377 INFO:teuthology.orchestra.run.smithi098.stdout: python3-oauthlib-2.1.0-1.el8.noarch 2023-07-14T19:21:14.377 INFO:teuthology.orchestra.run.smithi098.stdout: python3-pecan-1.3.2-9.el8.noarch 2023-07-14T19:21:14.377 INFO:teuthology.orchestra.run.smithi098.stdout: python3-portend-2.6-1.el8.noarch 2023-07-14T19:21:14.378 INFO:teuthology.orchestra.run.smithi098.stdout: python3-prettytable-0.7.2-14.el8.noarch 2023-07-14T19:21:14.378 INFO:teuthology.orchestra.run.smithi098.stdout: python3-pyOpenSSL-19.0.0-1.el8.noarch 2023-07-14T19:21:14.378 INFO:teuthology.orchestra.run.smithi098.stdout: python3-pyasn1-0.3.7-6.el8.noarch 2023-07-14T19:21:14.378 INFO:teuthology.orchestra.run.smithi098.stdout: python3-pyasn1-modules-0.3.7-6.el8.noarch 2023-07-14T19:21:14.378 INFO:teuthology.orchestra.run.smithi098.stdout: python3-pycparser-2.14-14.el8.noarch 2023-07-14T19:21:14.378 INFO:teuthology.orchestra.run.smithi098.stdout: python3-pytz-2017.2-9.el8.noarch 2023-07-14T19:21:14.378 INFO:teuthology.orchestra.run.smithi098.stdout: python3-rados-2:18.0.0-4940.g81c9387d.el8.x86_64 2023-07-14T19:21:14.379 INFO:teuthology.orchestra.run.smithi098.stdout: python3-rbd-2:18.0.0-4940.g81c9387d.el8.x86_64 2023-07-14T19:21:14.379 INFO:teuthology.orchestra.run.smithi098.stdout: python3-repoze-lru-0.7-6.el8.noarch 2023-07-14T19:21:14.379 INFO:teuthology.orchestra.run.smithi098.stdout: python3-requests-oauthlib-1.0.0-1.el8.noarch 2023-07-14T19:21:14.379 INFO:teuthology.orchestra.run.smithi098.stdout: python3-rgw-2:18.0.0-4940.g81c9387d.el8.x86_64 2023-07-14T19:21:14.379 INFO:teuthology.orchestra.run.smithi098.stdout: python3-routes-2.4.1-12.el8.noarch 2023-07-14T19:21:14.379 INFO:teuthology.orchestra.run.smithi098.stdout: python3-rsa-4.8-1.el8.noarch 2023-07-14T19:21:14.379 INFO:teuthology.orchestra.run.smithi098.stdout: python3-scipy-1.0.0-21.module+el8.5.0+10916+41bd434d.x86_64 2023-07-14T19:21:14.380 INFO:teuthology.orchestra.run.smithi098.stdout: python3-simplegeneric-0.8.1-17.el8.noarch 2023-07-14T19:21:14.380 INFO:teuthology.orchestra.run.smithi098.stdout: python3-singledispatch-3.4.0.3-18.el8.noarch 2023-07-14T19:21:14.381 INFO:teuthology.orchestra.run.smithi098.stdout: python3-tempora-1.14.1-5.el8.noarch 2023-07-14T19:21:14.381 INFO:teuthology.orchestra.run.smithi098.stdout: python3-trustme-0.6.0-4.el8.noarch 2023-07-14T19:21:14.381 INFO:teuthology.orchestra.run.smithi098.stdout: python3-waitress-1.4.3-1.el8.noarch 2023-07-14T19:21:14.381 INFO:teuthology.orchestra.run.smithi098.stdout: python3-webob-1.8.5-1.el8.1.noarch 2023-07-14T19:21:14.381 INFO:teuthology.orchestra.run.smithi098.stdout: python3-websocket-client-0.56.0-5.el8.noarch 2023-07-14T19:21:14.381 INFO:teuthology.orchestra.run.smithi098.stdout: python3-webtest-2.0.33-1.el8.noarch 2023-07-14T19:21:14.381 INFO:teuthology.orchestra.run.smithi098.stdout: python3-werkzeug-0.12.2-4.el8.noarch 2023-07-14T19:21:14.382 INFO:teuthology.orchestra.run.smithi098.stdout: python3-zc-lockfile-2.0-2.el8.noarch 2023-07-14T19:21:14.382 INFO:teuthology.orchestra.run.smithi098.stdout: rbd-fuse-2:18.0.0-4940.g81c9387d.el8.x86_64 2023-07-14T19:21:14.382 INFO:teuthology.orchestra.run.smithi098.stdout: rbd-mirror-2:18.0.0-4940.g81c9387d.el8.x86_64 2023-07-14T19:21:14.382 INFO:teuthology.orchestra.run.smithi098.stdout: rbd-nbd-2:18.0.0-4940.g81c9387d.el8.x86_64 2023-07-14T19:21:14.382 INFO:teuthology.orchestra.run.smithi098.stdout: socat-1.7.4.1-1.el8.x86_64 2023-07-14T19:21:14.382 INFO:teuthology.orchestra.run.smithi098.stdout: thrift-0.13.0-2.el8.x86_64 2023-07-14T19:21:14.382 INFO:teuthology.orchestra.run.smithi098.stdout: uuid-1.6.2-43.el8.x86_64 2023-07-14T19:21:14.383 INFO:teuthology.orchestra.run.smithi098.stdout: xfsprogs-devel-5.0.0-10.el8.x86_64 2023-07-14T19:21:14.384 INFO:teuthology.orchestra.run.smithi098.stdout: xmlstarlet-1.6.1-20.el8.x86_64 2023-07-14T19:21:14.384 INFO:teuthology.orchestra.run.smithi098.stdout: zlib-devel-1.2.11-18.el8_5.x86_64 2023-07-14T19:21:14.385 INFO:teuthology.orchestra.run.smithi098.stdout: 2023-07-14T19:21:14.385 INFO:teuthology.orchestra.run.smithi098.stdout:Complete! 2023-07-14T19:21:14.496 INFO:teuthology.orchestra.run.smithi139.stdout: 2023-07-14T19:21:14.496 INFO:teuthology.orchestra.run.smithi139.stdout:Upgraded: 2023-07-14T19:21:14.497 INFO:teuthology.orchestra.run.smithi139.stdout: iproute-5.15.0-4.el8_6.1.x86_64 2023-07-14T19:21:14.497 INFO:teuthology.orchestra.run.smithi139.stdout: librados2-2:18.0.0-4940.g81c9387d.el8.x86_64 2023-07-14T19:21:14.497 INFO:teuthology.orchestra.run.smithi139.stdout: librbd1-2:18.0.0-4940.g81c9387d.el8.x86_64 2023-07-14T19:21:14.497 INFO:teuthology.orchestra.run.smithi139.stdout:Installed: 2023-07-14T19:21:14.497 INFO:teuthology.orchestra.run.smithi139.stdout: NetworkManager-initscripts-updown-1:1.36.0-9.el8_6.noarch 2023-07-14T19:21:14.497 INFO:teuthology.orchestra.run.smithi139.stdout: automake-1.16.1-7.el8.noarch 2023-07-14T19:21:14.497 INFO:teuthology.orchestra.run.smithi139.stdout: bison-3.0.4-10.el8.x86_64 2023-07-14T19:21:14.498 INFO:teuthology.orchestra.run.smithi139.stdout: ceph-2:18.0.0-4940.g81c9387d.el8.x86_64 2023-07-14T19:21:14.498 INFO:teuthology.orchestra.run.smithi139.stdout: ceph-base-2:18.0.0-4940.g81c9387d.el8.x86_64 2023-07-14T19:21:14.498 INFO:teuthology.orchestra.run.smithi139.stdout: ceph-common-2:18.0.0-4940.g81c9387d.el8.x86_64 2023-07-14T19:21:14.498 INFO:teuthology.orchestra.run.smithi139.stdout: ceph-fuse-2:18.0.0-4940.g81c9387d.el8.x86_64 2023-07-14T19:21:14.498 INFO:teuthology.orchestra.run.smithi139.stdout: ceph-grafana-dashboards-2:18.0.0-4940.g81c9387d.el8.noarch 2023-07-14T19:21:14.498 INFO:teuthology.orchestra.run.smithi139.stdout: ceph-immutable-object-cache-2:18.0.0-4940.g81c9387d.el8.x86_64 2023-07-14T19:21:14.498 INFO:teuthology.orchestra.run.smithi139.stdout: ceph-mds-2:18.0.0-4940.g81c9387d.el8.x86_64 2023-07-14T19:21:14.499 INFO:teuthology.orchestra.run.smithi139.stdout: ceph-mgr-2:18.0.0-4940.g81c9387d.el8.x86_64 2023-07-14T19:21:14.499 INFO:teuthology.orchestra.run.smithi139.stdout: ceph-mgr-cephadm-2:18.0.0-4940.g81c9387d.el8.noarch 2023-07-14T19:21:14.499 INFO:teuthology.orchestra.run.smithi139.stdout: ceph-mgr-dashboard-2:18.0.0-4940.g81c9387d.el8.noarch 2023-07-14T19:21:14.499 INFO:teuthology.orchestra.run.smithi139.stdout: ceph-mgr-diskprediction-local-2:18.0.0-4940.g81c9387d.el8.noarch 2023-07-14T19:21:14.499 INFO:teuthology.orchestra.run.smithi139.stdout: ceph-mgr-modules-core-2:18.0.0-4940.g81c9387d.el8.noarch 2023-07-14T19:21:14.499 INFO:teuthology.orchestra.run.smithi139.stdout: ceph-mgr-rook-2:18.0.0-4940.g81c9387d.el8.noarch 2023-07-14T19:21:14.499 INFO:teuthology.orchestra.run.smithi139.stdout: ceph-mon-2:18.0.0-4940.g81c9387d.el8.x86_64 2023-07-14T19:21:14.499 INFO:teuthology.orchestra.run.smithi139.stdout: ceph-osd-2:18.0.0-4940.g81c9387d.el8.x86_64 2023-07-14T19:21:14.500 INFO:teuthology.orchestra.run.smithi139.stdout: ceph-prometheus-alerts-2:18.0.0-4940.g81c9387d.el8.noarch 2023-07-14T19:21:14.500 INFO:teuthology.orchestra.run.smithi139.stdout: ceph-radosgw-2:18.0.0-4940.g81c9387d.el8.x86_64 2023-07-14T19:21:14.500 INFO:teuthology.orchestra.run.smithi139.stdout: ceph-selinux-2:18.0.0-4940.g81c9387d.el8.x86_64 2023-07-14T19:21:14.500 INFO:teuthology.orchestra.run.smithi139.stdout: ceph-test-2:18.0.0-4940.g81c9387d.el8.x86_64 2023-07-14T19:21:14.500 INFO:teuthology.orchestra.run.smithi139.stdout: ceph-volume-2:18.0.0-4940.g81c9387d.el8.noarch 2023-07-14T19:21:14.500 INFO:teuthology.orchestra.run.smithi139.stdout: cephadm-2:18.0.0-4940.g81c9387d.el8.noarch 2023-07-14T19:21:14.501 INFO:teuthology.orchestra.run.smithi139.stdout: cephfs-mirror-2:18.0.0-4940.g81c9387d.el8.x86_64 2023-07-14T19:21:14.501 INFO:teuthology.orchestra.run.smithi139.stdout: cephfs-top-2:18.0.0-4940.g81c9387d.el8.noarch 2023-07-14T19:21:14.501 INFO:teuthology.orchestra.run.smithi139.stdout: elfutils-libelf-devel-0.186-1.el8.x86_64 2023-07-14T19:21:14.501 INFO:teuthology.orchestra.run.smithi139.stdout: flex-2.6.1-9.el8.x86_64 2023-07-14T19:21:14.501 INFO:teuthology.orchestra.run.smithi139.stdout: fuse-2.9.7-15.el8.x86_64 2023-07-14T19:21:14.501 INFO:teuthology.orchestra.run.smithi139.stdout: gperftools-libs-1:2.7-9.el8.x86_64 2023-07-14T19:21:14.501 INFO:teuthology.orchestra.run.smithi139.stdout: jq-1.6-3.el8.x86_64 2023-07-14T19:21:14.502 INFO:teuthology.orchestra.run.smithi139.stdout: keyutils-libs-devel-1.5.10-9.el8.x86_64 2023-07-14T19:21:14.502 INFO:teuthology.orchestra.run.smithi139.stdout: krb5-devel-1.18.2-14.el8.x86_64 2023-07-14T19:21:14.502 INFO:teuthology.orchestra.run.smithi139.stdout: libaio-devel-0.3.112-1.el8.x86_64 2023-07-14T19:21:14.502 INFO:teuthology.orchestra.run.smithi139.stdout: libcephfs-devel-2:18.0.0-4940.g81c9387d.el8.x86_64 2023-07-14T19:21:14.502 INFO:teuthology.orchestra.run.smithi139.stdout: libcephfs2-2:18.0.0-4940.g81c9387d.el8.x86_64 2023-07-14T19:21:14.502 INFO:teuthology.orchestra.run.smithi139.stdout: libcephsqlite-2:18.0.0-4940.g81c9387d.el8.x86_64 2023-07-14T19:21:14.502 INFO:teuthology.orchestra.run.smithi139.stdout: libcom_err-devel-1.45.6-4.el8.x86_64 2023-07-14T19:21:14.503 INFO:teuthology.orchestra.run.smithi139.stdout: libgfortran-8.5.0-10.1.el8_6.x86_64 2023-07-14T19:21:14.503 INFO:teuthology.orchestra.run.smithi139.stdout: liboath-2.6.2-3.el8.x86_64 2023-07-14T19:21:14.503 INFO:teuthology.orchestra.run.smithi139.stdout: libpq-13.5-1.el8.x86_64 2023-07-14T19:21:14.503 INFO:teuthology.orchestra.run.smithi139.stdout: libquadmath-8.5.0-10.1.el8_6.x86_64 2023-07-14T19:21:14.503 INFO:teuthology.orchestra.run.smithi139.stdout: librabbitmq-0.9.0-3.el8.x86_64 2023-07-14T19:21:14.503 INFO:teuthology.orchestra.run.smithi139.stdout: librados-devel-2:18.0.0-4940.g81c9387d.el8.x86_64 2023-07-14T19:21:14.503 INFO:teuthology.orchestra.run.smithi139.stdout: libradosstriper1-2:18.0.0-4940.g81c9387d.el8.x86_64 2023-07-14T19:21:14.504 INFO:teuthology.orchestra.run.smithi139.stdout: librdkafka-0.11.4-3.el8.x86_64 2023-07-14T19:21:14.504 INFO:teuthology.orchestra.run.smithi139.stdout: librgw2-2:18.0.0-4940.g81c9387d.el8.x86_64 2023-07-14T19:21:14.504 INFO:teuthology.orchestra.run.smithi139.stdout: libselinux-devel-2.9-5.el8.x86_64 2023-07-14T19:21:14.504 INFO:teuthology.orchestra.run.smithi139.stdout: libsepol-devel-2.9-3.el8.x86_64 2023-07-14T19:21:14.504 INFO:teuthology.orchestra.run.smithi139.stdout: libtool-2.4.6-25.el8.x86_64 2023-07-14T19:21:14.504 INFO:teuthology.orchestra.run.smithi139.stdout: libunwind-1.3.1-3.el8.x86_64 2023-07-14T19:21:14.505 INFO:teuthology.orchestra.run.smithi139.stdout: libuuid-devel-2.32.1-35.el8.x86_64 2023-07-14T19:21:14.505 INFO:teuthology.orchestra.run.smithi139.stdout: libverto-devel-0.3.0-5.el8.x86_64 2023-07-14T19:21:14.505 INFO:teuthology.orchestra.run.smithi139.stdout: lttng-ust-2.8.1-11.el8.x86_64 2023-07-14T19:21:14.505 INFO:teuthology.orchestra.run.smithi139.stdout: oniguruma-6.8.2-2.el8.x86_64 2023-07-14T19:21:14.505 INFO:teuthology.orchestra.run.smithi139.stdout: openblas-0.3.15-3.el8.x86_64 2023-07-14T19:21:14.506 INFO:teuthology.orchestra.run.smithi139.stdout: openblas-threads-0.3.15-3.el8.x86_64 2023-07-14T19:21:14.506 INFO:teuthology.orchestra.run.smithi139.stdout: openssl-devel-1:1.1.1k-7.el8_6.x86_64 2023-07-14T19:21:14.506 INFO:teuthology.orchestra.run.smithi139.stdout: pcre2-devel-10.32-2.el8.x86_64 2023-07-14T19:21:14.506 INFO:teuthology.orchestra.run.smithi139.stdout: pcre2-utf16-10.32-2.el8.x86_64 2023-07-14T19:21:14.506 INFO:teuthology.orchestra.run.smithi139.stdout: pcre2-utf32-10.32-2.el8.x86_64 2023-07-14T19:21:14.506 INFO:teuthology.orchestra.run.smithi139.stdout: postgresql-10.21-2.module+el8.6.0+15342+53518fac.x86_64 2023-07-14T19:21:14.506 INFO:teuthology.orchestra.run.smithi139.stdout: postgresql-contrib-10.21-2.module+el8.6.0+15342+53518fac.x86_64 2023-07-14T19:21:14.506 INFO:teuthology.orchestra.run.smithi139.stdout: postgresql-server-10.21-2.module+el8.6.0+15342+53518fac.x86_64 2023-07-14T19:21:14.507 INFO:teuthology.orchestra.run.smithi139.stdout: python3-asyncssh-2.7.0-2.el8.noarch 2023-07-14T19:21:14.507 INFO:teuthology.orchestra.run.smithi139.stdout: python3-babel-2.5.1-7.el8.noarch 2023-07-14T19:21:14.507 INFO:teuthology.orchestra.run.smithi139.stdout: python3-bcrypt-3.1.6-2.el8.1.x86_64 2023-07-14T19:21:14.507 INFO:teuthology.orchestra.run.smithi139.stdout: python3-beautifulsoup4-4.6.3-2.el8.1.noarch 2023-07-14T19:21:14.507 INFO:teuthology.orchestra.run.smithi139.stdout: python3-cachetools-3.1.1-4.el8.noarch 2023-07-14T19:21:14.507 INFO:teuthology.orchestra.run.smithi139.stdout: python3-ceph-argparse-2:18.0.0-4940.g81c9387d.el8.x86_64 2023-07-14T19:21:14.507 INFO:teuthology.orchestra.run.smithi139.stdout: python3-ceph-common-2:18.0.0-4940.g81c9387d.el8.x86_64 2023-07-14T19:21:14.508 INFO:teuthology.orchestra.run.smithi139.stdout: python3-cephfs-2:18.0.0-4940.g81c9387d.el8.x86_64 2023-07-14T19:21:14.508 INFO:teuthology.orchestra.run.smithi139.stdout: python3-certifi-2018.10.15-7.el8.noarch 2023-07-14T19:21:14.508 INFO:teuthology.orchestra.run.smithi139.stdout: python3-cffi-1.11.5-5.el8.x86_64 2023-07-14T19:21:14.508 INFO:teuthology.orchestra.run.smithi139.stdout: python3-cheroot-8.5.2-1.el8.noarch 2023-07-14T19:21:14.508 INFO:teuthology.orchestra.run.smithi139.stdout: python3-cherrypy-18.4.0-1.el8.noarch 2023-07-14T19:21:14.508 INFO:teuthology.orchestra.run.smithi139.stdout: python3-cryptography-3.2.1-5.el8.x86_64 2023-07-14T19:21:14.508 INFO:teuthology.orchestra.run.smithi139.stdout: python3-google-auth-1:1.1.1-10.el8.noarch 2023-07-14T19:21:14.509 INFO:teuthology.orchestra.run.smithi139.stdout: python3-jaraco-6.2-6.el8.noarch 2023-07-14T19:21:14.509 INFO:teuthology.orchestra.run.smithi139.stdout: python3-jaraco-functools-2.0-4.el8.noarch 2023-07-14T19:21:14.509 INFO:teuthology.orchestra.run.smithi139.stdout: python3-jinja2-2.10.1-3.el8.noarch 2023-07-14T19:21:14.509 INFO:teuthology.orchestra.run.smithi139.stdout: python3-jsonpatch-1.21-2.el8.noarch 2023-07-14T19:21:14.509 INFO:teuthology.orchestra.run.smithi139.stdout: python3-jsonpointer-1.10-11.el8.noarch 2023-07-14T19:21:14.509 INFO:teuthology.orchestra.run.smithi139.stdout: python3-jwt-1.6.1-2.el8.noarch 2023-07-14T19:21:14.509 INFO:teuthology.orchestra.run.smithi139.stdout: python3-kubernetes-1:11.0.0-6.el8.noarch 2023-07-14T19:21:14.509 INFO:teuthology.orchestra.run.smithi139.stdout: python3-logutils-0.3.5-11.el8.noarch 2023-07-14T19:21:14.510 INFO:teuthology.orchestra.run.smithi139.stdout: python3-mako-1.0.6-13.el8.noarch 2023-07-14T19:21:14.510 INFO:teuthology.orchestra.run.smithi139.stdout: python3-markupsafe-0.23-19.el8.x86_64 2023-07-14T19:21:14.510 INFO:teuthology.orchestra.run.smithi139.stdout: python3-more-itertools-7.2.0-3.el8.noarch 2023-07-14T19:21:14.510 INFO:teuthology.orchestra.run.smithi139.stdout: python3-natsort-7.1.1-2.el8.noarch 2023-07-14T19:21:14.510 INFO:teuthology.orchestra.run.smithi139.stdout: python3-numpy-1:1.14.3-10.el8.x86_64 2023-07-14T19:21:14.510 INFO:teuthology.orchestra.run.smithi139.stdout: python3-numpy-f2py-1:1.14.3-10.el8.x86_64 2023-07-14T19:21:14.510 INFO:teuthology.orchestra.run.smithi139.stdout: python3-oauthlib-2.1.0-1.el8.noarch 2023-07-14T19:21:14.510 INFO:teuthology.orchestra.run.smithi139.stdout: python3-pecan-1.3.2-9.el8.noarch 2023-07-14T19:21:14.511 INFO:teuthology.orchestra.run.smithi139.stdout: python3-portend-2.6-1.el8.noarch 2023-07-14T19:21:14.511 INFO:teuthology.orchestra.run.smithi139.stdout: python3-prettytable-0.7.2-14.el8.noarch 2023-07-14T19:21:14.511 INFO:teuthology.orchestra.run.smithi139.stdout: python3-pyOpenSSL-19.0.0-1.el8.noarch 2023-07-14T19:21:14.511 INFO:teuthology.orchestra.run.smithi139.stdout: python3-pyasn1-0.3.7-6.el8.noarch 2023-07-14T19:21:14.511 INFO:teuthology.orchestra.run.smithi139.stdout: python3-pyasn1-modules-0.3.7-6.el8.noarch 2023-07-14T19:21:14.511 INFO:teuthology.orchestra.run.smithi139.stdout: python3-pycparser-2.14-14.el8.noarch 2023-07-14T19:21:14.511 INFO:teuthology.orchestra.run.smithi139.stdout: python3-pytz-2017.2-9.el8.noarch 2023-07-14T19:21:14.512 INFO:teuthology.orchestra.run.smithi139.stdout: python3-rados-2:18.0.0-4940.g81c9387d.el8.x86_64 2023-07-14T19:21:14.512 INFO:teuthology.orchestra.run.smithi139.stdout: python3-rbd-2:18.0.0-4940.g81c9387d.el8.x86_64 2023-07-14T19:21:14.512 INFO:teuthology.orchestra.run.smithi139.stdout: python3-repoze-lru-0.7-6.el8.noarch 2023-07-14T19:21:14.512 INFO:teuthology.orchestra.run.smithi139.stdout: python3-requests-oauthlib-1.0.0-1.el8.noarch 2023-07-14T19:21:14.512 INFO:teuthology.orchestra.run.smithi139.stdout: python3-rgw-2:18.0.0-4940.g81c9387d.el8.x86_64 2023-07-14T19:21:14.512 INFO:teuthology.orchestra.run.smithi139.stdout: python3-routes-2.4.1-12.el8.noarch 2023-07-14T19:21:14.512 INFO:teuthology.orchestra.run.smithi139.stdout: python3-rsa-4.8-1.el8.noarch 2023-07-14T19:21:14.513 INFO:teuthology.orchestra.run.smithi139.stdout: python3-scipy-1.0.0-21.module+el8.5.0+10916+41bd434d.x86_64 2023-07-14T19:21:14.513 INFO:teuthology.orchestra.run.smithi139.stdout: python3-simplegeneric-0.8.1-17.el8.noarch 2023-07-14T19:21:14.513 INFO:teuthology.orchestra.run.smithi139.stdout: python3-singledispatch-3.4.0.3-18.el8.noarch 2023-07-14T19:21:14.513 INFO:teuthology.orchestra.run.smithi139.stdout: python3-tempora-1.14.1-5.el8.noarch 2023-07-14T19:21:14.513 INFO:teuthology.orchestra.run.smithi139.stdout: python3-trustme-0.6.0-4.el8.noarch 2023-07-14T19:21:14.513 INFO:teuthology.orchestra.run.smithi139.stdout: python3-waitress-1.4.3-1.el8.noarch 2023-07-14T19:21:14.513 INFO:teuthology.orchestra.run.smithi139.stdout: python3-webob-1.8.5-1.el8.1.noarch 2023-07-14T19:21:14.513 INFO:teuthology.orchestra.run.smithi139.stdout: python3-websocket-client-0.56.0-5.el8.noarch 2023-07-14T19:21:14.514 INFO:teuthology.orchestra.run.smithi139.stdout: python3-webtest-2.0.33-1.el8.noarch 2023-07-14T19:21:14.514 INFO:teuthology.orchestra.run.smithi139.stdout: python3-werkzeug-0.12.2-4.el8.noarch 2023-07-14T19:21:14.514 INFO:teuthology.orchestra.run.smithi139.stdout: python3-zc-lockfile-2.0-2.el8.noarch 2023-07-14T19:21:14.514 INFO:teuthology.orchestra.run.smithi139.stdout: rbd-fuse-2:18.0.0-4940.g81c9387d.el8.x86_64 2023-07-14T19:21:14.514 INFO:teuthology.orchestra.run.smithi139.stdout: rbd-mirror-2:18.0.0-4940.g81c9387d.el8.x86_64 2023-07-14T19:21:14.514 INFO:teuthology.orchestra.run.smithi139.stdout: rbd-nbd-2:18.0.0-4940.g81c9387d.el8.x86_64 2023-07-14T19:21:14.514 INFO:teuthology.orchestra.run.smithi139.stdout: socat-1.7.4.1-1.el8.x86_64 2023-07-14T19:21:14.514 INFO:teuthology.orchestra.run.smithi139.stdout: thrift-0.13.0-2.el8.x86_64 2023-07-14T19:21:14.515 INFO:teuthology.orchestra.run.smithi139.stdout: uuid-1.6.2-43.el8.x86_64 2023-07-14T19:21:14.515 INFO:teuthology.orchestra.run.smithi139.stdout: xfsprogs-devel-5.0.0-10.el8.x86_64 2023-07-14T19:21:14.515 INFO:teuthology.orchestra.run.smithi139.stdout: xmlstarlet-1.6.1-20.el8.x86_64 2023-07-14T19:21:14.516 INFO:teuthology.orchestra.run.smithi139.stdout: zlib-devel-1.2.11-18.el8_5.x86_64 2023-07-14T19:21:14.516 INFO:teuthology.orchestra.run.smithi139.stdout: 2023-07-14T19:21:14.516 INFO:teuthology.orchestra.run.smithi139.stdout:Complete! 2023-07-14T19:21:14.579 DEBUG:teuthology.parallel:result is None 2023-07-14T19:21:14.689 DEBUG:teuthology.parallel:result is None 2023-07-14T19:21:14.691 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=81c9387d2e7576da065747ec9c16d1c7d69016ec 2023-07-14T19:21:15.072 DEBUG:teuthology.orchestra.run.smithi098:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2023-07-14T19:21:15.097 INFO:teuthology.orchestra.run.smithi098.stdout:18.0.0-4940.g81c9387d.el8 2023-07-14T19:21:15.097 INFO:teuthology.packaging:The installed version of ceph is 18.0.0-4940.g81c9387d.el8 2023-07-14T19:21:15.099 INFO:teuthology.task.install:The correct ceph version 18.0.0-4940.g81c9387d is installed. 2023-07-14T19:21:15.099 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=81c9387d2e7576da065747ec9c16d1c7d69016ec 2023-07-14T19:21:15.413 DEBUG:teuthology.orchestra.run.smithi101:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2023-07-14T19:21:15.442 INFO:teuthology.orchestra.run.smithi101.stdout:18.0.0-4940.g81c9387d.el8 2023-07-14T19:21:15.442 INFO:teuthology.packaging:The installed version of ceph is 18.0.0-4940.g81c9387d.el8 2023-07-14T19:21:15.443 INFO:teuthology.task.install:The correct ceph version 18.0.0-4940.g81c9387d is installed. 2023-07-14T19:21:15.443 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=81c9387d2e7576da065747ec9c16d1c7d69016ec 2023-07-14T19:21:15.769 DEBUG:teuthology.orchestra.run.smithi139:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2023-07-14T19:21:15.796 INFO:teuthology.orchestra.run.smithi139.stdout:18.0.0-4940.g81c9387d.el8 2023-07-14T19:21:15.797 INFO:teuthology.packaging:The installed version of ceph is 18.0.0-4940.g81c9387d.el8 2023-07-14T19:21:15.798 INFO:teuthology.task.install:The correct ceph version 18.0.0-4940.g81c9387d is installed. 2023-07-14T19:21:15.798 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2023-07-14T19:21:15.799 DEBUG:teuthology.orchestra.run.smithi098:> set -ex 2023-07-14T19:21:15.799 DEBUG:teuthology.orchestra.run.smithi098:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2023-07-14T19:21:15.846 DEBUG:teuthology.orchestra.run.smithi101:> set -ex 2023-07-14T19:21:15.847 DEBUG:teuthology.orchestra.run.smithi101:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2023-07-14T19:21:15.893 DEBUG:teuthology.orchestra.run.smithi139:> set -ex 2023-07-14T19:21:15.894 DEBUG:teuthology.orchestra.run.smithi139:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2023-07-14T19:21:15.940 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2023-07-14T19:21:15.941 DEBUG:teuthology.orchestra.run.smithi098:> set -ex 2023-07-14T19:21:15.941 DEBUG:teuthology.orchestra.run.smithi098:> sudo dd of=/usr/bin/daemon-helper 2023-07-14T19:21:15.972 DEBUG:teuthology.orchestra.run.smithi098:> sudo chmod a=rx -- /usr/bin/daemon-helper 2023-07-14T19:21:16.042 DEBUG:teuthology.orchestra.run.smithi101:> set -ex 2023-07-14T19:21:16.042 DEBUG:teuthology.orchestra.run.smithi101:> sudo dd of=/usr/bin/daemon-helper 2023-07-14T19:21:16.074 DEBUG:teuthology.orchestra.run.smithi101:> sudo chmod a=rx -- /usr/bin/daemon-helper 2023-07-14T19:21:16.145 DEBUG:teuthology.orchestra.run.smithi139:> set -ex 2023-07-14T19:21:16.146 DEBUG:teuthology.orchestra.run.smithi139:> sudo dd of=/usr/bin/daemon-helper 2023-07-14T19:21:16.177 DEBUG:teuthology.orchestra.run.smithi139:> sudo chmod a=rx -- /usr/bin/daemon-helper 2023-07-14T19:21:16.247 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2023-07-14T19:21:16.248 DEBUG:teuthology.orchestra.run.smithi098:> set -ex 2023-07-14T19:21:16.249 DEBUG:teuthology.orchestra.run.smithi098:> sudo dd of=/usr/bin/adjust-ulimits 2023-07-14T19:21:16.280 DEBUG:teuthology.orchestra.run.smithi098:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2023-07-14T19:21:16.351 DEBUG:teuthology.orchestra.run.smithi101:> set -ex 2023-07-14T19:21:16.351 DEBUG:teuthology.orchestra.run.smithi101:> sudo dd of=/usr/bin/adjust-ulimits 2023-07-14T19:21:16.382 DEBUG:teuthology.orchestra.run.smithi101:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2023-07-14T19:21:16.452 DEBUG:teuthology.orchestra.run.smithi139:> set -ex 2023-07-14T19:21:16.453 DEBUG:teuthology.orchestra.run.smithi139:> sudo dd of=/usr/bin/adjust-ulimits 2023-07-14T19:21:16.485 DEBUG:teuthology.orchestra.run.smithi139:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2023-07-14T19:21:16.554 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2023-07-14T19:21:16.555 DEBUG:teuthology.orchestra.run.smithi098:> set -ex 2023-07-14T19:21:16.555 DEBUG:teuthology.orchestra.run.smithi098:> sudo dd of=/usr/bin/stdin-killer 2023-07-14T19:21:16.586 DEBUG:teuthology.orchestra.run.smithi098:> sudo chmod a=rx -- /usr/bin/stdin-killer 2023-07-14T19:21:16.655 DEBUG:teuthology.orchestra.run.smithi101:> set -ex 2023-07-14T19:21:16.655 DEBUG:teuthology.orchestra.run.smithi101:> sudo dd of=/usr/bin/stdin-killer 2023-07-14T19:21:16.686 DEBUG:teuthology.orchestra.run.smithi101:> sudo chmod a=rx -- /usr/bin/stdin-killer 2023-07-14T19:21:16.756 DEBUG:teuthology.orchestra.run.smithi139:> set -ex 2023-07-14T19:21:16.756 DEBUG:teuthology.orchestra.run.smithi139:> sudo dd of=/usr/bin/stdin-killer 2023-07-14T19:21:16.787 DEBUG:teuthology.orchestra.run.smithi139:> sudo chmod a=rx -- /usr/bin/stdin-killer 2023-07-14T19:21:16.858 INFO:teuthology.run_tasks:Running task cephadm... 2023-07-14T19:21:16.955 INFO:tasks.cephadm:Config: {'roleless': False, 'cephfs': {'max_mds': 3, 'session_timeout': 300, 'standby_replay': True}, 'conf': {'client': {'client mount timeout': 600, 'debug client': 20, 'debug ms': 1, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'global': {'ms die on skipped message': False}, 'mds': {'debug mds': 20, 'debug mds balancer': 20, 'debug ms': 1, 'mds bal fragment size max': 10000, 'mds bal merge size': 5, 'mds bal split bits': 3, 'mds bal split size': 100, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds op complaint time': 180, 'mds verify scatter': True, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mgr': {'cephsqlite lock renewal timeout': 900000, 'debug cephsqlite': 20, 'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon op complaint time': 120}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'bitmap', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'filestore flush min': 0, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd heartbeat grace': 60, 'osd objectstore': 'bluestore', 'osd op complaint time': 180, 'osd shutdown pgref assert': True, 'osd_max_omap_entries_per_request': 10}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'overall HEALTH_', '\\(FS_DEGRADED\\)', '\\(MDS_FAILED\\)', '\\(MDS_DEGRADED\\)', '\\(FS_WITH_FAILED_MDS\\)', '\\(MDS_DAMAGE\\)', '\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(FS_INLINE_DATA_DEPRECATED\\)', 'overall HEALTH_', '\\(OSD_DOWN\\)', '\\(OSD_', 'but it is still running', 'is not responding', 'slow metadata IO', 'SLOW_OPS', 'slow request', 'SLOW_OPS', 'slow request'], 'sha1': '81c9387d2e7576da065747ec9c16d1c7d69016ec', 'subvols': {'create': 2}} 2023-07-14T19:21:16.956 INFO:tasks.cephadm:Cluster image is quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81c9387d2e7576da065747ec9c16d1c7d69016ec 2023-07-14T19:21:16.967 INFO:tasks.cephadm:Cluster fsid is 9adea648-227b-11ee-9b33-001a4aab830c 2023-07-14T19:21:16.968 INFO:tasks.cephadm:Choosing monitor IPs and ports... 2023-07-14T19:21:16.969 INFO:tasks.cephadm:Monitor IPs: {'mon.a': '172.21.15.98', 'mon.b': '172.21.15.101', 'mon.c': '172.21.15.139'} 2023-07-14T19:21:16.970 INFO:tasks.cephadm:First mon is mon.a on smithi098 2023-07-14T19:21:16.971 INFO:tasks.cephadm:First mgr is x 2023-07-14T19:21:16.971 INFO:tasks.cephadm:Normalizing hostnames... 2023-07-14T19:21:16.972 DEBUG:teuthology.orchestra.run.smithi098:> sudo hostname $(hostname -s) 2023-07-14T19:21:17.005 DEBUG:teuthology.orchestra.run.smithi101:> sudo hostname $(hostname -s) 2023-07-14T19:21:17.037 DEBUG:teuthology.orchestra.run.smithi139:> sudo hostname $(hostname -s) 2023-07-14T19:21:17.068 INFO:tasks.cephadm:Downloading "compiled" cephadm from cachra 2023-07-14T19:21:17.069 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=81c9387d2e7576da065747ec9c16d1c7d69016ec 2023-07-14T19:21:17.470 INFO:tasks.cephadm:builder_project result: [{'status': 'ready', 'sha1': '81c9387d2e7576da065747ec9c16d1c7d69016ec', 'extra': {'build_url': 'https://jenkins.ceph.com/job/ceph-dev-new-build/ARCH=arm64,AVAILABLE_ARCH=arm64,AVAILABLE_DIST=centos8,DIST=centos8,MACHINE_SIZE=gigantic/71431/', 'root_build_cause': 'SCMTRIGGER', 'version': '18.0.0-4940-g81c9387d', 'node_name': '172.21.4.65+confusa03', 'job_name': 'ceph-dev-new-build/ARCH=arm64,AVAILABLE_ARCH=arm64,AVAILABLE_DIST=centos8,DIST=centos8,MACHINE_SIZE=gigantic', 'package_manager_version': '18.0.0-4940.g81c9387d'}, 'url': 'https://4.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20230714.134335/81c9387d2e7576da065747ec9c16d1c7d69016ec/centos/8/flavors/default/', 'distro_codename': None, 'modified': '2023-07-14 16:50:26.249583', 'distro_version': '8', 'project': 'ceph', 'flavor': 'default', 'ref': 'wip-pdonnell-testing-20230714.134335', 'chacra_url': 'https://4.chacra.ceph.com/repos/ceph/wip-pdonnell-testing-20230714.134335/81c9387d2e7576da065747ec9c16d1c7d69016ec/centos/8/flavors/default/', 'archs': ['source', 'x86_64', 'arm64'], 'distro': 'centos'}] 2023-07-14T19:21:17.845 INFO:tasks.util.chacra:got chacra host 4.chacra.ceph.com, ref wip-pdonnell-testing-20230714.134335, sha1 81c9387d2e7576da065747ec9c16d1c7d69016ec from https://shaman.ceph.com/api/search/?project=ceph&distros=centos%2F8%2Fx86_64&flavor=default&sha1=81c9387d2e7576da065747ec9c16d1c7d69016ec 2023-07-14T19:21:17.845 INFO:tasks.cephadm:Discovered cachra url: https://4.chacra.ceph.com/binaries/ceph/wip-pdonnell-testing-20230714.134335/81c9387d2e7576da065747ec9c16d1c7d69016ec/centos/8/x86_64/flavors/default/cephadm 2023-07-14T19:21:17.846 DEBUG:teuthology.orchestra.run.smithi098:> curl --silent -L https://4.chacra.ceph.com/binaries/ceph/wip-pdonnell-testing-20230714.134335/81c9387d2e7576da065747ec9c16d1c7d69016ec/centos/8/x86_64/flavors/default/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2023-07-14T19:21:19.671 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-rw-r--. 1 ubuntu ubuntu 670007 Jul 14 19:21 /home/ubuntu/cephtest/cephadm 2023-07-14T19:21:19.672 DEBUG:teuthology.orchestra.run.smithi101:> curl --silent -L https://4.chacra.ceph.com/binaries/ceph/wip-pdonnell-testing-20230714.134335/81c9387d2e7576da065747ec9c16d1c7d69016ec/centos/8/x86_64/flavors/default/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2023-07-14T19:21:21.685 INFO:teuthology.orchestra.run.smithi101.stdout:-rw-rw-r--. 1 ubuntu ubuntu 670007 Jul 14 19:21 /home/ubuntu/cephtest/cephadm 2023-07-14T19:21:21.685 DEBUG:teuthology.orchestra.run.smithi139:> curl --silent -L https://4.chacra.ceph.com/binaries/ceph/wip-pdonnell-testing-20230714.134335/81c9387d2e7576da065747ec9c16d1c7d69016ec/centos/8/x86_64/flavors/default/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2023-07-14T19:21:27.914 INFO:teuthology.orchestra.run.smithi139.stdout:-rw-rw-r--. 1 ubuntu ubuntu 670007 Jul 14 19:21 /home/ubuntu/cephtest/cephadm 2023-07-14T19:21:27.915 DEBUG:teuthology.orchestra.run.smithi098:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2023-07-14T19:21:27.937 DEBUG:teuthology.orchestra.run.smithi101:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2023-07-14T19:21:27.960 DEBUG:teuthology.orchestra.run.smithi139:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2023-07-14T19:21:28.005 INFO:tasks.cephadm:Pulling image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81c9387d2e7576da065747ec9c16d1c7d69016ec on all hosts... 2023-07-14T19:21:28.006 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81c9387d2e7576da065747ec9c16d1c7d69016ec pull 2023-07-14T19:21:28.009 DEBUG:teuthology.orchestra.run.smithi101:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81c9387d2e7576da065747ec9c16d1c7d69016ec pull 2023-07-14T19:21:28.012 DEBUG:teuthology.orchestra.run.smithi139:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81c9387d2e7576da065747ec9c16d1c7d69016ec pull 2023-07-14T19:21:30.194 INFO:teuthology.orchestra.run.smithi101.stderr:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81c9387d2e7576da065747ec9c16d1c7d69016ec... 2023-07-14T19:21:31.348 INFO:teuthology.orchestra.run.smithi098.stderr:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81c9387d2e7576da065747ec9c16d1c7d69016ec... 2023-07-14T19:21:31.648 INFO:teuthology.orchestra.run.smithi139.stderr:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81c9387d2e7576da065747ec9c16d1c7d69016ec... 2023-07-14T19:21:55.883 INFO:teuthology.orchestra.run.smithi101.stdout:{ 2023-07-14T19:21:55.883 INFO:teuthology.orchestra.run.smithi101.stdout: "ceph_version": "ceph version 18.0.0-4940-g81c9387d (81c9387d2e7576da065747ec9c16d1c7d69016ec) reef (dev)", 2023-07-14T19:21:55.883 INFO:teuthology.orchestra.run.smithi101.stdout: "image_id": "219b180907a819a02c0d1790ec69e152120a3974adf061aaf947f90f6ccf9705", 2023-07-14T19:21:55.883 INFO:teuthology.orchestra.run.smithi101.stdout: "repo_digests": [ 2023-07-14T19:21:55.884 INFO:teuthology.orchestra.run.smithi101.stdout: "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:1302190083259ae87fcd27babf423c1dd04ae7417edb252031c1d861adeb3110" 2023-07-14T19:21:55.884 INFO:teuthology.orchestra.run.smithi101.stdout: ] 2023-07-14T19:21:55.884 INFO:teuthology.orchestra.run.smithi101.stdout:} 2023-07-14T19:21:58.926 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2023-07-14T19:21:58.926 INFO:teuthology.orchestra.run.smithi098.stdout: "ceph_version": "ceph version 18.0.0-4940-g81c9387d (81c9387d2e7576da065747ec9c16d1c7d69016ec) reef (dev)", 2023-07-14T19:21:58.926 INFO:teuthology.orchestra.run.smithi098.stdout: "image_id": "219b180907a819a02c0d1790ec69e152120a3974adf061aaf947f90f6ccf9705", 2023-07-14T19:21:58.926 INFO:teuthology.orchestra.run.smithi098.stdout: "repo_digests": [ 2023-07-14T19:21:58.926 INFO:teuthology.orchestra.run.smithi098.stdout: "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:1302190083259ae87fcd27babf423c1dd04ae7417edb252031c1d861adeb3110" 2023-07-14T19:21:58.927 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2023-07-14T19:21:58.927 INFO:teuthology.orchestra.run.smithi098.stdout:} 2023-07-14T19:21:59.442 INFO:teuthology.orchestra.run.smithi139.stdout:{ 2023-07-14T19:21:59.442 INFO:teuthology.orchestra.run.smithi139.stdout: "ceph_version": "ceph version 18.0.0-4940-g81c9387d (81c9387d2e7576da065747ec9c16d1c7d69016ec) reef (dev)", 2023-07-14T19:21:59.443 INFO:teuthology.orchestra.run.smithi139.stdout: "image_id": "219b180907a819a02c0d1790ec69e152120a3974adf061aaf947f90f6ccf9705", 2023-07-14T19:21:59.443 INFO:teuthology.orchestra.run.smithi139.stdout: "repo_digests": [ 2023-07-14T19:21:59.443 INFO:teuthology.orchestra.run.smithi139.stdout: "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:1302190083259ae87fcd27babf423c1dd04ae7417edb252031c1d861adeb3110" 2023-07-14T19:21:59.443 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2023-07-14T19:21:59.443 INFO:teuthology.orchestra.run.smithi139.stdout:} 2023-07-14T19:21:59.461 DEBUG:teuthology.orchestra.run.smithi098:> sudo mkdir -p /etc/ceph 2023-07-14T19:21:59.496 DEBUG:teuthology.orchestra.run.smithi101:> sudo mkdir -p /etc/ceph 2023-07-14T19:21:59.532 DEBUG:teuthology.orchestra.run.smithi139:> sudo mkdir -p /etc/ceph 2023-07-14T19:21:59.563 DEBUG:teuthology.orchestra.run.smithi098:> sudo chmod 777 /etc/ceph 2023-07-14T19:21:59.591 DEBUG:teuthology.orchestra.run.smithi101:> sudo chmod 777 /etc/ceph 2023-07-14T19:21:59.622 DEBUG:teuthology.orchestra.run.smithi139:> sudo chmod 777 /etc/ceph 2023-07-14T19:21:59.649 INFO:tasks.cephadm:Writing seed config... 2023-07-14T19:21:59.650 INFO:tasks.cephadm: override: [client] client mount timeout = 600 2023-07-14T19:21:59.651 INFO:tasks.cephadm: override: [client] debug client = 20 2023-07-14T19:21:59.651 INFO:tasks.cephadm: override: [client] debug ms = 1 2023-07-14T19:21:59.651 INFO:tasks.cephadm: override: [client] rados mon op timeout = 900 2023-07-14T19:21:59.651 INFO:tasks.cephadm: override: [client] rados osd op timeout = 900 2023-07-14T19:21:59.651 INFO:tasks.cephadm: override: [global] ms die on skipped message = False 2023-07-14T19:21:59.652 INFO:tasks.cephadm: override: [mds] debug mds = 20 2023-07-14T19:21:59.652 INFO:tasks.cephadm: override: [mds] debug mds balancer = 20 2023-07-14T19:21:59.652 INFO:tasks.cephadm: override: [mds] debug ms = 1 2023-07-14T19:21:59.652 INFO:tasks.cephadm: override: [mds] mds bal fragment size max = 10000 2023-07-14T19:21:59.652 INFO:tasks.cephadm: override: [mds] mds bal merge size = 5 2023-07-14T19:21:59.653 INFO:tasks.cephadm: override: [mds] mds bal split bits = 3 2023-07-14T19:21:59.653 INFO:tasks.cephadm: override: [mds] mds bal split size = 100 2023-07-14T19:21:59.653 INFO:tasks.cephadm: override: [mds] mds debug frag = True 2023-07-14T19:21:59.653 INFO:tasks.cephadm: override: [mds] mds debug scatterstat = True 2023-07-14T19:21:59.653 INFO:tasks.cephadm: override: [mds] mds op complaint time = 180 2023-07-14T19:21:59.654 INFO:tasks.cephadm: override: [mds] mds verify scatter = True 2023-07-14T19:21:59.654 INFO:tasks.cephadm: override: [mds] osd op complaint time = 180 2023-07-14T19:21:59.654 INFO:tasks.cephadm: override: [mds] rados mon op timeout = 900 2023-07-14T19:21:59.654 INFO:tasks.cephadm: override: [mds] rados osd op timeout = 900 2023-07-14T19:21:59.654 INFO:tasks.cephadm: override: [mgr] cephsqlite lock renewal timeout = 900000 2023-07-14T19:21:59.655 INFO:tasks.cephadm: override: [mgr] debug cephsqlite = 20 2023-07-14T19:21:59.655 INFO:tasks.cephadm: override: [mgr] debug mgr = 20 2023-07-14T19:21:59.655 INFO:tasks.cephadm: override: [mgr] debug ms = 1 2023-07-14T19:21:59.655 INFO:tasks.cephadm: override: [mon] debug mon = 20 2023-07-14T19:21:59.655 INFO:tasks.cephadm: override: [mon] debug ms = 1 2023-07-14T19:21:59.655 INFO:tasks.cephadm: override: [mon] debug paxos = 20 2023-07-14T19:21:59.656 INFO:tasks.cephadm: override: [mon] mon op complaint time = 120 2023-07-14T19:21:59.656 INFO:tasks.cephadm: override: [osd] bdev async discard = True 2023-07-14T19:21:59.657 INFO:tasks.cephadm: override: [osd] bdev enable discard = True 2023-07-14T19:21:59.657 INFO:tasks.cephadm: override: [osd] bluestore allocator = bitmap 2023-07-14T19:21:59.658 INFO:tasks.cephadm: override: [osd] bluestore block size = 96636764160 2023-07-14T19:21:59.658 INFO:tasks.cephadm: override: [osd] bluestore fsck on mount = True 2023-07-14T19:21:59.658 INFO:tasks.cephadm: override: [osd] debug bluefs = 1/20 2023-07-14T19:21:59.658 INFO:tasks.cephadm: override: [osd] debug bluestore = 1/20 2023-07-14T19:21:59.658 INFO:tasks.cephadm: override: [osd] debug ms = 1 2023-07-14T19:21:59.659 INFO:tasks.cephadm: override: [osd] debug osd = 20 2023-07-14T19:21:59.659 INFO:tasks.cephadm: override: [osd] debug rocksdb = 4/10 2023-07-14T19:21:59.659 INFO:tasks.cephadm: override: [osd] filestore flush min = 0 2023-07-14T19:21:59.659 INFO:tasks.cephadm: override: [osd] mon osd backfillfull_ratio = 0.85 2023-07-14T19:21:59.659 INFO:tasks.cephadm: override: [osd] mon osd full ratio = 0.9 2023-07-14T19:21:59.660 INFO:tasks.cephadm: override: [osd] mon osd nearfull ratio = 0.8 2023-07-14T19:21:59.660 INFO:tasks.cephadm: override: [osd] osd failsafe full ratio = 0.95 2023-07-14T19:21:59.660 INFO:tasks.cephadm: override: [osd] osd heartbeat grace = 60 2023-07-14T19:21:59.660 INFO:tasks.cephadm: override: [osd] osd objectstore = bluestore 2023-07-14T19:21:59.660 INFO:tasks.cephadm: override: [osd] osd op complaint time = 180 2023-07-14T19:21:59.661 INFO:tasks.cephadm: override: [osd] osd shutdown pgref assert = True 2023-07-14T19:21:59.661 INFO:tasks.cephadm: override: [osd] osd_max_omap_entries_per_request = 10 2023-07-14T19:21:59.662 DEBUG:teuthology.orchestra.run.smithi098:> set -ex 2023-07-14T19:21:59.662 DEBUG:teuthology.orchestra.run.smithi098:> dd of=/home/ubuntu/cephtest/seed.ceph.conf 2023-07-14T19:21:59.680 DEBUG:tasks.cephadm:Final config: [global] # make logging friendly to teuthology log_to_file = true log_to_stderr = false log to journald = false mon cluster log file level = debug mon clock drift allowed = 1.000 # replicate across OSDs, not hosts osd crush chooseleaf type = 0 #osd pool default size = 2 osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd # enable some debugging auth debug = true ms die on old message = true ms die on bug = true debug asserts on shutdown = true # adjust warnings mon max pg per osd = 10000# >= luminous mon pg warn max object skew = 0 mon osd allow primary affinity = true mon osd allow pg remap = true mon warn on legacy crush tunables = false mon warn on crush straw calc version zero = false mon warn on no sortbitwise = false mon warn on osd down out interval zero = false mon warn on too few osds = false mon_warn_on_pool_pg_num_not_power_of_two = false # disable pg_autoscaler by default for new pools osd_pool_default_pg_autoscale_mode = off # tests delete pools mon allow pool delete = true fsid = 9adea648-227b-11ee-9b33-001a4aab830c ms die on skipped message = False [osd] osd scrub load threshold = 5.0 osd scrub max interval = 600 osd mclock profile = high_recovery_ops osd recover clone overlap = true osd recovery max chunk = 1048576 osd deep scrub update digest min age = 30 osd map max advance = 10 osd memory target autotune = true # debugging osd debug shutdown = true osd debug op order = true osd debug verify stray on activate = true osd debug pg log writeout = true osd debug verify cached snaps = true osd debug verify missing on start = true osd debug misdirected ops = true osd op queue = debug_random osd op queue cut off = debug_random osd shutdown pgref assert = True bdev debug aio = true osd sloppy crc = true bdev async discard = True bdev enable discard = True bluestore allocator = bitmap bluestore block size = 96636764160 bluestore fsck on mount = True debug bluefs = 1/20 debug bluestore = 1/20 debug ms = 1 debug osd = 20 debug rocksdb = 4/10 filestore flush min = 0 mon osd backfillfull_ratio = 0.85 mon osd full ratio = 0.9 mon osd nearfull ratio = 0.8 osd failsafe full ratio = 0.95 osd heartbeat grace = 60 osd objectstore = bluestore osd op complaint time = 180 osd_max_omap_entries_per_request = 10 [mgr] mon reweight min pgs per osd = 4 mon reweight min bytes per osd = 10 mgr/telemetry/nag = false cephsqlite lock renewal timeout = 900000 debug cephsqlite = 20 debug mgr = 20 debug ms = 1 [mon] mon data avail warn = 5 mon mgr mkfs grace = 240 mon reweight min pgs per osd = 4 mon osd reporter subtree level = osd mon osd prime pg temp = true mon reweight min bytes per osd = 10 # rotate auth tickets quickly to exercise renewal paths auth mon ticket ttl = 660# 11m auth service ticket ttl = 240# 4m # don't complain about global id reclaim mon_warn_on_insecure_global_id_reclaim = false mon_warn_on_insecure_global_id_reclaim_allowed = false debug mon = 20 debug ms = 1 debug paxos = 20 mon op complaint time = 120 [client.rgw] rgw cache enabled = true rgw enable ops log = true rgw enable usage log = true [client] client mount timeout = 600 debug client = 20 debug ms = 1 rados mon op timeout = 900 rados osd op timeout = 900 [mds] debug mds = 20 debug mds balancer = 20 debug ms = 1 mds bal fragment size max = 10000 mds bal merge size = 5 mds bal split bits = 3 mds bal split size = 100 mds debug frag = True mds debug scatterstat = True mds op complaint time = 180 mds verify scatter = True osd op complaint time = 180 rados mon op timeout = 900 rados osd op timeout = 900 2023-07-14T19:21:59.680 DEBUG:teuthology.orchestra.run.smithi098:mon.a> sudo journalctl -f -n 0 -u ceph-9adea648-227b-11ee-9b33-001a4aab830c@mon.a.service 2023-07-14T19:21:59.722 DEBUG:teuthology.orchestra.run.smithi098:mgr.x> sudo journalctl -f -n 0 -u ceph-9adea648-227b-11ee-9b33-001a4aab830c@mgr.x.service 2023-07-14T19:21:59.748 INFO:journalctl@ceph.mon.a.smithi098.stdout:-- Logs begin at Fri 2023-07-14 18:57:10 UTC. -- 2023-07-14T19:21:59.749 INFO:tasks.cephadm:Bootstrapping... 2023-07-14T19:21:59.749 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81c9387d2e7576da065747ec9c16d1c7d69016ec -v bootstrap --fsid 9adea648-227b-11ee-9b33-001a4aab830c --config /home/ubuntu/cephtest/seed.ceph.conf --output-config /etc/ceph/ceph.conf --output-keyring /etc/ceph/ceph.client.admin.keyring --output-pub-ssh-key /home/ubuntu/cephtest/ceph.pub --mon-id a --mgr-id x --orphan-initial-daemons --skip-monitoring-stack --mon-ip 172.21.15.98 --skip-admin-label && sudo chmod +r /etc/ceph/ceph.client.admin.keyring 2023-07-14T19:21:59.774 INFO:journalctl@ceph.mgr.x.smithi098.stdout:-- Logs begin at Fri 2023-07-14 18:57:10 UTC. -- 2023-07-14T19:21:59.906 INFO:teuthology.orchestra.run.smithi098.stdout:-------------------------------------------------------------------------------- 2023-07-14T19:21:59.907 INFO:teuthology.orchestra.run.smithi098.stdout:cephadm ['--image', 'quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81c9387d2e7576da065747ec9c16d1c7d69016ec', '-v', 'bootstrap', '--fsid', '9adea648-227b-11ee-9b33-001a4aab830c', '--config', '/home/ubuntu/cephtest/seed.ceph.conf', '--output-config', '/etc/ceph/ceph.conf', '--output-keyring', '/etc/ceph/ceph.client.admin.keyring', '--output-pub-ssh-key', '/home/ubuntu/cephtest/ceph.pub', '--mon-id', 'a', '--mgr-id', 'x', '--orphan-initial-daemons', '--skip-monitoring-stack', '--mon-ip', '172.21.15.98', '--skip-admin-label'] 2023-07-14T19:22:00.054 INFO:teuthology.orchestra.run.smithi098.stdout:/bin/podman: stdout 4.1.1 2023-07-14T19:22:00.055 INFO:teuthology.orchestra.run.smithi098.stderr:Specifying an fsid for your cluster offers no advantages and may increase the likelihood of fsid conflicts. 2023-07-14T19:22:00.056 INFO:teuthology.orchestra.run.smithi098.stdout:Verifying podman|docker is present... 2023-07-14T19:22:00.201 INFO:teuthology.orchestra.run.smithi098.stdout:/bin/podman: stdout 4.1.1 2023-07-14T19:22:00.201 INFO:teuthology.orchestra.run.smithi098.stdout:Verifying lvm2 is present... 2023-07-14T19:22:00.201 INFO:teuthology.orchestra.run.smithi098.stdout:Verifying time synchronization is in place... 2023-07-14T19:22:00.208 INFO:teuthology.orchestra.run.smithi098.stdout:Non-zero exit code 1 from systemctl is-enabled chrony.service 2023-07-14T19:22:00.208 INFO:teuthology.orchestra.run.smithi098.stdout:systemctl: stderr Failed to get unit file state for chrony.service: No such file or directory 2023-07-14T19:22:00.213 INFO:teuthology.orchestra.run.smithi098.stdout:Non-zero exit code 3 from systemctl is-active chrony.service 2023-07-14T19:22:00.214 INFO:teuthology.orchestra.run.smithi098.stdout:systemctl: stdout inactive 2023-07-14T19:22:00.219 INFO:teuthology.orchestra.run.smithi098.stdout:systemctl: stdout enabled 2023-07-14T19:22:00.225 INFO:teuthology.orchestra.run.smithi098.stdout:systemctl: stdout active 2023-07-14T19:22:00.225 INFO:teuthology.orchestra.run.smithi098.stdout:Unit chronyd.service is enabled and running 2023-07-14T19:22:00.225 INFO:teuthology.orchestra.run.smithi098.stdout:Repeating the final host check... 2023-07-14T19:22:00.369 INFO:teuthology.orchestra.run.smithi098.stdout:/bin/podman: stdout 4.1.1 2023-07-14T19:22:00.369 INFO:teuthology.orchestra.run.smithi098.stdout:podman (/bin/podman) version 4.1.1 is present 2023-07-14T19:22:00.369 INFO:teuthology.orchestra.run.smithi098.stdout:systemctl is present 2023-07-14T19:22:00.369 INFO:teuthology.orchestra.run.smithi098.stdout:lvcreate is present 2023-07-14T19:22:00.375 INFO:teuthology.orchestra.run.smithi098.stdout:Non-zero exit code 1 from systemctl is-enabled chrony.service 2023-07-14T19:22:00.375 INFO:teuthology.orchestra.run.smithi098.stdout:systemctl: stderr Failed to get unit file state for chrony.service: No such file or directory 2023-07-14T19:22:00.381 INFO:teuthology.orchestra.run.smithi098.stdout:Non-zero exit code 3 from systemctl is-active chrony.service 2023-07-14T19:22:00.381 INFO:teuthology.orchestra.run.smithi098.stdout:systemctl: stdout inactive 2023-07-14T19:22:00.386 INFO:teuthology.orchestra.run.smithi098.stdout:systemctl: stdout enabled 2023-07-14T19:22:00.393 INFO:teuthology.orchestra.run.smithi098.stdout:systemctl: stdout active 2023-07-14T19:22:00.393 INFO:teuthology.orchestra.run.smithi098.stdout:Unit chronyd.service is enabled and running 2023-07-14T19:22:00.394 INFO:teuthology.orchestra.run.smithi098.stdout:Host looks OK 2023-07-14T19:22:00.394 INFO:teuthology.orchestra.run.smithi098.stdout:Cluster fsid: 9adea648-227b-11ee-9b33-001a4aab830c 2023-07-14T19:22:00.394 INFO:teuthology.orchestra.run.smithi098.stdout:Acquiring lock 139661395425768 on /run/cephadm/9adea648-227b-11ee-9b33-001a4aab830c.lock 2023-07-14T19:22:00.394 INFO:teuthology.orchestra.run.smithi098.stdout:Lock 139661395425768 acquired on /run/cephadm/9adea648-227b-11ee-9b33-001a4aab830c.lock 2023-07-14T19:22:00.395 INFO:teuthology.orchestra.run.smithi098.stdout:Verifying IP 172.21.15.98 port 3300 ... 2023-07-14T19:22:00.395 INFO:teuthology.orchestra.run.smithi098.stdout:Verifying IP 172.21.15.98 port 6789 ... 2023-07-14T19:22:00.396 INFO:teuthology.orchestra.run.smithi098.stdout:Base mon IP(s) is [172.21.15.98:3300, 172.21.15.98:6789], mon addrv is [v2:172.21.15.98:3300,v1:172.21.15.98:6789] 2023-07-14T19:22:00.397 INFO:teuthology.orchestra.run.smithi098.stdout:/sbin/ip: stdout default via 172.21.15.254 dev enp3s0f1 proto dhcp metric 100 2023-07-14T19:22:00.397 INFO:teuthology.orchestra.run.smithi098.stdout:/sbin/ip: stdout 172.21.0.0/20 dev enp3s0f1 proto kernel scope link src 172.21.15.98 metric 100 2023-07-14T19:22:00.399 INFO:teuthology.orchestra.run.smithi098.stdout:/sbin/ip: stdout ::1 dev lo proto kernel metric 256 pref medium 2023-07-14T19:22:00.399 INFO:teuthology.orchestra.run.smithi098.stdout:/sbin/ip: stdout fe80::/64 dev enp3s0f1 proto kernel metric 1024 pref medium 2023-07-14T19:22:00.400 INFO:teuthology.orchestra.run.smithi098.stdout:/sbin/ip: stdout default via fe80::327c:5e00:6487:71e0 dev enp3s0f1 proto ra metric 100 pref medium 2023-07-14T19:22:00.402 INFO:teuthology.orchestra.run.smithi098.stdout:/sbin/ip: stdout 1: lo: mtu 65536 state UNKNOWN qlen 1000 2023-07-14T19:22:00.402 INFO:teuthology.orchestra.run.smithi098.stdout:/sbin/ip: stdout inet6 ::1/128 scope host 2023-07-14T19:22:00.402 INFO:teuthology.orchestra.run.smithi098.stdout:/sbin/ip: stdout valid_lft forever preferred_lft forever 2023-07-14T19:22:00.402 INFO:teuthology.orchestra.run.smithi098.stdout:/sbin/ip: stdout 5: enp3s0f1: mtu 1500 state UP qlen 1000 2023-07-14T19:22:00.403 INFO:teuthology.orchestra.run.smithi098.stdout:/sbin/ip: stdout inet6 fe80::d4ed:112f:381a:8754/64 scope link noprefixroute 2023-07-14T19:22:00.403 INFO:teuthology.orchestra.run.smithi098.stdout:/sbin/ip: stdout valid_lft forever preferred_lft forever 2023-07-14T19:22:00.403 INFO:teuthology.orchestra.run.smithi098.stdout:Mon IP `172.21.15.98` is in CIDR network `172.21.0.0/20` 2023-07-14T19:22:00.403 INFO:teuthology.orchestra.run.smithi098.stdout:Mon IP `172.21.15.98` is in CIDR network `172.21.0.0/20` 2023-07-14T19:22:00.404 INFO:teuthology.orchestra.run.smithi098.stdout:Inferred mon public CIDR from local network configuration ['172.21.0.0/20', '172.21.0.0/20'] 2023-07-14T19:22:00.404 INFO:teuthology.orchestra.run.smithi098.stdout:Internal network (--cluster-network) has not been provided, OSD replication will default to the public_network 2023-07-14T19:22:00.404 INFO:teuthology.orchestra.run.smithi098.stdout:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81c9387d2e7576da065747ec9c16d1c7d69016ec... 2023-07-14T19:22:00.847 INFO:teuthology.orchestra.run.smithi098.stdout:/bin/podman: stdout 219b180907a819a02c0d1790ec69e152120a3974adf061aaf947f90f6ccf9705 2023-07-14T19:22:00.848 INFO:teuthology.orchestra.run.smithi098.stdout:/bin/podman: stderr Trying to pull quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81c9387d2e7576da065747ec9c16d1c7d69016ec... 2023-07-14T19:22:00.848 INFO:teuthology.orchestra.run.smithi098.stdout:/bin/podman: stderr Getting image source signatures 2023-07-14T19:22:00.848 INFO:teuthology.orchestra.run.smithi098.stdout:/bin/podman: stderr Copying blob sha256:0e454b422ba90e2ad13e636c40610401086a3576fe973b0460943839fe5f2508 2023-07-14T19:22:00.848 INFO:teuthology.orchestra.run.smithi098.stdout:/bin/podman: stderr Copying blob sha256:6022c611d474f2b306ddd1edfc8360123ece5266119096f25fde843d7967e636 2023-07-14T19:22:00.848 INFO:teuthology.orchestra.run.smithi098.stdout:/bin/podman: stderr Copying config sha256:219b180907a819a02c0d1790ec69e152120a3974adf061aaf947f90f6ccf9705 2023-07-14T19:22:00.848 INFO:teuthology.orchestra.run.smithi098.stdout:/bin/podman: stderr Writing manifest to image destination 2023-07-14T19:22:00.849 INFO:teuthology.orchestra.run.smithi098.stdout:/bin/podman: stderr Storing signatures 2023-07-14T19:22:01.594 INFO:teuthology.orchestra.run.smithi098.stdout:ceph: stdout ceph version 18.0.0-4940-g81c9387d (81c9387d2e7576da065747ec9c16d1c7d69016ec) reef (dev) 2023-07-14T19:22:01.594 INFO:teuthology.orchestra.run.smithi098.stdout:Ceph version: ceph version 18.0.0-4940-g81c9387d (81c9387d2e7576da065747ec9c16d1c7d69016ec) reef (dev) 2023-07-14T19:22:01.595 INFO:teuthology.orchestra.run.smithi098.stdout:Extracting ceph user uid/gid from container image... 2023-07-14T19:22:02.346 INFO:teuthology.orchestra.run.smithi098.stdout:stat: stdout 167 167 2023-07-14T19:22:02.346 INFO:teuthology.orchestra.run.smithi098.stdout:Creating initial keys... 2023-07-14T19:22:03.005 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph-authtool: stdout AQBaoLFk27nsKRAAuhv7krW2T2okMRNmUeE+0w== 2023-07-14T19:22:03.751 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph-authtool: stdout AQBboLFkDDw+FhAAxd0ZuaFr9hrs5NM7s6dmbw== 2023-07-14T19:22:04.501 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph-authtool: stdout AQBcoLFkzjclBhAA5pyyYiWKIEoYe+zvsZB0Rg== 2023-07-14T19:22:04.502 INFO:teuthology.orchestra.run.smithi098.stdout:Creating initial monmap... 2023-07-14T19:22:05.256 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: monmap file /tmp/monmap 2023-07-14T19:22:05.256 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/monmaptool: stdout setting min_mon_release = pacific 2023-07-14T19:22:05.256 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: set fsid to 9adea648-227b-11ee-9b33-001a4aab830c 2023-07-14T19:22:05.256 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2023-07-14T19:22:05.257 INFO:teuthology.orchestra.run.smithi098.stdout:monmaptool for a [v2:172.21.15.98:3300,v1:172.21.15.98:6789] on /usr/bin/monmaptool: monmap file /tmp/monmap 2023-07-14T19:22:05.257 INFO:teuthology.orchestra.run.smithi098.stdout:setting min_mon_release = pacific 2023-07-14T19:22:05.257 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/monmaptool: set fsid to 9adea648-227b-11ee-9b33-001a4aab830c 2023-07-14T19:22:05.257 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2023-07-14T19:22:05.257 INFO:teuthology.orchestra.run.smithi098.stdout: 2023-07-14T19:22:05.257 INFO:teuthology.orchestra.run.smithi098.stdout:Creating mon... 2023-07-14T19:22:06.006 INFO:teuthology.orchestra.run.smithi098.stdout:create mon.a on 2023-07-14T19:22:06.126 INFO:teuthology.orchestra.run.smithi098.stdout:systemctl: stderr Removed /etc/systemd/system/multi-user.target.wants/ceph.target. 2023-07-14T19:22:06.211 INFO:teuthology.orchestra.run.smithi098.stdout:systemctl: stderr Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /etc/systemd/system/ceph.target. 2023-07-14T19:22:06.354 INFO:teuthology.orchestra.run.smithi098.stdout:systemctl: stderr Created symlink /etc/systemd/system/multi-user.target.wants/ceph-9adea648-227b-11ee-9b33-001a4aab830c.target → /etc/systemd/system/ceph-9adea648-227b-11ee-9b33-001a4aab830c.target. 2023-07-14T19:22:06.355 INFO:teuthology.orchestra.run.smithi098.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph.target.wants/ceph-9adea648-227b-11ee-9b33-001a4aab830c.target → /etc/systemd/system/ceph-9adea648-227b-11ee-9b33-001a4aab830c.target. 2023-07-14T19:22:06.517 INFO:teuthology.orchestra.run.smithi098.stdout:Non-zero exit code 1 from systemctl reset-failed ceph-9adea648-227b-11ee-9b33-001a4aab830c@mon.a 2023-07-14T19:22:06.517 INFO:teuthology.orchestra.run.smithi098.stdout:systemctl: stderr Failed to reset failed state of unit ceph-9adea648-227b-11ee-9b33-001a4aab830c@mon.a.service: Unit ceph-9adea648-227b-11ee-9b33-001a4aab830c@mon.a.service not loaded. 2023-07-14T19:22:06.601 INFO:teuthology.orchestra.run.smithi098.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph-9adea648-227b-11ee-9b33-001a4aab830c.target.wants/ceph-9adea648-227b-11ee-9b33-001a4aab830c@mon.a.service → /etc/systemd/system/ceph-9adea648-227b-11ee-9b33-001a4aab830c@.service. 2023-07-14T19:22:06.781 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:22:06 smithi098 systemd[1]: Starting Ceph mon.a for 9adea648-227b-11ee-9b33-001a4aab830c... 2023-07-14T19:22:07.181 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:22:06 smithi098 podman[126951]: 2023-07-14T19:22:07.182 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:22:07 smithi098 bash[126951]: 41ff90f0acb9b1945ffa6712b8982e868cfa8c80f7047d1497685ffb8d7aee51 2023-07-14T19:22:07.202 INFO:teuthology.orchestra.run.smithi098.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2023-07-14T19:22:07.202 INFO:teuthology.orchestra.run.smithi098.stdout:systemctl: stdout disabled 2023-07-14T19:22:07.209 INFO:teuthology.orchestra.run.smithi098.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2023-07-14T19:22:07.209 INFO:teuthology.orchestra.run.smithi098.stdout:systemctl: stdout inactive 2023-07-14T19:22:07.209 INFO:teuthology.orchestra.run.smithi098.stdout:firewalld.service is not enabled 2023-07-14T19:22:07.210 INFO:teuthology.orchestra.run.smithi098.stdout:Not possible to enable service . firewalld.service is not available 2023-07-14T19:22:07.210 INFO:teuthology.orchestra.run.smithi098.stdout:Waiting for mon to start... 2023-07-14T19:22:07.210 INFO:teuthology.orchestra.run.smithi098.stdout:Waiting for mon... 2023-07-14T19:22:07.448 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:22:07 smithi098 ceph-mon[126990]: mkfs 9adea648-227b-11ee-9b33-001a4aab830c 2023-07-14T19:22:07.449 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:22:07 smithi098 ceph-mon[126990]: mon.a is new leader, mons a in quorum (ranks 0) 2023-07-14T19:22:07.449 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:22:07 smithi098 systemd[1]: Started Ceph mon.a for 9adea648-227b-11ee-9b33-001a4aab830c. 2023-07-14T19:22:08.060 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout cluster: 2023-07-14T19:22:08.060 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout id: 9adea648-227b-11ee-9b33-001a4aab830c 2023-07-14T19:22:08.060 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout health: HEALTH_OK 2023-07-14T19:22:08.060 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout 2023-07-14T19:22:08.060 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout services: 2023-07-14T19:22:08.060 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout mon: 1 daemons, quorum a (age 0.558974s) 2023-07-14T19:22:08.061 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout mgr: no daemons active 2023-07-14T19:22:08.061 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout osd: 0 osds: 0 up, 0 in 2023-07-14T19:22:08.061 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout 2023-07-14T19:22:08.061 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout data: 2023-07-14T19:22:08.061 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout pools: 0 pools, 0 pgs 2023-07-14T19:22:08.061 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout objects: 0 objects, 0 B 2023-07-14T19:22:08.062 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout usage: 0 B used, 0 B / 0 B avail 2023-07-14T19:22:08.062 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout pgs: 2023-07-14T19:22:08.062 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout 2023-07-14T19:22:08.062 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:07.661+0000 7fe9d3ff6700 1 Processor -- start 2023-07-14T19:22:08.063 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:07.661+0000 7fe9d3ff6700 1 -- start start 2023-07-14T19:22:08.063 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:07.661+0000 7fe9d3ff6700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe9cc106180 0x7fe9cc108560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:22:08.063 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:07.661+0000 7fe9d3ff6700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe9cc073ee0 con 0x7fe9cc106180 2023-07-14T19:22:08.063 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:07.661+0000 7fe9d1d92700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe9cc106180 0x7fe9cc108560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:22:08.063 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:07.661+0000 7fe9d1d92700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe9cc106180 0x7fe9cc108560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:44312/0 (socket says 172.21.15.98:44312) 2023-07-14T19:22:08.063 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:07.661+0000 7fe9d1d92700 1 -- 172.21.15.98:0/3825904433 learned_addr learned my addr 172.21.15.98:0/3825904433 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-07-14T19:22:08.063 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:07.662+0000 7fe9d1d92700 1 -- 172.21.15.98:0/3825904433 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe9cc074020 con 0x7fe9cc106180 2023-07-14T19:22:08.064 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:07.662+0000 7fe9d1d92700 1 --2- 172.21.15.98:0/3825904433 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe9cc106180 0x7fe9cc108560 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7fe9bc0098a0 tx=0x7fe9bc009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=26ec7caa4427b25d server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:22:08.064 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:07.662+0000 7fe9d0d90700 1 -- 172.21.15.98:0/3825904433 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fe9bc004160 con 0x7fe9cc106180 2023-07-14T19:22:08.064 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:07.662+0000 7fe9d0d90700 1 -- 172.21.15.98:0/3825904433 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe9bc0042c0 con 0x7fe9cc106180 2023-07-14T19:22:08.064 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:07.662+0000 7fe9d0d90700 1 -- 172.21.15.98:0/3825904433 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fe9bc0045a0 con 0x7fe9cc106180 2023-07-14T19:22:08.064 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:07.662+0000 7fe9d3ff6700 1 -- 172.21.15.98:0/3825904433 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe9cc106180 msgr2=0x7fe9cc108560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:22:08.064 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:07.662+0000 7fe9d3ff6700 1 --2- 172.21.15.98:0/3825904433 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe9cc106180 0x7fe9cc108560 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7fe9bc0098a0 tx=0x7fe9bc009bb0 comp rx=0 tx=0).stop 2023-07-14T19:22:08.064 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:07.663+0000 7fe9d3ff6700 1 -- 172.21.15.98:0/3825904433 shutdown_connections 2023-07-14T19:22:08.065 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:07.663+0000 7fe9d3ff6700 1 --2- 172.21.15.98:0/3825904433 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe9cc106180 0x7fe9cc108560 unknown :-1 s=CLOSED pgs=1 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:22:08.065 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:07.663+0000 7fe9d3ff6700 1 -- 172.21.15.98:0/3825904433 >> 172.21.15.98:0/3825904433 conn(0x7fe9cc0ffbc0 msgr2=0x7fe9cc101ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:22:08.065 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:07.663+0000 7fe9d3ff6700 1 -- 172.21.15.98:0/3825904433 shutdown_connections 2023-07-14T19:22:08.065 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:07.663+0000 7fe9d3ff6700 1 -- 172.21.15.98:0/3825904433 wait complete. 2023-07-14T19:22:08.066 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:07.663+0000 7fe9d3ff6700 1 Processor -- start 2023-07-14T19:22:08.066 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:07.663+0000 7fe9d3ff6700 1 -- start start 2023-07-14T19:22:08.066 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:07.664+0000 7fe9d3ff6700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe9cc106180 0x7fe9cc198c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:22:08.066 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:07.664+0000 7fe9d3ff6700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe9cc199150 con 0x7fe9cc106180 2023-07-14T19:22:08.066 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:07.664+0000 7fe9d1d92700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe9cc106180 0x7fe9cc198c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:22:08.066 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:07.664+0000 7fe9d1d92700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe9cc106180 0x7fe9cc198c10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:44314/0 (socket says 172.21.15.98:44314) 2023-07-14T19:22:08.066 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:07.664+0000 7fe9d1d92700 1 -- 172.21.15.98:0/3114476385 learned_addr learned my addr 172.21.15.98:0/3114476385 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-07-14T19:22:08.067 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:07.664+0000 7fe9d1d92700 1 -- 172.21.15.98:0/3114476385 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe9bc009550 con 0x7fe9cc106180 2023-07-14T19:22:08.067 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:07.664+0000 7fe9d1d92700 1 --2- 172.21.15.98:0/3114476385 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe9cc106180 0x7fe9cc198c10 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7fe9bc004000 tx=0x7fe9bc004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:22:08.067 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:07.664+0000 7fe9c2ffd700 1 -- 172.21.15.98:0/3114476385 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fe9bc011630 con 0x7fe9cc106180 2023-07-14T19:22:08.067 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:07.664+0000 7fe9c2ffd700 1 -- 172.21.15.98:0/3114476385 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe9bc011790 con 0x7fe9cc106180 2023-07-14T19:22:08.067 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:07.664+0000 7fe9d3ff6700 1 -- 172.21.15.98:0/3114476385 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe9cc199350 con 0x7fe9cc106180 2023-07-14T19:22:08.067 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:07.664+0000 7fe9c2ffd700 1 -- 172.21.15.98:0/3114476385 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fe9bc011a30 con 0x7fe9cc106180 2023-07-14T19:22:08.067 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:07.664+0000 7fe9d3ff6700 1 -- 172.21.15.98:0/3114476385 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe9cc199770 con 0x7fe9cc106180 2023-07-14T19:22:08.067 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:07.665+0000 7fe9c2ffd700 1 -- 172.21.15.98:0/3114476385 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7fe9bc022070 con 0x7fe9cc106180 2023-07-14T19:22:08.068 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:07.665+0000 7fe9c2ffd700 1 -- 172.21.15.98:0/3114476385 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7fe9bc01c450 con 0x7fe9cc106180 2023-07-14T19:22:08.068 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:07.665+0000 7fe9d3ff6700 1 -- 172.21.15.98:0/3114476385 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe9cc192950 con 0x7fe9cc106180 2023-07-14T19:22:08.068 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:07.668+0000 7fe9c2ffd700 1 -- 172.21.15.98:0/3114476385 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72446 (secure 0 0 0) 0x7fe9bc01b450 con 0x7fe9cc106180 2023-07-14T19:22:08.068 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:07.736+0000 7fe9d3ff6700 1 -- 172.21.15.98:0/3114476385 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "status"} v 0) v1 -- 0x7fe9cc061250 con 0x7fe9cc106180 2023-07-14T19:22:08.068 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:07.736+0000 7fe9c2ffd700 1 -- 172.21.15.98:0/3114476385 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "status"}]=0 v0) v1 ==== 54+0+317 (secure 0 0 0) 0x7fe9bc01b5f0 con 0x7fe9cc106180 2023-07-14T19:22:08.068 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:07.737+0000 7fe9d3ff6700 1 -- 172.21.15.98:0/3114476385 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe9cc106180 msgr2=0x7fe9cc198c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:22:08.068 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:07.737+0000 7fe9d3ff6700 1 --2- 172.21.15.98:0/3114476385 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe9cc106180 0x7fe9cc198c10 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7fe9bc004000 tx=0x7fe9bc004750 comp rx=0 tx=0).stop 2023-07-14T19:22:08.069 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:07.737+0000 7fe9d3ff6700 1 -- 172.21.15.98:0/3114476385 shutdown_connections 2023-07-14T19:22:08.069 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:07.737+0000 7fe9d3ff6700 1 --2- 172.21.15.98:0/3114476385 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe9cc106180 0x7fe9cc198c10 unknown :-1 s=CLOSED pgs=2 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:22:08.069 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:07.737+0000 7fe9d3ff6700 1 -- 172.21.15.98:0/3114476385 >> 172.21.15.98:0/3114476385 conn(0x7fe9cc0ffbc0 msgr2=0x7fe9cc101590 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:22:08.069 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:07.737+0000 7fe9d3ff6700 1 -- 172.21.15.98:0/3114476385 shutdown_connections 2023-07-14T19:22:08.069 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:07.737+0000 7fe9d3ff6700 1 -- 172.21.15.98:0/3114476385 wait complete. 2023-07-14T19:22:08.069 INFO:teuthology.orchestra.run.smithi098.stdout:mon is available 2023-07-14T19:22:08.069 INFO:teuthology.orchestra.run.smithi098.stdout:Assimilating anything we can from ceph.conf... 2023-07-14T19:22:08.346 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:22:08 smithi098 ceph-mon[126990]: mon.a is new leader, mons a in quorum (ranks 0) 2023-07-14T19:22:08.347 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:22:08 smithi098 ceph-mon[126990]: monmap e1: 1 mons at {a=[v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0]} removed_ranks: {} 2023-07-14T19:22:08.347 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:22:08 smithi098 ceph-mon[126990]: fsmap 2023-07-14T19:22:08.347 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:22:08 smithi098 ceph-mon[126990]: osdmap e1: 0 total, 0 up, 0 in 2023-07-14T19:22:08.347 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:22:08 smithi098 ceph-mon[126990]: mgrmap e1: no daemons active 2023-07-14T19:22:08.347 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:22:08 smithi098 ceph-mon[126990]: from='client.? 172.21.15.98:0/3114476385' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch 2023-07-14T19:22:08.999 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout 2023-07-14T19:22:09.001 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout [global] 2023-07-14T19:22:09.001 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout fsid = 9adea648-227b-11ee-9b33-001a4aab830c 2023-07-14T19:22:09.001 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout mon_host = [v2:172.21.15.98:3300,v1:172.21.15.98:6789] 2023-07-14T19:22:09.001 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout mon_osd_allow_pg_remap = true 2023-07-14T19:22:09.002 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout mon_osd_allow_primary_affinity = true 2023-07-14T19:22:09.002 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout mon_warn_on_no_sortbitwise = false 2023-07-14T19:22:09.002 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout osd_crush_chooseleaf_type = 0 2023-07-14T19:22:09.002 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout 2023-07-14T19:22:09.002 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout [mgr] 2023-07-14T19:22:09.002 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout mgr/telemetry/nag = false 2023-07-14T19:22:09.002 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout 2023-07-14T19:22:09.007 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout [osd] 2023-07-14T19:22:09.007 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout filestore_flush_min = 0 2023-07-14T19:22:09.008 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout mon_osd_backfillfull_ratio = 0.85 2023-07-14T19:22:09.008 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout mon_osd_full_ratio = 0.9 2023-07-14T19:22:09.008 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout mon_osd_nearfull_ratio = 0.8 2023-07-14T19:22:09.009 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout osd_map_max_advance = 10 2023-07-14T19:22:09.009 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout osd_sloppy_crc = true 2023-07-14T19:22:09.009 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:08.551+0000 7ff7d13a6700 1 Processor -- start 2023-07-14T19:22:09.010 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:08.551+0000 7ff7d13a6700 1 -- start start 2023-07-14T19:22:09.010 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:08.551+0000 7ff7d13a6700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff7cc1061c0 0x7ff7cc1085a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:22:09.010 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:08.551+0000 7ff7d13a6700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff7cc073ea0 con 0x7ff7cc1061c0 2023-07-14T19:22:09.010 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:08.552+0000 7ff7caffd700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff7cc1061c0 0x7ff7cc1085a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:22:09.011 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:08.552+0000 7ff7caffd700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff7cc1061c0 0x7ff7cc1085a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:44316/0 (socket says 172.21.15.98:44316) 2023-07-14T19:22:09.011 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:08.552+0000 7ff7caffd700 1 -- 172.21.15.98:0/4093440282 learned_addr learned my addr 172.21.15.98:0/4093440282 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-07-14T19:22:09.011 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:08.552+0000 7ff7caffd700 1 -- 172.21.15.98:0/4093440282 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff7cc073fe0 con 0x7ff7cc1061c0 2023-07-14T19:22:09.012 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:08.552+0000 7ff7caffd700 1 --2- 172.21.15.98:0/4093440282 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff7cc1061c0 0x7ff7cc1085a0 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 crypto rx=0x7ff7c00098a0 tx=0x7ff7c0009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=203289492adcae6d server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:22:09.012 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:08.552+0000 7ff7c9ffb700 1 -- 172.21.15.98:0/4093440282 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7ff7c0004160 con 0x7ff7cc1061c0 2023-07-14T19:22:09.012 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:08.552+0000 7ff7c9ffb700 1 -- 172.21.15.98:0/4093440282 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff7c00042c0 con 0x7ff7cc1061c0 2023-07-14T19:22:09.013 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:08.552+0000 7ff7c9ffb700 1 -- 172.21.15.98:0/4093440282 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7ff7c00045a0 con 0x7ff7cc1061c0 2023-07-14T19:22:09.013 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:08.553+0000 7ff7d13a6700 1 -- 172.21.15.98:0/4093440282 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff7cc1061c0 msgr2=0x7ff7cc1085a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:22:09.013 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:08.553+0000 7ff7d13a6700 1 --2- 172.21.15.98:0/4093440282 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff7cc1061c0 0x7ff7cc1085a0 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 crypto rx=0x7ff7c00098a0 tx=0x7ff7c0009bb0 comp rx=0 tx=0).stop 2023-07-14T19:22:09.013 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:08.553+0000 7ff7d13a6700 1 -- 172.21.15.98:0/4093440282 shutdown_connections 2023-07-14T19:22:09.013 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:08.553+0000 7ff7d13a6700 1 --2- 172.21.15.98:0/4093440282 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff7cc1061c0 0x7ff7cc1085a0 unknown :-1 s=CLOSED pgs=3 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:22:09.013 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:08.553+0000 7ff7d13a6700 1 -- 172.21.15.98:0/4093440282 >> 172.21.15.98:0/4093440282 conn(0x7ff7cc0ffbc0 msgr2=0x7ff7cc101ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:22:09.014 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:08.553+0000 7ff7d13a6700 1 -- 172.21.15.98:0/4093440282 shutdown_connections 2023-07-14T19:22:09.014 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:08.553+0000 7ff7d13a6700 1 -- 172.21.15.98:0/4093440282 wait complete. 2023-07-14T19:22:09.014 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:08.553+0000 7ff7d13a6700 1 Processor -- start 2023-07-14T19:22:09.014 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:08.554+0000 7ff7d13a6700 1 -- start start 2023-07-14T19:22:09.014 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:08.554+0000 7ff7d13a6700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff7cc1061c0 0x7ff7cc105b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:22:09.014 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:08.554+0000 7ff7d13a6700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff7cc106060 con 0x7ff7cc1061c0 2023-07-14T19:22:09.015 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:08.554+0000 7ff7caffd700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff7cc1061c0 0x7ff7cc105b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:22:09.015 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:08.554+0000 7ff7caffd700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff7cc1061c0 0x7ff7cc105b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:44318/0 (socket says 172.21.15.98:44318) 2023-07-14T19:22:09.015 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:08.554+0000 7ff7caffd700 1 -- 172.21.15.98:0/2361102029 learned_addr learned my addr 172.21.15.98:0/2361102029 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-07-14T19:22:09.015 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:08.554+0000 7ff7caffd700 1 -- 172.21.15.98:0/2361102029 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff7c0009550 con 0x7ff7cc1061c0 2023-07-14T19:22:09.015 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:08.554+0000 7ff7caffd700 1 --2- 172.21.15.98:0/2361102029 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff7cc1061c0 0x7ff7cc105b20 secure :-1 s=READY pgs=4 cs=0 l=1 rev1=1 crypto rx=0x7ff7c0004000 tx=0x7ff7c0004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:22:09.015 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:08.554+0000 7ff7b3fff700 1 -- 172.21.15.98:0/2361102029 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7ff7c0011630 con 0x7ff7cc1061c0 2023-07-14T19:22:09.015 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:08.555+0000 7ff7b3fff700 1 -- 172.21.15.98:0/2361102029 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff7c0011790 con 0x7ff7cc1061c0 2023-07-14T19:22:09.016 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:08.555+0000 7ff7d13a6700 1 -- 172.21.15.98:0/2361102029 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff7cc1041d0 con 0x7ff7cc1061c0 2023-07-14T19:22:09.016 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:08.555+0000 7ff7b3fff700 1 -- 172.21.15.98:0/2361102029 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7ff7c0011a30 con 0x7ff7cc1061c0 2023-07-14T19:22:09.016 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:08.555+0000 7ff7d13a6700 1 -- 172.21.15.98:0/2361102029 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff7cc104650 con 0x7ff7cc1061c0 2023-07-14T19:22:09.016 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:08.555+0000 7ff7b3fff700 1 -- 172.21.15.98:0/2361102029 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7ff7c0022070 con 0x7ff7cc1061c0 2023-07-14T19:22:09.016 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:08.555+0000 7ff7b3fff700 1 -- 172.21.15.98:0/2361102029 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7ff7c001c450 con 0x7ff7cc1061c0 2023-07-14T19:22:09.017 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:08.556+0000 7ff7d13a6700 1 -- 172.21.15.98:0/2361102029 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff7cc061c70 con 0x7ff7cc1061c0 2023-07-14T19:22:09.017 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:08.558+0000 7ff7b3fff700 1 -- 172.21.15.98:0/2361102029 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72446 (secure 0 0 0) 0x7ff7c001b450 con 0x7ff7cc1061c0 2023-07-14T19:22:09.017 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:08.624+0000 7ff7d13a6700 1 -- 172.21.15.98:0/2361102029 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "config assimilate-conf"} v 0) v1 -- 0x7ff7cc192460 con 0x7ff7cc1061c0 2023-07-14T19:22:09.017 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:08.628+0000 7ff7b3fff700 1 -- 172.21.15.98:0/2361102029 <== mon.0 v2:172.21.15.98:3300/0 7 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7ff7c001b5f0 con 0x7ff7cc1061c0 2023-07-14T19:22:09.017 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:08.628+0000 7ff7b3fff700 1 -- 172.21.15.98:0/2361102029 <== mon.0 v2:172.21.15.98:3300/0 8 ==== mon_command_ack([{"prefix": "config assimilate-conf"}]=0 v2) v1 ==== 70+0+454 (secure 0 0 0) 0x7ff7c00375d0 con 0x7ff7cc1061c0 2023-07-14T19:22:09.017 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:08.629+0000 7ff7d13a6700 1 -- 172.21.15.98:0/2361102029 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff7cc1061c0 msgr2=0x7ff7cc105b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:22:09.018 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:08.629+0000 7ff7d13a6700 1 --2- 172.21.15.98:0/2361102029 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff7cc1061c0 0x7ff7cc105b20 secure :-1 s=READY pgs=4 cs=0 l=1 rev1=1 crypto rx=0x7ff7c0004000 tx=0x7ff7c0004750 comp rx=0 tx=0).stop 2023-07-14T19:22:09.018 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:08.629+0000 7ff7d13a6700 1 -- 172.21.15.98:0/2361102029 shutdown_connections 2023-07-14T19:22:09.018 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:08.629+0000 7ff7d13a6700 1 --2- 172.21.15.98:0/2361102029 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff7cc1061c0 0x7ff7cc105b20 unknown :-1 s=CLOSED pgs=4 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:22:09.018 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:08.629+0000 7ff7d13a6700 1 -- 172.21.15.98:0/2361102029 >> 172.21.15.98:0/2361102029 conn(0x7ff7cc0ffbc0 msgr2=0x7ff7cc1015b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:22:09.018 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:08.629+0000 7ff7d13a6700 1 -- 172.21.15.98:0/2361102029 shutdown_connections 2023-07-14T19:22:09.018 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:08.629+0000 7ff7d13a6700 1 -- 172.21.15.98:0/2361102029 wait complete. 2023-07-14T19:22:09.018 INFO:teuthology.orchestra.run.smithi098.stdout:Generating new minimal ceph.conf... 2023-07-14T19:22:09.268 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:22:09 smithi098 ceph-mon[126990]: from='client.? 172.21.15.98:0/2361102029' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2023-07-14T19:22:09.269 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:22:09 smithi098 ceph-mon[126990]: from='client.? 172.21.15.98:0/2361102029' entity='client.admin' cmd='[{"prefix": "config assimilate-conf"}]': finished 2023-07-14T19:22:09.948 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:09.468+0000 7eff35432700 1 Processor -- start 2023-07-14T19:22:09.948 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:09.468+0000 7eff35432700 1 -- start start 2023-07-14T19:22:09.949 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:09.468+0000 7eff35432700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7eff301061c0 0x7eff301085a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:22:09.949 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:09.468+0000 7eff35432700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7eff30079d00 con 0x7eff301061c0 2023-07-14T19:22:09.949 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:09.468+0000 7eff2effd700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7eff301061c0 0x7eff301085a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:22:09.949 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:09.468+0000 7eff2effd700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7eff301061c0 0x7eff301085a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:44320/0 (socket says 172.21.15.98:44320) 2023-07-14T19:22:09.949 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:09.468+0000 7eff2effd700 1 -- 172.21.15.98:0/1949159789 learned_addr learned my addr 172.21.15.98:0/1949159789 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-07-14T19:22:09.949 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:09.468+0000 7eff2effd700 1 -- 172.21.15.98:0/1949159789 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7eff30079e40 con 0x7eff301061c0 2023-07-14T19:22:09.949 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:09.468+0000 7eff2effd700 1 --2- 172.21.15.98:0/1949159789 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7eff301061c0 0x7eff301085a0 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7eff200098a0 tx=0x7eff20009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=daa8f2f1292b3f17 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:22:09.950 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:09.469+0000 7eff2dffb700 1 -- 172.21.15.98:0/1949159789 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7eff20004160 con 0x7eff301061c0 2023-07-14T19:22:09.950 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:09.469+0000 7eff2dffb700 1 -- 172.21.15.98:0/1949159789 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7eff200042c0 con 0x7eff301061c0 2023-07-14T19:22:09.950 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:09.469+0000 7eff2dffb700 1 -- 172.21.15.98:0/1949159789 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7eff20011a00 con 0x7eff301061c0 2023-07-14T19:22:09.950 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:09.469+0000 7eff35432700 1 -- 172.21.15.98:0/1949159789 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7eff301061c0 msgr2=0x7eff301085a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:22:09.950 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:09.469+0000 7eff35432700 1 --2- 172.21.15.98:0/1949159789 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7eff301061c0 0x7eff301085a0 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7eff200098a0 tx=0x7eff20009bb0 comp rx=0 tx=0).stop 2023-07-14T19:22:09.950 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:09.469+0000 7eff35432700 1 -- 172.21.15.98:0/1949159789 shutdown_connections 2023-07-14T19:22:09.950 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:09.469+0000 7eff35432700 1 --2- 172.21.15.98:0/1949159789 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7eff301061c0 0x7eff301085a0 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:22:09.951 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:09.469+0000 7eff35432700 1 -- 172.21.15.98:0/1949159789 >> 172.21.15.98:0/1949159789 conn(0x7eff300ffba0 msgr2=0x7eff30101fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:22:09.951 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:09.469+0000 7eff35432700 1 -- 172.21.15.98:0/1949159789 shutdown_connections 2023-07-14T19:22:09.951 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:09.469+0000 7eff35432700 1 -- 172.21.15.98:0/1949159789 wait complete. 2023-07-14T19:22:09.951 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:09.470+0000 7eff35432700 1 Processor -- start 2023-07-14T19:22:09.952 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:09.470+0000 7eff35432700 1 -- start start 2023-07-14T19:22:09.952 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:09.470+0000 7eff35432700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7eff301061c0 0x7eff30199400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:22:09.952 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:09.470+0000 7eff35432700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7eff30199940 con 0x7eff301061c0 2023-07-14T19:22:09.952 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:09.471+0000 7eff2effd700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7eff301061c0 0x7eff30199400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:22:09.952 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:09.471+0000 7eff2effd700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7eff301061c0 0x7eff30199400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:44322/0 (socket says 172.21.15.98:44322) 2023-07-14T19:22:09.953 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:09.471+0000 7eff2effd700 1 -- 172.21.15.98:0/1811007434 learned_addr learned my addr 172.21.15.98:0/1811007434 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-07-14T19:22:09.953 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:09.471+0000 7eff2effd700 1 -- 172.21.15.98:0/1811007434 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7eff20009550 con 0x7eff301061c0 2023-07-14T19:22:09.953 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:09.471+0000 7eff2effd700 1 --2- 172.21.15.98:0/1811007434 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7eff301061c0 0x7eff30199400 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7eff20009fb0 tx=0x7eff20011ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:22:09.953 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:09.471+0000 7eff17fff700 1 -- 172.21.15.98:0/1811007434 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7eff20003930 con 0x7eff301061c0 2023-07-14T19:22:09.953 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:09.471+0000 7eff17fff700 1 -- 172.21.15.98:0/1811007434 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7eff20003a90 con 0x7eff301061c0 2023-07-14T19:22:09.953 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:09.471+0000 7eff35432700 1 -- 172.21.15.98:0/1811007434 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7eff30199b40 con 0x7eff301061c0 2023-07-14T19:22:09.954 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:09.471+0000 7eff17fff700 1 -- 172.21.15.98:0/1811007434 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7eff200195e0 con 0x7eff301061c0 2023-07-14T19:22:09.954 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:09.471+0000 7eff35432700 1 -- 172.21.15.98:0/1811007434 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7eff30199f60 con 0x7eff301061c0 2023-07-14T19:22:09.954 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:09.471+0000 7eff17fff700 1 -- 172.21.15.98:0/1811007434 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7eff20019bc0 con 0x7eff301061c0 2023-07-14T19:22:09.954 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:09.471+0000 7eff17fff700 1 -- 172.21.15.98:0/1811007434 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7eff200243f0 con 0x7eff301061c0 2023-07-14T19:22:09.954 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:09.472+0000 7eff35432700 1 -- 172.21.15.98:0/1811007434 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7eff30193410 con 0x7eff301061c0 2023-07-14T19:22:09.954 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:09.474+0000 7eff17fff700 1 -- 172.21.15.98:0/1811007434 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72446 (secure 0 0 0) 0x7eff20003c00 con 0x7eff301061c0 2023-07-14T19:22:09.955 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:09.540+0000 7eff35432700 1 -- 172.21.15.98:0/1811007434 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "config generate-minimal-conf"} v 0) v1 -- 0x7eff3002c8e0 con 0x7eff301061c0 2023-07-14T19:22:09.955 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:09.540+0000 7eff17fff700 1 -- 172.21.15.98:0/1811007434 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "config generate-minimal-conf"}]=0 v2) v1 ==== 76+0+175 (secure 0 0 0) 0x7eff20035090 con 0x7eff301061c0 2023-07-14T19:22:09.955 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:09.541+0000 7eff35432700 1 -- 172.21.15.98:0/1811007434 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7eff301061c0 msgr2=0x7eff30199400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:22:09.955 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:09.541+0000 7eff35432700 1 --2- 172.21.15.98:0/1811007434 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7eff301061c0 0x7eff30199400 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7eff20009fb0 tx=0x7eff20011ad0 comp rx=0 tx=0).stop 2023-07-14T19:22:09.955 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:09.541+0000 7eff35432700 1 -- 172.21.15.98:0/1811007434 shutdown_connections 2023-07-14T19:22:09.955 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:09.541+0000 7eff35432700 1 --2- 172.21.15.98:0/1811007434 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7eff301061c0 0x7eff30199400 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:22:09.956 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:09.541+0000 7eff35432700 1 -- 172.21.15.98:0/1811007434 >> 172.21.15.98:0/1811007434 conn(0x7eff300ffba0 msgr2=0x7eff30101310 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:22:09.956 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:09.541+0000 7eff35432700 1 -- 172.21.15.98:0/1811007434 shutdown_connections 2023-07-14T19:22:09.956 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:09.541+0000 7eff35432700 1 -- 172.21.15.98:0/1811007434 wait complete. 2023-07-14T19:22:09.956 INFO:teuthology.orchestra.run.smithi098.stdout:Restarting the monitor... 2023-07-14T19:22:10.186 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:22:09 smithi098 systemd[1]: Stopping Ceph mon.a for 9adea648-227b-11ee-9b33-001a4aab830c... 2023-07-14T19:22:10.525 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:22:10 smithi098 ceph-mon[126990]: from='client.? 172.21.15.98:0/1811007434' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T19:22:10.525 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:22:10 smithi098 ceph-9adea648-227b-11ee-9b33-001a4aab830c-mon-a[126966]: 2023-07-14T19:22:10.283+0000 7fe42f8d9700 -1 received signal: Terminated from /dev/init -- /usr/bin/ceph-mon -n mon.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2023-07-14T19:22:10.525 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:22:10 smithi098 ceph-9adea648-227b-11ee-9b33-001a4aab830c-mon-a[126966]: 2023-07-14T19:22:10.283+0000 7fe42f8d9700 -1 mon.a@0(leader) e1 *** Got Signal Terminated *** 2023-07-14T19:22:10.781 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:22:10 smithi098 bash[127320]: ceph-9adea648-227b-11ee-9b33-001a4aab830c-mon-a 2023-07-14T19:22:10.781 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:22:10 smithi098 systemd[1]: ceph-9adea648-227b-11ee-9b33-001a4aab830c@mon.a.service: Succeeded. 2023-07-14T19:22:10.781 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:22:10 smithi098 systemd[1]: Stopped Ceph mon.a for 9adea648-227b-11ee-9b33-001a4aab830c. 2023-07-14T19:22:10.782 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:22:10 smithi098 systemd[1]: Starting Ceph mon.a for 9adea648-227b-11ee-9b33-001a4aab830c... 2023-07-14T19:22:11.201 INFO:teuthology.orchestra.run.smithi098.stdout:Setting public_network to 172.21.0.0/20 in mon config section 2023-07-14T19:22:11.282 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:22:11 smithi098 podman[127446]: 2023-07-14T19:22:11.282 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:22:11 smithi098 ceph-mon[127486]: set uid:gid to 167:167 (ceph:ceph) 2023-07-14T19:22:11.282 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:22:11 smithi098 ceph-mon[127486]: ceph version 18.0.0-4940-g81c9387d (81c9387d2e7576da065747ec9c16d1c7d69016ec) reef (dev), process ceph-mon, pid 7 2023-07-14T19:22:11.282 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:22:11 smithi098 bash[127446]: 99ff082ac7c0f72820e4250f2b9a7927138f76dc354040574eff850104c15c49 2023-07-14T19:22:11.282 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:22:11 smithi098 ceph-mon[127486]: pidfile_write: ignore empty --pid-file 2023-07-14T19:22:11.283 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:22:11 smithi098 ceph-mon[127486]: load: jerasure load: lrc 2023-07-14T19:22:11.283 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:22:11 smithi098 ceph-mon[127486]: rocksdb: RocksDB version: 7.9.2 2023-07-14T19:22:11.283 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:22:11 smithi098 ceph-mon[127486]: rocksdb: Git sha 0 2023-07-14T19:22:11.283 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:22:11 smithi098 ceph-mon[127486]: rocksdb: Compile date 2023-07-14 14:45:42 2023-07-14T19:22:11.283 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:22:11 smithi098 ceph-mon[127486]: rocksdb: DB SUMMARY 2023-07-14T19:22:11.284 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:22:11 smithi098 ceph-mon[127486]: rocksdb: DB Session ID: DRSGD8Y7LAPNTZGIAKVJ 2023-07-14T19:22:11.284 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:22:11 smithi098 ceph-mon[127486]: rocksdb: CURRENT file: CURRENT 2023-07-14T19:22:11.284 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:22:11 smithi098 ceph-mon[127486]: rocksdb: IDENTITY file: IDENTITY 2023-07-14T19:22:11.285 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:22:11 smithi098 ceph-mon[127486]: rocksdb: MANIFEST file: MANIFEST-000010 size: 179 Bytes 2023-07-14T19:22:11.285 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:22:11 smithi098 ceph-mon[127486]: rocksdb: SST files in /var/lib/ceph/mon/ceph-a/store.db dir, Total Num: 1, files: 000008.sst 2023-07-14T19:22:11.285 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:22:11 smithi098 ceph-mon[127486]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-a/store.db: 000009.log size: 99943 ; 2023-07-14T19:22:11.285 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:22:11 smithi098 ceph-mon[127486]: rocksdb: Options.error_if_exists: 0 2023-07-14T19:22:11.285 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:22:11 smithi098 ceph-mon[127486]: rocksdb: Options.create_if_missing: 0 2023-07-14T19:22:11.286 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:22:11 smithi098 ceph-mon[127486]: rocksdb: Options.paranoid_checks: 1 2023-07-14T19:22:11.286 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:22:11 smithi098 ceph-mon[127486]: rocksdb: Options.flush_verify_memtable_count: 1 2023-07-14T19:22:11.286 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:22:11 smithi098 ceph-mon[127486]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2023-07-14T19:22:11.286 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:22:11 smithi098 ceph-mon[127486]: rocksdb: Options.verify_sst_unique_id_in_manifest: 1 2023-07-14T19:22:11.286 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:22:11 smithi098 ceph-mon[127486]: rocksdb: Options.env: 0x55f11f184c60 2023-07-14T19:22:11.286 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:22:11 smithi098 ceph-mon[127486]: rocksdb: Options.fs: PosixFileSystem 2023-07-14T19:22:11.287 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:22:11 smithi098 ceph-mon[127486]: rocksdb: Options.info_log: 0x55f1209532a0 2023-07-14T19:22:11.287 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:22:11 smithi098 ceph-mon[127486]: rocksdb: Options.max_file_opening_threads: 16 2023-07-14T19:22:11.287 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:22:11 smithi098 ceph-mon[127486]: rocksdb: Options.statistics: (nil) 2023-07-14T19:22:11.287 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:22:11 smithi098 ceph-mon[127486]: rocksdb: Options.use_fsync: 0 2023-07-14T19:22:11.288 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:22:11 smithi098 ceph-mon[127486]: rocksdb: Options.max_log_file_size: 0 2023-07-14T19:22:11.288 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:22:11 smithi098 ceph-mon[127486]: rocksdb: Options.max_manifest_file_size: 1073741824 2023-07-14T19:22:11.288 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:22:11 smithi098 ceph-mon[127486]: rocksdb: Options.log_file_time_to_roll: 0 2023-07-14T19:22:11.288 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:22:11 smithi098 ceph-mon[127486]: rocksdb: Options.keep_log_file_num: 1000 2023-07-14T19:22:11.289 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:22:11 smithi098 ceph-mon[127486]: rocksdb: Options.recycle_log_file_num: 0 2023-07-14T19:22:11.289 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:22:11 smithi098 ceph-mon[127486]: rocksdb: Options.allow_fallocate: 1 2023-07-14T19:22:11.289 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:22:11 smithi098 ceph-mon[127486]: rocksdb: Options.allow_mmap_reads: 0 2023-07-14T19:22:11.289 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:22:11 smithi098 ceph-mon[127486]: rocksdb: Options.allow_mmap_writes: 0 2023-07-14T19:22:11.289 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:22:11 smithi098 ceph-mon[127486]: rocksdb: Options.use_direct_reads: 0 2023-07-14T19:22:11.289 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:22:11 smithi098 ceph-mon[127486]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2023-07-14T19:22:11.290 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:22:11 smithi098 ceph-mon[127486]: rocksdb: Options.create_missing_column_families: 0 2023-07-14T19:22:11.290 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:22:11 smithi098 ceph-mon[127486]: rocksdb: Options.db_log_dir: 2023-07-14T19:22:11.290 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:22:11 smithi098 ceph-mon[127486]: rocksdb: Options.wal_dir: 2023-07-14T19:22:11.290 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:22:11 smithi098 ceph-mon[127486]: rocksdb: Options.table_cache_numshardbits: 6 2023-07-14T19:22:11.290 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:22:11 smithi098 ceph-mon[127486]: rocksdb: Options.WAL_ttl_seconds: 0 2023-07-14T19:22:11.290 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:22:11 smithi098 ceph-mon[127486]: rocksdb: Options.WAL_size_limit_MB: 0 2023-07-14T19:22:11.290 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:22:11 smithi098 ceph-mon[127486]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2023-07-14T19:22:11.291 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:22:11 smithi098 ceph-mon[127486]: rocksdb: Options.manifest_preallocation_size: 4194304 2023-07-14T19:22:11.291 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:22:11 smithi098 ceph-mon[127486]: rocksdb: Options.is_fd_close_on_exec: 1 2023-07-14T19:22:11.291 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:22:11 smithi098 ceph-mon[127486]: rocksdb: Options.advise_random_on_open: 1 2023-07-14T19:22:11.291 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:22:11 smithi098 ceph-mon[127486]: rocksdb: Options.db_write_buffer_size: 0 2023-07-14T19:22:11.291 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:22:11 smithi098 ceph-mon[127486]: rocksdb: Options.write_buffer_manager: 0x55f1208743c0 2023-07-14T19:22:11.291 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:22:11 smithi098 ceph-mon[127486]: rocksdb: Options.access_hint_on_compaction_start: 1 2023-07-14T19:22:11.291 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:22:11 smithi098 ceph-mon[127486]: rocksdb: Options.random_access_max_buffer_size: 1048576 2023-07-14T19:22:11.291 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:22:11 smithi098 ceph-mon[127486]: rocksdb: Options.use_adaptive_mutex: 0 2023-07-14T19:22:11.292 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:22:11 smithi098 ceph-mon[127486]: rocksdb: Options.rate_limiter: (nil) 2023-07-14T19:22:11.292 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:22:11 smithi098 ceph-mon[127486]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2023-07-14T19:22:11.292 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:22:11 smithi098 ceph-mon[127486]: rocksdb: Options.wal_recovery_mode: 2 2023-07-14T19:22:11.292 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:22:11 smithi098 ceph-mon[127486]: rocksdb: Options.enable_thread_tracking: 0 2023-07-14T19:22:11.292 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:22:11 smithi098 ceph-mon[127486]: rocksdb: Options.enable_pipelined_write: 0 2023-07-14T19:22:11.292 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:22:11 smithi098 ceph-mon[127486]: rocksdb: Options.unordered_write: 0 2023-07-14T19:22:11.292 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:22:11 smithi098 ceph-mon[127486]: rocksdb: Options.allow_concurrent_memtable_write: 1 2023-07-14T19:22:11.293 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:22:11 smithi098 ceph-mon[127486]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2023-07-14T19:22:11.293 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:22:11 smithi098 ceph-mon[127486]: rocksdb: Options.write_thread_max_yield_usec: 100 2023-07-14T19:22:11.293 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:22:11 smithi098 ceph-mon[127486]: rocksdb: Options.write_thread_slow_yield_usec: 3 2023-07-14T19:22:11.293 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:22:11 smithi098 ceph-mon[127486]: rocksdb: Options.row_cache: None 2023-07-14T19:22:11.293 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:22:11 smithi098 ceph-mon[127486]: rocksdb: Options.wal_filter: None 2023-07-14T19:22:11.293 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:22:11 smithi098 ceph-mon[127486]: rocksdb: Options.avoid_flush_during_recovery: 0 2023-07-14T19:22:11.293 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:22:11 smithi098 ceph-mon[127486]: rocksdb: Options.allow_ingest_behind: 0 2023-07-14T19:22:11.294 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:22:11 smithi098 ceph-mon[127486]: rocksdb: Options.two_write_queues: 0 2023-07-14T19:22:11.294 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:22:11 smithi098 ceph-mon[127486]: rocksdb: Options.manual_wal_flush: 0 2023-07-14T19:22:11.294 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:22:11 smithi098 ceph-mon[127486]: rocksdb: Options.wal_compression: 0 2023-07-14T19:22:11.294 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:22:11 smithi098 ceph-mon[127486]: rocksdb: Options.atomic_flush: 0 2023-07-14T19:22:11.295 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:22:11 smithi098 ceph-mon[127486]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2023-07-14T19:22:11.295 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:22:11 smithi098 ceph-mon[127486]: rocksdb: Options.persist_stats_to_disk: 0 2023-07-14T19:22:11.295 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:22:11 smithi098 ceph-mon[127486]: rocksdb: Options.write_dbid_to_manifest: 0 2023-07-14T19:22:11.295 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:22:11 smithi098 ceph-mon[127486]: rocksdb: Options.log_readahead_size: 0 2023-07-14T19:22:11.296 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:22:11 smithi098 ceph-mon[127486]: rocksdb: Options.file_checksum_gen_factory: Unknown 2023-07-14T19:22:11.296 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:22:11 smithi098 ceph-mon[127486]: rocksdb: Options.best_efforts_recovery: 0 2023-07-14T19:22:11.296 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:22:11 smithi098 ceph-mon[127486]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2023-07-14T19:22:11.297 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:22:11 smithi098 ceph-mon[127486]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2023-07-14T19:22:11.299 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:22:11 smithi098 ceph-mon[127486]: rocksdb: Options.allow_data_in_errors: 0 2023-07-14T19:22:11.299 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:22:11 smithi098 ceph-mon[127486]: rocksdb: Options.db_host_id: __hostname__ 2023-07-14T19:22:11.299 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:22:11 smithi098 ceph-mon[127486]: rocksdb: Options.enforce_single_del_contracts: true 2023-07-14T19:22:11.300 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:22:11 smithi098 ceph-mon[127486]: rocksdb: Options.max_background_jobs: 2 2023-07-14T19:22:11.300 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:22:11 smithi098 ceph-mon[127486]: rocksdb: Options.max_background_compactions: -1 2023-07-14T19:22:11.300 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:22:11 smithi098 ceph-mon[127486]: rocksdb: Options.max_subcompactions: 1 2023-07-14T19:22:11.301 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:22:11 smithi098 ceph-mon[127486]: rocksdb: Options.avoid_flush_during_shutdown: 0 2023-07-14T19:22:11.301 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:22:11 smithi098 ceph-mon[127486]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2023-07-14T19:22:11.301 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:22:11 smithi098 ceph-mon[127486]: rocksdb: Options.delayed_write_rate : 16777216 2023-07-14T19:22:11.302 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:22:11 smithi098 ceph-mon[127486]: rocksdb: Options.max_total_wal_size: 0 2023-07-14T19:22:11.302 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:22:11 smithi098 ceph-mon[127486]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2023-07-14T19:22:11.302 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:22:11 smithi098 ceph-mon[127486]: rocksdb: Options.stats_dump_period_sec: 600 2023-07-14T19:22:11.303 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:22:11 smithi098 ceph-mon[127486]: rocksdb: Options.stats_persist_period_sec: 600 2023-07-14T19:22:11.303 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:22:11 smithi098 ceph-mon[127486]: rocksdb: Options.stats_history_buffer_size: 1048576 2023-07-14T19:22:11.303 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:22:11 smithi098 ceph-mon[127486]: rocksdb: Options.max_open_files: -1 2023-07-14T19:22:11.303 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:22:11 smithi098 ceph-mon[127486]: rocksdb: Options.bytes_per_sync: 0 2023-07-14T19:22:11.304 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:22:11 smithi098 ceph-mon[127486]: rocksdb: Options.wal_bytes_per_sync: 0 2023-07-14T19:22:11.304 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:22:11 smithi098 ceph-mon[127486]: rocksdb: Options.strict_bytes_per_sync: 0 2023-07-14T19:22:11.304 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:22:11 smithi098 ceph-mon[127486]: rocksdb: Options.compaction_readahead_size: 0 2023-07-14T19:22:11.305 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:22:11 smithi098 ceph-mon[127486]: rocksdb: Options.max_background_flushes: -1 2023-07-14T19:22:11.305 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:22:11 smithi098 ceph-mon[127486]: rocksdb: Compression algorithms supported: 2023-07-14T19:22:11.305 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:22:11 smithi098 ceph-mon[127486]: rocksdb: kZSTDNotFinalCompression supported: 0 2023-07-14T19:22:11.305 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:22:11 smithi098 ceph-mon[127486]: rocksdb: kZSTD supported: 0 2023-07-14T19:22:11.306 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:22:11 smithi098 ceph-mon[127486]: rocksdb: kXpressCompression supported: 0 2023-07-14T19:22:11.306 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:22:11 smithi098 ceph-mon[127486]: rocksdb: kLZ4HCCompression supported: 1 2023-07-14T19:22:11.306 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:22:11 smithi098 ceph-mon[127486]: rocksdb: kZlibCompression supported: 1 2023-07-14T19:22:11.306 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:22:11 smithi098 ceph-mon[127486]: rocksdb: kSnappyCompression supported: 1 2023-07-14T19:22:11.306 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:22:11 smithi098 ceph-mon[127486]: rocksdb: kLZ4Compression supported: 1 2023-07-14T19:22:11.307 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:22:11 smithi098 ceph-mon[127486]: rocksdb: kBZip2Compression supported: 0 2023-07-14T19:22:11.307 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:22:11 smithi098 ceph-mon[127486]: rocksdb: Fast CRC32 supported: Supported on x86 2023-07-14T19:22:11.307 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:22:11 smithi098 ceph-mon[127486]: rocksdb: DMutex implementation: pthread_mutex_t 2023-07-14T19:22:11.308 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:22:11 smithi098 ceph-mon[127486]: rocksdb: [db/version_set.cc:5527] Recovering from manifest file: /var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000010 2023-07-14T19:22:11.308 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:22:11 smithi098 ceph-mon[127486]: rocksdb: [db/column_family.cc:630] --------------- Options for column family [default]: 2023-07-14T19:22:11.308 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:22:11 smithi098 ceph-mon[127486]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2023-07-14T19:22:11.309 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:22:11 smithi098 ceph-mon[127486]: rocksdb: Options.merge_operator: 2023-07-14T19:22:11.309 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:22:11 smithi098 ceph-mon[127486]: rocksdb: Options.compaction_filter: None 2023-07-14T19:22:11.309 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:22:11 smithi098 ceph-mon[127486]: rocksdb: Options.compaction_filter_factory: None 2023-07-14T19:22:11.310 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:22:11 smithi098 ceph-mon[127486]: rocksdb: Options.sst_partitioner_factory: None 2023-07-14T19:22:11.310 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:22:11 smithi098 ceph-mon[127486]: rocksdb: Options.memtable_factory: SkipListFactory 2023-07-14T19:22:11.310 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:22:11 smithi098 ceph-mon[127486]: rocksdb: Options.table_factory: BlockBasedTable 2023-07-14T19:22:11.310 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:22:11 smithi098 ceph-mon[127486]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x55f1209533a0) 2023-07-14T19:22:11.311 INFO:journalctl@ceph.mon.a.smithi098.stdout: cache_index_and_filter_blocks: 1 2023-07-14T19:22:11.311 INFO:journalctl@ceph.mon.a.smithi098.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2023-07-14T19:22:11.311 INFO:journalctl@ceph.mon.a.smithi098.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2023-07-14T19:22:11.312 INFO:journalctl@ceph.mon.a.smithi098.stdout: pin_top_level_index_and_filter: 1 2023-07-14T19:22:11.312 INFO:journalctl@ceph.mon.a.smithi098.stdout: index_type: 0 2023-07-14T19:22:11.312 INFO:journalctl@ceph.mon.a.smithi098.stdout: data_block_index_type: 0 2023-07-14T19:22:11.312 INFO:journalctl@ceph.mon.a.smithi098.stdout: index_shortening: 1 2023-07-14T19:22:11.313 INFO:journalctl@ceph.mon.a.smithi098.stdout: data_block_hash_table_util_ratio: 0.750000 2023-07-14T19:22:11.313 INFO:journalctl@ceph.mon.a.smithi098.stdout: checksum: 4 2023-07-14T19:22:11.313 INFO:journalctl@ceph.mon.a.smithi098.stdout: no_block_cache: 0 2023-07-14T19:22:11.313 INFO:journalctl@ceph.mon.a.smithi098.stdout: block_cache: 0x55f1209131f0 2023-07-14T19:22:11.314 INFO:journalctl@ceph.mon.a.smithi098.stdout: block_cache_name: BinnedLRUCache 2023-07-14T19:22:11.314 INFO:journalctl@ceph.mon.a.smithi098.stdout: block_cache_options: 2023-07-14T19:22:11.314 INFO:journalctl@ceph.mon.a.smithi098.stdout: capacity : 536870912 2023-07-14T19:22:11.314 INFO:journalctl@ceph.mon.a.smithi098.stdout: num_shard_bits : 4 2023-07-14T19:22:11.315 INFO:journalctl@ceph.mon.a.smithi098.stdout: strict_capacity_limit : 0 2023-07-14T19:22:11.315 INFO:journalctl@ceph.mon.a.smithi098.stdout: high_pri_pool_ratio: 0.000 2023-07-14T19:22:11.315 INFO:journalctl@ceph.mon.a.smithi098.stdout: block_cache_compressed: (nil) 2023-07-14T19:22:11.315 INFO:journalctl@ceph.mon.a.smithi098.stdout: persistent_cache: (nil) 2023-07-14T19:22:11.315 INFO:journalctl@ceph.mon.a.smithi098.stdout: block_size: 4096 2023-07-14T19:22:11.316 INFO:journalctl@ceph.mon.a.smithi098.stdout: block_size_deviation: 10 2023-07-14T19:22:11.316 INFO:journalctl@ceph.mon.a.smithi098.stdout: block_restart_interval: 16 2023-07-14T19:22:11.316 INFO:journalctl@ceph.mon.a.smithi098.stdout: index_block_restart_interval: 1 2023-07-14T19:22:11.316 INFO:journalctl@ceph.mon.a.smithi098.stdout: metadata_block_size: 4096 2023-07-14T19:22:11.316 INFO:journalctl@ceph.mon.a.smithi098.stdout: partition_filters: 0 2023-07-14T19:22:11.316 INFO:journalctl@ceph.mon.a.smithi098.stdout: use_delta_encoding: 1 2023-07-14T19:22:11.316 INFO:journalctl@ceph.mon.a.smithi098.stdout: filter_policy: bloomfilter 2023-07-14T19:22:11.317 INFO:journalctl@ceph.mon.a.smithi098.stdout: whole_key_filtering: 1 2023-07-14T19:22:11.317 INFO:journalctl@ceph.mon.a.smithi098.stdout: verify_compression: 0 2023-07-14T19:22:11.317 INFO:journalctl@ceph.mon.a.smithi098.stdout: read_amp_bytes_per_bit: 0 2023-07-14T19:22:11.317 INFO:journalctl@ceph.mon.a.smithi098.stdout: format_version: 5 2023-07-14T19:22:11.317 INFO:journalctl@ceph.mon.a.smithi098.stdout: enable_index_compression: 1 2023-07-14T19:22:11.317 INFO:journalctl@ceph.mon.a.smithi098.stdout: block_align: 0 2023-07-14T19:22:11.317 INFO:journalctl@ceph.mon.a.smithi098.stdout: max_auto_readahead_size: 262144 2023-07-14T19:22:11.318 INFO:journalctl@ceph.mon.a.smithi098.stdout: prepopulate_block_cache: 0 2023-07-14T19:22:11.318 INFO:journalctl@ceph.mon.a.smithi098.stdout: initial_auto_readahead_size: 8192 2023-07-14T19:22:11.318 INFO:journalctl@ceph.mon.a.smithi098.stdout: num_file_reads_for_auto_readahead: 2 2023-07-14T19:22:11.318 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:22:11 smithi098 ceph-mon[127486]: rocksdb: Options.write_buffer_size: 33554432 2023-07-14T19:22:11.318 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:22:11 smithi098 ceph-mon[127486]: rocksdb: Options.max_write_buffer_number: 2 2023-07-14T19:22:11.318 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:22:11 smithi098 ceph-mon[127486]: rocksdb: Options.compression: NoCompression 2023-07-14T19:22:11.319 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:22:11 smithi098 ceph-mon[127486]: rocksdb: Options.bottommost_compression: Disabled 2023-07-14T19:22:11.319 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:22:11 smithi098 ceph-mon[127486]: rocksdb: Options.prefix_extractor: nullptr 2023-07-14T19:22:11.319 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:22:11 smithi098 ceph-mon[127486]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2023-07-14T19:22:11.319 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:22:11 smithi098 ceph-mon[127486]: rocksdb: Options.num_levels: 7 2023-07-14T19:22:11.319 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:22:11 smithi098 ceph-mon[127486]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2023-07-14T19:22:11.319 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:22:11 smithi098 ceph-mon[127486]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2023-07-14T19:22:11.320 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:22:11 smithi098 ceph-mon[127486]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2023-07-14T19:22:11.320 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:22:11 smithi098 ceph-mon[127486]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2023-07-14T19:22:11.320 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:22:11 smithi098 ceph-mon[127486]: rocksdb: Options.bottommost_compression_opts.level: 32767 2023-07-14T19:22:11.320 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:22:11 smithi098 ceph-mon[127486]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2023-07-14T19:22:11.320 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:22:11 smithi098 ceph-mon[127486]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2023-07-14T19:22:11.320 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:22:11 smithi098 ceph-mon[127486]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2023-07-14T19:22:11.320 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:22:11 smithi098 ceph-mon[127486]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2023-07-14T19:22:11.321 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:22:11 smithi098 ceph-mon[127486]: rocksdb: Options.bottommost_compression_opts.enabled: false 2023-07-14T19:22:11.321 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:22:11 smithi098 ceph-mon[127486]: rocksdb: Options.bottommost_compression_opts.max_dict_buffer_bytes: 0 2023-07-14T19:22:11.321 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:22:11 smithi098 ceph-mon[127486]: rocksdb: Options.bottommost_compression_opts.use_zstd_dict_trainer: true 2023-07-14T19:22:11.321 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:22:11 smithi098 ceph-mon[127486]: rocksdb: Options.compression_opts.window_bits: -14 2023-07-14T19:22:11.321 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:22:11 smithi098 ceph-mon[127486]: rocksdb: Options.compression_opts.level: 32767 2023-07-14T19:22:11.321 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:22:11 smithi098 ceph-mon[127486]: rocksdb: Options.compression_opts.strategy: 0 2023-07-14T19:22:11.321 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:22:11 smithi098 ceph-mon[127486]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2023-07-14T19:22:11.322 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:22:11 smithi098 ceph-mon[127486]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2023-07-14T19:22:11.322 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:22:11 smithi098 ceph-mon[127486]: rocksdb: Options.compression_opts.use_zstd_dict_trainer: true 2023-07-14T19:22:11.322 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:22:11 smithi098 ceph-mon[127486]: rocksdb: Options.compression_opts.parallel_threads: 1 2023-07-14T19:22:11.322 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:22:11 smithi098 ceph-mon[127486]: rocksdb: Options.compression_opts.enabled: false 2023-07-14T19:22:11.322 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:22:11 smithi098 ceph-mon[127486]: rocksdb: Options.compression_opts.max_dict_buffer_bytes: 0 2023-07-14T19:22:11.322 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:22:11 smithi098 ceph-mon[127486]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2023-07-14T19:22:11.322 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:22:11 smithi098 ceph-mon[127486]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2023-07-14T19:22:11.323 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:22:11 smithi098 ceph-mon[127486]: rocksdb: Options.level0_stop_writes_trigger: 36 2023-07-14T19:22:11.323 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:22:11 smithi098 ceph-mon[127486]: rocksdb: Options.target_file_size_base: 67108864 2023-07-14T19:22:11.323 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:22:11 smithi098 ceph-mon[127486]: rocksdb: Options.target_file_size_multiplier: 1 2023-07-14T19:22:11.323 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:22:11 smithi098 ceph-mon[127486]: rocksdb: Options.max_bytes_for_level_base: 268435456 2023-07-14T19:22:11.323 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:22:11 smithi098 ceph-mon[127486]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2023-07-14T19:22:11.323 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:22:11 smithi098 ceph-mon[127486]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2023-07-14T19:22:11.323 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:22:11 smithi098 ceph-mon[127486]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2023-07-14T19:22:11.324 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:22:11 smithi098 ceph-mon[127486]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2023-07-14T19:22:11.324 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:22:11 smithi098 ceph-mon[127486]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2023-07-14T19:22:11.324 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:22:11 smithi098 ceph-mon[127486]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2023-07-14T19:22:11.324 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:22:11 smithi098 ceph-mon[127486]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2023-07-14T19:22:11.324 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:22:11 smithi098 ceph-mon[127486]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2023-07-14T19:22:11.325 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:22:11 smithi098 ceph-mon[127486]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2023-07-14T19:22:11.325 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:22:11 smithi098 ceph-mon[127486]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2023-07-14T19:22:11.325 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:22:11 smithi098 ceph-mon[127486]: rocksdb: Options.max_compaction_bytes: 1677721600 2023-07-14T19:22:11.325 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:22:11 smithi098 ceph-mon[127486]: rocksdb: Options.ignore_max_compaction_bytes_for_input: true 2023-07-14T19:22:11.325 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:22:11 smithi098 ceph-mon[127486]: rocksdb: Options.arena_block_size: 1048576 2023-07-14T19:22:11.325 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:22:11 smithi098 ceph-mon[127486]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2023-07-14T19:22:11.326 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:22:11 smithi098 ceph-mon[127486]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2023-07-14T19:22:11.326 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:22:11 smithi098 ceph-mon[127486]: rocksdb: Options.disable_auto_compactions: 0 2023-07-14T19:22:11.326 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:22:11 smithi098 ceph-mon[127486]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2023-07-14T19:22:11.326 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:22:11 smithi098 ceph-mon[127486]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2023-07-14T19:22:11.326 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:22:11 smithi098 ceph-mon[127486]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2023-07-14T19:22:11.326 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:22:11 smithi098 ceph-mon[127486]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2023-07-14T19:22:11.326 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:22:11 smithi098 ceph-mon[127486]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2023-07-14T19:22:11.326 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:22:11 smithi098 ceph-mon[127486]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2023-07-14T19:22:11.327 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:22:11 smithi098 ceph-mon[127486]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2023-07-14T19:22:11.327 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:22:11 smithi098 ceph-mon[127486]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2023-07-14T19:22:11.327 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:22:11 smithi098 ceph-mon[127486]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2023-07-14T19:22:11.327 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:22:11 smithi098 ceph-mon[127486]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2023-07-14T19:22:11.327 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:22:11 smithi098 ceph-mon[127486]: rocksdb: Options.table_properties_collectors: 2023-07-14T19:22:11.327 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:22:11 smithi098 ceph-mon[127486]: rocksdb: Options.inplace_update_support: 0 2023-07-14T19:22:11.328 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:22:11 smithi098 ceph-mon[127486]: rocksdb: Options.inplace_update_num_locks: 10000 2023-07-14T19:22:11.328 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:22:11 smithi098 ceph-mon[127486]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2023-07-14T19:22:11.328 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:22:11 smithi098 ceph-mon[127486]: rocksdb: Options.memtable_whole_key_filtering: 0 2023-07-14T19:22:11.328 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:22:11 smithi098 ceph-mon[127486]: rocksdb: Options.memtable_huge_page_size: 0 2023-07-14T19:22:11.329 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:22:11 smithi098 ceph-mon[127486]: rocksdb: Options.bloom_locality: 0 2023-07-14T19:22:11.329 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:22:11 smithi098 ceph-mon[127486]: rocksdb: Options.max_successive_merges: 0 2023-07-14T19:22:11.329 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:22:11 smithi098 ceph-mon[127486]: rocksdb: Options.optimize_filters_for_hits: 0 2023-07-14T19:22:11.329 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:22:11 smithi098 ceph-mon[127486]: rocksdb: Options.paranoid_file_checks: 0 2023-07-14T19:22:11.329 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:22:11 smithi098 ceph-mon[127486]: rocksdb: Options.force_consistency_checks: 1 2023-07-14T19:22:11.330 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:22:11 smithi098 ceph-mon[127486]: rocksdb: Options.report_bg_io_stats: 0 2023-07-14T19:22:11.330 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:22:11 smithi098 ceph-mon[127486]: rocksdb: Options.ttl: 2592000 2023-07-14T19:22:11.330 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:22:11 smithi098 ceph-mon[127486]: rocksdb: Options.periodic_compaction_seconds: 0 2023-07-14T19:22:11.330 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:22:11 smithi098 ceph-mon[127486]: rocksdb: Options.preclude_last_level_data_seconds: 0 2023-07-14T19:22:11.330 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:22:11 smithi098 ceph-mon[127486]: rocksdb: Options.preserve_internal_time_seconds: 0 2023-07-14T19:22:11.330 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:22:11 smithi098 ceph-mon[127486]: rocksdb: Options.enable_blob_files: false 2023-07-14T19:22:11.330 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:22:11 smithi098 ceph-mon[127486]: rocksdb: Options.min_blob_size: 0 2023-07-14T19:22:11.331 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:22:11 smithi098 ceph-mon[127486]: rocksdb: Options.blob_file_size: 268435456 2023-07-14T19:22:11.331 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:22:11 smithi098 ceph-mon[127486]: rocksdb: Options.blob_compression_type: NoCompression 2023-07-14T19:22:11.331 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:22:11 smithi098 ceph-mon[127486]: rocksdb: Options.enable_blob_garbage_collection: false 2023-07-14T19:22:11.331 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:22:11 smithi098 ceph-mon[127486]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2023-07-14T19:22:11.331 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:22:11 smithi098 ceph-mon[127486]: rocksdb: Options.blob_garbage_collection_force_threshold: 1.000000 2023-07-14T19:22:11.331 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:22:11 smithi098 ceph-mon[127486]: rocksdb: Options.blob_compaction_readahead_size: 0 2023-07-14T19:22:11.331 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:22:11 smithi098 ceph-mon[127486]: rocksdb: Options.blob_file_starting_level: 0 2023-07-14T19:22:11.332 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:22:11 smithi098 ceph-mon[127486]: rocksdb: Options.experimental_mempurge_threshold: 0.000000 2023-07-14T19:22:11.332 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:22:11 smithi098 ceph-mon[127486]: rocksdb: [db/version_set.cc:5566] Recovered from manifest file:/var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000010 succeeded,manifest_file_number is 10, next_file_number is 12, last_sequence is 5, log_number is 5,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 5 2023-07-14T19:22:11.332 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:22:11 smithi098 ceph-mon[127486]: rocksdb: [db/version_set.cc:5581] Column family [default] (ID 0), log number is 5 2023-07-14T19:22:11.332 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:22:11 smithi098 ceph-mon[127486]: rocksdb: [db/db_impl/db_impl_open.cc:539] DB ID: 529082c9-36f5-402d-a4c9-9b8d203f779b 2023-07-14T19:22:11.332 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:22:11 smithi098 ceph-mon[127486]: rocksdb: EVENT_LOG_v1 {"time_micros": 1689362531196638, "job": 1, "event": "recovery_started", "wal_files": [9]} 2023-07-14T19:22:11.332 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:22:11 smithi098 ceph-mon[127486]: rocksdb: [db/db_impl/db_impl_open.cc:1043] Recovering log #9 mode 2 2023-07-14T19:22:11.332 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:22:11 smithi098 ceph-mon[127486]: rocksdb: EVENT_LOG_v1 {"time_micros": 1689362531197583, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 13, "file_size": 95018, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 8, "largest_seqno": 320, "table_properties": {"data_size": 93013, "index_size": 287, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 709, "raw_key_size": 13975, "raw_average_key_size": 50, "raw_value_size": 85449, "raw_average_value_size": 306, "num_data_blocks": 11, "num_entries": 279, "num_filter_entries": 279, "num_deletions": 3, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1689362531, "oldest_key_time": 0, "file_creation_time": 0, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "529082c9-36f5-402d-a4c9-9b8d203f779b", "db_session_id": "DRSGD8Y7LAPNTZGIAKVJ", "orig_file_number": 13, "seqno_to_time_mapping": "N/A"}} 2023-07-14T19:22:11.333 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:22:11 smithi098 ceph-mon[127486]: rocksdb: EVENT_LOG_v1 {"time_micros": 1689362531197663, "job": 1, "event": "recovery_finished"} 2023-07-14T19:22:11.333 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:22:11 smithi098 ceph-mon[127486]: rocksdb: [db/version_set.cc:5047] Creating manifest 15 2023-07-14T19:22:11.333 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:22:11 smithi098 systemd[1]: Started Ceph mon.a for 9adea648-227b-11ee-9b33-001a4aab830c. 2023-07-14T19:22:11.333 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:22:11 smithi098 ceph-mon[127486]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-a/store.db/000009.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2023-07-14T19:22:11.333 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:22:11 smithi098 ceph-mon[127486]: rocksdb: [db/db_impl/db_impl_open.cc:1987] SstFileManager instance 0x55f1209ac000 2023-07-14T19:22:11.333 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:22:11 smithi098 ceph-mon[127486]: rocksdb: DB pointer 0x55f120998000 2023-07-14T19:22:11.333 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:22:11 smithi098 ceph-mon[127486]: rocksdb: [db/db_impl/db_impl.cc:1109] ------- DUMPING STATS ------- 2023-07-14T19:22:11.333 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:22:11 smithi098 ceph-mon[127486]: rocksdb: [db/db_impl/db_impl.cc:1111] 2023-07-14T19:22:11.334 INFO:journalctl@ceph.mon.a.smithi098.stdout: ** DB Stats ** 2023-07-14T19:22:11.334 INFO:journalctl@ceph.mon.a.smithi098.stdout: Uptime(secs): 0.0 total, 0.0 interval 2023-07-14T19:22:11.334 INFO:journalctl@ceph.mon.a.smithi098.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2023-07-14T19:22:11.334 INFO:journalctl@ceph.mon.a.smithi098.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2023-07-14T19:22:11.334 INFO:journalctl@ceph.mon.a.smithi098.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2023-07-14T19:22:11.334 INFO:journalctl@ceph.mon.a.smithi098.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2023-07-14T19:22:11.334 INFO:journalctl@ceph.mon.a.smithi098.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2023-07-14T19:22:11.335 INFO:journalctl@ceph.mon.a.smithi098.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2023-07-14T19:22:11.335 INFO:journalctl@ceph.mon.a.smithi098.stdout: 2023-07-14T19:22:11.335 INFO:journalctl@ceph.mon.a.smithi098.stdout: ** Compaction Stats [default] ** 2023-07-14T19:22:11.335 INFO:journalctl@ceph.mon.a.smithi098.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2023-07-14T19:22:11.335 INFO:journalctl@ceph.mon.a.smithi098.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 2023-07-14T19:22:11.335 INFO:journalctl@ceph.mon.a.smithi098.stdout: L0 2/0 94.63 KB 0.5 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 154.6 0.00 0.00 1 0.001 0 0 0.0 0.0 2023-07-14T19:22:11.335 INFO:journalctl@ceph.mon.a.smithi098.stdout: Sum 2/0 94.63 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 154.6 0.00 0.00 1 0.001 0 0 0.0 0.0 2023-07-14T19:22:11.335 INFO:journalctl@ceph.mon.a.smithi098.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 154.6 0.00 0.00 1 0.001 0 0 0.0 0.0 2023-07-14T19:22:11.336 INFO:journalctl@ceph.mon.a.smithi098.stdout: 2023-07-14T19:22:11.336 INFO:journalctl@ceph.mon.a.smithi098.stdout: ** Compaction Stats [default] ** 2023-07-14T19:22:11.336 INFO:journalctl@ceph.mon.a.smithi098.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2023-07-14T19:22:11.336 INFO:journalctl@ceph.mon.a.smithi098.stdout: --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2023-07-14T19:22:11.336 INFO:journalctl@ceph.mon.a.smithi098.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 154.6 0.00 0.00 1 0.001 0 0 0.0 0.0 2023-07-14T19:22:11.336 INFO:journalctl@ceph.mon.a.smithi098.stdout: 2023-07-14T19:22:11.336 INFO:journalctl@ceph.mon.a.smithi098.stdout: Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 2023-07-14T19:22:11.337 INFO:journalctl@ceph.mon.a.smithi098.stdout: 2023-07-14T19:22:11.337 INFO:journalctl@ceph.mon.a.smithi098.stdout: Uptime(secs): 0.0 total, 0.0 interval 2023-07-14T19:22:11.337 INFO:journalctl@ceph.mon.a.smithi098.stdout: Flush(GB): cumulative 0.000, interval 0.000 2023-07-14T19:22:11.337 INFO:journalctl@ceph.mon.a.smithi098.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2023-07-14T19:22:11.337 INFO:journalctl@ceph.mon.a.smithi098.stdout: AddFile(Total Files): cumulative 0, interval 0 2023-07-14T19:22:11.337 INFO:journalctl@ceph.mon.a.smithi098.stdout: AddFile(L0 Files): cumulative 0, interval 0 2023-07-14T19:22:11.337 INFO:journalctl@ceph.mon.a.smithi098.stdout: AddFile(Keys): cumulative 0, interval 0 2023-07-14T19:22:11.337 INFO:journalctl@ceph.mon.a.smithi098.stdout: Cumulative compaction: 0.00 GB write, 29.97 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2023-07-14T19:22:11.338 INFO:journalctl@ceph.mon.a.smithi098.stdout: Interval compaction: 0.00 GB write, 29.97 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2023-07-14T19:22:11.338 INFO:journalctl@ceph.mon.a.smithi098.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2023-07-14T19:22:11.338 INFO:journalctl@ceph.mon.a.smithi098.stdout: Block cache BinnedLRUCache@0x55f1209131f0#7 capacity: 512.00 MB usage: 1.36 KB table_size: 0 occupancy: 18446744073709551615 collections: 1 last_copies: 0 last_secs: 1.1e-05 secs_since: 0 2023-07-14T19:22:11.338 INFO:journalctl@ceph.mon.a.smithi098.stdout: Block cache entry stats(count,size,portion): FilterBlock(2,0.89 KB,0.000169873%) IndexBlock(2,0.47 KB,8.9407e-05%) Misc(1,0.00 KB,0%) 2023-07-14T19:22:11.338 INFO:journalctl@ceph.mon.a.smithi098.stdout: 2023-07-14T19:22:11.338 INFO:journalctl@ceph.mon.a.smithi098.stdout: ** File Read Latency Histogram By Level [default] ** 2023-07-14T19:22:11.338 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:22:11 smithi098 ceph-mon[127486]: starting mon.a rank 0 at public addrs [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] at bind addrs [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon_data /var/lib/ceph/mon/ceph-a fsid 9adea648-227b-11ee-9b33-001a4aab830c 2023-07-14T19:22:11.339 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:22:11 smithi098 ceph-mon[127486]: mon.a@-1(???) e1 preinit fsid 9adea648-227b-11ee-9b33-001a4aab830c 2023-07-14T19:22:11.339 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:22:11 smithi098 ceph-mon[127486]: mon.a@-1(???).mds e1 new map 2023-07-14T19:22:11.339 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:22:11 smithi098 ceph-mon[127486]: mon.a@-1(???).mds e1 print_map 2023-07-14T19:22:11.339 INFO:journalctl@ceph.mon.a.smithi098.stdout: e1 2023-07-14T19:22:11.339 INFO:journalctl@ceph.mon.a.smithi098.stdout: enable_multiple, ever_enabled_multiple: 1,1 2023-07-14T19:22:11.339 INFO:journalctl@ceph.mon.a.smithi098.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2023-07-14T19:22:11.339 INFO:journalctl@ceph.mon.a.smithi098.stdout: legacy client fscid: -1 2023-07-14T19:22:11.339 INFO:journalctl@ceph.mon.a.smithi098.stdout: 2023-07-14T19:22:11.340 INFO:journalctl@ceph.mon.a.smithi098.stdout: No filesystems configured 2023-07-14T19:22:11.340 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:22:11 smithi098 ceph-mon[127486]: mon.a@-1(???).osd e1 crush map has features 3314932999778484224, adjusting msgr requires 2023-07-14T19:22:11.340 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:22:11 smithi098 ceph-mon[127486]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2023-07-14T19:22:11.340 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:22:11 smithi098 ceph-mon[127486]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2023-07-14T19:22:11.340 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:22:11 smithi098 ceph-mon[127486]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2023-07-14T19:22:11.340 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:22:11 smithi098 ceph-mon[127486]: mon.a@-1(???).paxosservice(auth 1..2) refresh upgraded, format 0 -> 3 2023-07-14T19:22:11.340 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:22:11 smithi098 ceph-mon[127486]: expand_channel_meta expand map: {default=false} 2023-07-14T19:22:11.340 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:22:11 smithi098 ceph-mon[127486]: expand_channel_meta from 'false' to 'false' 2023-07-14T19:22:11.341 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:22:11 smithi098 ceph-mon[127486]: expand_channel_meta expanded map: {default=false} 2023-07-14T19:22:11.341 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:22:11 smithi098 ceph-mon[127486]: expand_channel_meta expand map: {default=info} 2023-07-14T19:22:11.341 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:22:11 smithi098 ceph-mon[127486]: expand_channel_meta from 'info' to 'info' 2023-07-14T19:22:11.341 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:22:11 smithi098 ceph-mon[127486]: expand_channel_meta expanded map: {default=info} 2023-07-14T19:22:11.341 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:22:11 smithi098 ceph-mon[127486]: expand_channel_meta expand map: {default=daemon} 2023-07-14T19:22:11.341 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:22:11 smithi098 ceph-mon[127486]: expand_channel_meta from 'daemon' to 'daemon' 2023-07-14T19:22:11.341 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:22:11 smithi098 ceph-mon[127486]: expand_channel_meta expanded map: {default=daemon} 2023-07-14T19:22:11.342 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:22:11 smithi098 ceph-mon[127486]: expand_channel_meta expand map: {default=debug} 2023-07-14T19:22:11.342 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:22:11 smithi098 ceph-mon[127486]: expand_channel_meta from 'debug' to 'debug' 2023-07-14T19:22:11.342 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:22:11 smithi098 ceph-mon[127486]: expand_channel_meta expanded map: {default=debug} 2023-07-14T19:22:11.782 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:22:11 smithi098 ceph-mon[127486]: mon.a is new leader, mons a in quorum (ranks 0) 2023-07-14T19:22:11.782 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:22:11 smithi098 ceph-mon[127486]: monmap e1: 1 mons at {a=[v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0]} removed_ranks: {} 2023-07-14T19:22:11.782 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:22:11 smithi098 ceph-mon[127486]: fsmap 2023-07-14T19:22:11.782 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:22:11 smithi098 ceph-mon[127486]: osdmap e1: 0 total, 0 up, 0 in 2023-07-14T19:22:11.782 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:22:11 smithi098 ceph-mon[127486]: mgrmap e1: no daemons active 2023-07-14T19:22:12.250 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:11.731+0000 7fc7217b6700 1 Processor -- start 2023-07-14T19:22:12.250 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:11.731+0000 7fc7217b6700 1 -- start start 2023-07-14T19:22:12.250 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:11.731+0000 7fc7217b6700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc71c1061a0 0x7fc71c108580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:22:12.250 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:11.731+0000 7fc7217b6700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc71c079d20 con 0x7fc71c1061a0 2023-07-14T19:22:12.251 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:11.731+0000 7fc71affd700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc71c1061a0 0x7fc71c108580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:22:12.251 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:11.731+0000 7fc71affd700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc71c1061a0 0x7fc71c108580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:44324/0 (socket says 172.21.15.98:44324) 2023-07-14T19:22:12.251 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:11.731+0000 7fc71affd700 1 -- 172.21.15.98:0/155249043 learned_addr learned my addr 172.21.15.98:0/155249043 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-07-14T19:22:12.251 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:11.732+0000 7fc71affd700 1 -- 172.21.15.98:0/155249043 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc71c079e60 con 0x7fc71c1061a0 2023-07-14T19:22:12.251 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:11.732+0000 7fc71affd700 1 --2- 172.21.15.98:0/155249043 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc71c1061a0 0x7fc71c108580 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7fc70c0098a0 tx=0x7fc70c009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3548c633a4782af server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:22:12.251 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:11.732+0000 7fc719ffb700 1 -- 172.21.15.98:0/155249043 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fc70c004160 con 0x7fc71c1061a0 2023-07-14T19:22:12.251 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:11.732+0000 7fc719ffb700 1 -- 172.21.15.98:0/155249043 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fc70c0042c0 con 0x7fc71c1061a0 2023-07-14T19:22:12.251 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:11.732+0000 7fc719ffb700 1 -- 172.21.15.98:0/155249043 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fc70c011a00 con 0x7fc71c1061a0 2023-07-14T19:22:12.252 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:11.732+0000 7fc7217b6700 1 -- 172.21.15.98:0/155249043 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc71c1061a0 msgr2=0x7fc71c108580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:22:12.252 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:11.732+0000 7fc7217b6700 1 --2- 172.21.15.98:0/155249043 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc71c1061a0 0x7fc71c108580 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7fc70c0098a0 tx=0x7fc70c009bb0 comp rx=0 tx=0).stop 2023-07-14T19:22:12.252 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:11.733+0000 7fc7217b6700 1 -- 172.21.15.98:0/155249043 shutdown_connections 2023-07-14T19:22:12.252 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:11.733+0000 7fc7217b6700 1 --2- 172.21.15.98:0/155249043 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc71c1061a0 0x7fc71c108580 unknown :-1 s=CLOSED pgs=1 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:22:12.252 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:11.733+0000 7fc7217b6700 1 -- 172.21.15.98:0/155249043 >> 172.21.15.98:0/155249043 conn(0x7fc71c0ffba0 msgr2=0x7fc71c101fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:22:12.252 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:11.733+0000 7fc7217b6700 1 -- 172.21.15.98:0/155249043 shutdown_connections 2023-07-14T19:22:12.252 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:11.733+0000 7fc7217b6700 1 -- 172.21.15.98:0/155249043 wait complete. 2023-07-14T19:22:12.253 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:11.733+0000 7fc7217b6700 1 Processor -- start 2023-07-14T19:22:12.253 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:11.733+0000 7fc7217b6700 1 -- start start 2023-07-14T19:22:12.253 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:11.734+0000 7fc7217b6700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc71c1061a0 0x7fc71c199380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:22:12.253 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:11.734+0000 7fc7217b6700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc71c1998c0 con 0x7fc71c1061a0 2023-07-14T19:22:12.253 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:11.734+0000 7fc71affd700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc71c1061a0 0x7fc71c199380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:22:12.253 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:11.734+0000 7fc71affd700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc71c1061a0 0x7fc71c199380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:44326/0 (socket says 172.21.15.98:44326) 2023-07-14T19:22:12.253 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:11.734+0000 7fc71affd700 1 -- 172.21.15.98:0/1337568008 learned_addr learned my addr 172.21.15.98:0/1337568008 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-07-14T19:22:12.254 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:11.734+0000 7fc71affd700 1 -- 172.21.15.98:0/1337568008 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc70c009550 con 0x7fc71c1061a0 2023-07-14T19:22:12.254 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:11.734+0000 7fc71affd700 1 --2- 172.21.15.98:0/1337568008 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc71c1061a0 0x7fc71c199380 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7fc70c009fb0 tx=0x7fc70c011ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:22:12.254 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:11.734+0000 7fc703fff700 1 -- 172.21.15.98:0/1337568008 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fc70c003930 con 0x7fc71c1061a0 2023-07-14T19:22:12.254 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:11.734+0000 7fc7217b6700 1 -- 172.21.15.98:0/1337568008 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc71c199b20 con 0x7fc71c1061a0 2023-07-14T19:22:12.254 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:11.734+0000 7fc703fff700 1 -- 172.21.15.98:0/1337568008 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fc70c003a90 con 0x7fc71c1061a0 2023-07-14T19:22:12.254 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:11.734+0000 7fc703fff700 1 -- 172.21.15.98:0/1337568008 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fc70c0195e0 con 0x7fc71c1061a0 2023-07-14T19:22:12.254 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:11.734+0000 7fc7217b6700 1 -- 172.21.15.98:0/1337568008 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc71c199fa0 con 0x7fc71c1061a0 2023-07-14T19:22:12.254 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:11.735+0000 7fc703fff700 1 -- 172.21.15.98:0/1337568008 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7fc70c0198a0 con 0x7fc71c1061a0 2023-07-14T19:22:12.255 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:11.735+0000 7fc703fff700 1 -- 172.21.15.98:0/1337568008 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7fc70c02da10 con 0x7fc71c1061a0 2023-07-14T19:22:12.255 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:11.735+0000 7fc7217b6700 1 -- 172.21.15.98:0/1337568008 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc71c1096c0 con 0x7fc71c1061a0 2023-07-14T19:22:12.255 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:11.738+0000 7fc703fff700 1 -- 172.21.15.98:0/1337568008 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72446 (secure 0 0 0) 0x7fc70c003c00 con 0x7fc71c1061a0 2023-07-14T19:22:12.255 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:11.804+0000 7fc7217b6700 1 -- 172.21.15.98:0/1337568008 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command([{prefix=config set, name=public_network}] v 0) v1 -- 0x7fc71c1098b0 con 0x7fc71c1061a0 2023-07-14T19:22:12.255 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:11.807+0000 7fc703fff700 1 -- 172.21.15.98:0/1337568008 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{prefix=config set, name=public_network}]=0 v3)=0 v3) v1 ==== 124+0+0 (secure 0 0 0) 0x7fc70c02d440 con 0x7fc71c1061a0 2023-07-14T19:22:12.255 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:11.807+0000 7fc7217b6700 1 -- 172.21.15.98:0/1337568008 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc71c1061a0 msgr2=0x7fc71c199380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:22:12.255 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:11.807+0000 7fc7217b6700 1 --2- 172.21.15.98:0/1337568008 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc71c1061a0 0x7fc71c199380 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7fc70c009fb0 tx=0x7fc70c011ad0 comp rx=0 tx=0).stop 2023-07-14T19:22:12.256 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:11.807+0000 7fc7217b6700 1 -- 172.21.15.98:0/1337568008 shutdown_connections 2023-07-14T19:22:12.256 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:11.807+0000 7fc7217b6700 1 --2- 172.21.15.98:0/1337568008 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc71c1061a0 0x7fc71c199380 unknown :-1 s=CLOSED pgs=2 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:22:12.256 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:11.807+0000 7fc7217b6700 1 -- 172.21.15.98:0/1337568008 >> 172.21.15.98:0/1337568008 conn(0x7fc71c0ffba0 msgr2=0x7fc71c1012f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:22:12.256 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:11.808+0000 7fc7217b6700 1 -- 172.21.15.98:0/1337568008 shutdown_connections 2023-07-14T19:22:12.256 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:11.808+0000 7fc7217b6700 1 -- 172.21.15.98:0/1337568008 wait complete. 2023-07-14T19:22:12.256 INFO:teuthology.orchestra.run.smithi098.stdout:Wrote config to /etc/ceph/ceph.conf 2023-07-14T19:22:12.486 INFO:teuthology.orchestra.run.smithi098.stdout:Wrote keyring to /etc/ceph/ceph.client.admin.keyring 2023-07-14T19:22:12.486 INFO:teuthology.orchestra.run.smithi098.stdout:Creating mgr... 2023-07-14T19:22:12.487 INFO:teuthology.orchestra.run.smithi098.stdout:Verifying port 9283 ... 2023-07-14T19:22:12.487 INFO:teuthology.orchestra.run.smithi098.stdout:Verifying port 8765 ... 2023-07-14T19:22:12.954 INFO:teuthology.orchestra.run.smithi098.stdout:Non-zero exit code 1 from systemctl reset-failed ceph-9adea648-227b-11ee-9b33-001a4aab830c@mgr.x 2023-07-14T19:22:12.957 INFO:teuthology.orchestra.run.smithi098.stdout:systemctl: stderr Failed to reset failed state of unit ceph-9adea648-227b-11ee-9b33-001a4aab830c@mgr.x.service: Unit ceph-9adea648-227b-11ee-9b33-001a4aab830c@mgr.x.service not loaded. 2023-07-14T19:22:13.038 INFO:teuthology.orchestra.run.smithi098.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph-9adea648-227b-11ee-9b33-001a4aab830c.target.wants/ceph-9adea648-227b-11ee-9b33-001a4aab830c@mgr.x.service → /etc/systemd/system/ceph-9adea648-227b-11ee-9b33-001a4aab830c@.service. 2023-07-14T19:22:13.281 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:22:12 smithi098 ceph-mon[127486]: from='client.? 172.21.15.98:0/1337568008' entity='client.admin' 2023-07-14T19:22:13.282 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Jul 14 19:22:13 smithi098 systemd[1]: Starting Ceph mgr.x for 9adea648-227b-11ee-9b33-001a4aab830c... 2023-07-14T19:22:14.281 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Jul 14 19:22:13 smithi098 podman[127718]: 2023-07-14T19:22:14.844 INFO:teuthology.orchestra.run.smithi098.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2023-07-14T19:22:14.844 INFO:teuthology.orchestra.run.smithi098.stdout:systemctl: stdout disabled 2023-07-14T19:22:14.851 INFO:teuthology.orchestra.run.smithi098.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2023-07-14T19:22:14.852 INFO:teuthology.orchestra.run.smithi098.stdout:systemctl: stdout inactive 2023-07-14T19:22:14.852 INFO:teuthology.orchestra.run.smithi098.stdout:firewalld.service is not enabled 2023-07-14T19:22:14.852 INFO:teuthology.orchestra.run.smithi098.stdout:Not possible to enable service . firewalld.service is not available 2023-07-14T19:22:14.860 INFO:teuthology.orchestra.run.smithi098.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2023-07-14T19:22:14.860 INFO:teuthology.orchestra.run.smithi098.stdout:systemctl: stdout disabled 2023-07-14T19:22:14.867 INFO:teuthology.orchestra.run.smithi098.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2023-07-14T19:22:14.868 INFO:teuthology.orchestra.run.smithi098.stdout:systemctl: stdout inactive 2023-07-14T19:22:14.868 INFO:teuthology.orchestra.run.smithi098.stdout:firewalld.service is not enabled 2023-07-14T19:22:14.868 INFO:teuthology.orchestra.run.smithi098.stdout:Not possible to open ports <[9283, 8765]>. firewalld.service is not available 2023-07-14T19:22:14.868 INFO:teuthology.orchestra.run.smithi098.stdout:Waiting for mgr to start... 2023-07-14T19:22:14.868 INFO:teuthology.orchestra.run.smithi098.stdout:Waiting for mgr... 2023-07-14T19:22:15.065 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Jul 14 19:22:14 smithi098 bash[127718]: b3f6c0ebf294e837c64b856e62eda73a57de01c6c8ddae30bf68f8a52b114775 2023-07-14T19:22:15.066 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Jul 14 19:22:14 smithi098 systemd[1]: Started Ceph mgr.x for 9adea648-227b-11ee-9b33-001a4aab830c. 2023-07-14T19:22:15.066 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Jul 14 19:22:14 smithi098 ceph-9adea648-227b-11ee-9b33-001a4aab830c-mgr-x[127734]: 2023-07-14T19:22:14.916+0000 7f73ba7c8200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-07-14T19:22:15.066 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Jul 14 19:22:14 smithi098 ceph-9adea648-227b-11ee-9b33-001a4aab830c-mgr-x[127734]: 2023-07-14T19:22:14.996+0000 7f73ba7c8200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-07-14T19:22:15.366 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Jul 14 19:22:15 smithi098 ceph-9adea648-227b-11ee-9b33-001a4aab830c-mgr-x[127734]: 2023-07-14T19:22:15.090+0000 7f73ba7c8200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-07-14T19:22:15.366 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Jul 14 19:22:15 smithi098 ceph-9adea648-227b-11ee-9b33-001a4aab830c-mgr-x[127734]: 2023-07-14T19:22:15.186+0000 7f73ba7c8200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-07-14T19:22:15.686 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Jul 14 19:22:15 smithi098 ceph-9adea648-227b-11ee-9b33-001a4aab830c-mgr-x[127734]: 2023-07-14T19:22:15.365+0000 7f73ba7c8200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-07-14T19:22:15.687 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:22:15 smithi098 ceph-mon[127486]: from='client.? 172.21.15.98:0/1689859755' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2023-07-14T19:22:15.817 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout 2023-07-14T19:22:15.817 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout { 2023-07-14T19:22:15.818 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "fsid": "9adea648-227b-11ee-9b33-001a4aab830c", 2023-07-14T19:22:15.818 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "health": { 2023-07-14T19:22:15.818 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2023-07-14T19:22:15.818 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "checks": {}, 2023-07-14T19:22:15.818 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "mutes": [] 2023-07-14T19:22:15.818 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout }, 2023-07-14T19:22:15.818 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2023-07-14T19:22:15.819 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "quorum": [ 2023-07-14T19:22:15.819 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout 0 2023-07-14T19:22:15.819 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout ], 2023-07-14T19:22:15.819 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2023-07-14T19:22:15.819 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "a" 2023-07-14T19:22:15.819 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout ], 2023-07-14T19:22:15.819 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "quorum_age": 4, 2023-07-14T19:22:15.819 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "monmap": { 2023-07-14T19:22:15.820 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "epoch": 1, 2023-07-14T19:22:15.820 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "reef", 2023-07-14T19:22:15.820 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "num_mons": 1 2023-07-14T19:22:15.820 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout }, 2023-07-14T19:22:15.820 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "osdmap": { 2023-07-14T19:22:15.820 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "epoch": 1, 2023-07-14T19:22:15.820 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2023-07-14T19:22:15.821 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2023-07-14T19:22:15.821 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2023-07-14T19:22:15.827 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2023-07-14T19:22:15.827 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2023-07-14T19:22:15.827 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2023-07-14T19:22:15.828 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout }, 2023-07-14T19:22:15.828 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "pgmap": { 2023-07-14T19:22:15.828 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2023-07-14T19:22:15.829 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2023-07-14T19:22:15.829 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2023-07-14T19:22:15.829 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2023-07-14T19:22:15.829 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2023-07-14T19:22:15.830 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2023-07-14T19:22:15.830 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2023-07-14T19:22:15.830 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2023-07-14T19:22:15.830 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout }, 2023-07-14T19:22:15.831 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "fsmap": { 2023-07-14T19:22:15.831 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "epoch": 1, 2023-07-14T19:22:15.831 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "by_rank": [], 2023-07-14T19:22:15.832 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "up:standby": 0 2023-07-14T19:22:15.832 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout }, 2023-07-14T19:22:15.832 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "mgrmap": { 2023-07-14T19:22:15.833 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "available": false, 2023-07-14T19:22:15.833 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2023-07-14T19:22:15.833 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "modules": [ 2023-07-14T19:22:15.834 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "iostat", 2023-07-14T19:22:15.834 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "nfs", 2023-07-14T19:22:15.834 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "restful" 2023-07-14T19:22:15.834 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout ], 2023-07-14T19:22:15.835 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "services": {} 2023-07-14T19:22:15.835 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout }, 2023-07-14T19:22:15.835 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "servicemap": { 2023-07-14T19:22:15.835 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "epoch": 1, 2023-07-14T19:22:15.835 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "modified": "2023-07-14T19:22:07.179303+0000", 2023-07-14T19:22:15.836 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "services": {} 2023-07-14T19:22:15.836 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout }, 2023-07-14T19:22:15.836 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "progress_events": {} 2023-07-14T19:22:15.836 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout } 2023-07-14T19:22:15.837 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:15.357+0000 7f2c433e4700 1 Processor -- start 2023-07-14T19:22:15.837 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:15.357+0000 7f2c433e4700 1 -- start start 2023-07-14T19:22:15.837 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:15.357+0000 7f2c433e4700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2c3c104570 0x7f2c3c104980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:22:15.837 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:15.357+0000 7f2c433e4700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2c3c104ec0 con 0x7f2c3c104570 2023-07-14T19:22:15.838 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:15.358+0000 7f2c41180700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2c3c104570 0x7f2c3c104980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:22:15.838 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:15.358+0000 7f2c41180700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2c3c104570 0x7f2c3c104980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:44332/0 (socket says 172.21.15.98:44332) 2023-07-14T19:22:15.838 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:15.358+0000 7f2c41180700 1 -- 172.21.15.98:0/913198274 learned_addr learned my addr 172.21.15.98:0/913198274 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-07-14T19:22:15.838 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:15.358+0000 7f2c41180700 1 -- 172.21.15.98:0/913198274 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2c3c105000 con 0x7f2c3c104570 2023-07-14T19:22:15.839 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:15.358+0000 7f2c41180700 1 --2- 172.21.15.98:0/913198274 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2c3c104570 0x7f2c3c104980 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7f2c380098a0 tx=0x7f2c38009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ee7abbf3bdabb5a4 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:22:15.839 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:15.358+0000 7f2c33fff700 1 -- 172.21.15.98:0/913198274 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f2c38004160 con 0x7f2c3c104570 2023-07-14T19:22:15.839 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:15.358+0000 7f2c33fff700 1 -- 172.21.15.98:0/913198274 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f2c380042c0 con 0x7f2c3c104570 2023-07-14T19:22:15.839 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:15.358+0000 7f2c33fff700 1 -- 172.21.15.98:0/913198274 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f2c38011a00 con 0x7f2c3c104570 2023-07-14T19:22:15.839 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:15.359+0000 7f2c433e4700 1 -- 172.21.15.98:0/913198274 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2c3c104570 msgr2=0x7f2c3c104980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:22:15.839 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:15.359+0000 7f2c433e4700 1 --2- 172.21.15.98:0/913198274 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2c3c104570 0x7f2c3c104980 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7f2c380098a0 tx=0x7f2c38009bb0 comp rx=0 tx=0).stop 2023-07-14T19:22:15.840 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:15.359+0000 7f2c433e4700 1 -- 172.21.15.98:0/913198274 shutdown_connections 2023-07-14T19:22:15.840 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:15.359+0000 7f2c433e4700 1 --2- 172.21.15.98:0/913198274 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2c3c104570 0x7f2c3c104980 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:22:15.840 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:15.359+0000 7f2c433e4700 1 -- 172.21.15.98:0/913198274 >> 172.21.15.98:0/913198274 conn(0x7f2c3c0ffbc0 msgr2=0x7f2c3c101ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:22:15.840 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:15.359+0000 7f2c433e4700 1 -- 172.21.15.98:0/913198274 shutdown_connections 2023-07-14T19:22:15.840 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:15.359+0000 7f2c433e4700 1 -- 172.21.15.98:0/913198274 wait complete. 2023-07-14T19:22:15.841 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:15.359+0000 7f2c433e4700 1 Processor -- start 2023-07-14T19:22:15.841 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:15.360+0000 7f2c433e4700 1 -- start start 2023-07-14T19:22:15.841 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:15.360+0000 7f2c433e4700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2c3c104570 0x7f2c3c10a5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:22:15.841 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:15.360+0000 7f2c433e4700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2c3c10aaf0 con 0x7f2c3c104570 2023-07-14T19:22:15.842 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:15.360+0000 7f2c41180700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2c3c104570 0x7f2c3c10a5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:22:15.842 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:15.360+0000 7f2c41180700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2c3c104570 0x7f2c3c10a5b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:44334/0 (socket says 172.21.15.98:44334) 2023-07-14T19:22:15.842 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:15.360+0000 7f2c41180700 1 -- 172.21.15.98:0/1689859755 learned_addr learned my addr 172.21.15.98:0/1689859755 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-07-14T19:22:15.842 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:15.360+0000 7f2c41180700 1 -- 172.21.15.98:0/1689859755 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2c38009550 con 0x7f2c3c104570 2023-07-14T19:22:15.842 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:15.360+0000 7f2c41180700 1 --2- 172.21.15.98:0/1689859755 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2c3c104570 0x7f2c3c10a5b0 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f2c38009130 tx=0x7f2c38011ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:22:15.842 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:15.360+0000 7f2c327fc700 1 -- 172.21.15.98:0/1689859755 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f2c38003b70 con 0x7f2c3c104570 2023-07-14T19:22:15.842 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:15.360+0000 7f2c327fc700 1 -- 172.21.15.98:0/1689859755 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f2c38003cd0 con 0x7f2c3c104570 2023-07-14T19:22:15.843 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:15.360+0000 7f2c327fc700 1 -- 172.21.15.98:0/1689859755 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f2c380197d0 con 0x7f2c3c104570 2023-07-14T19:22:15.843 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:15.360+0000 7f2c433e4700 1 -- 172.21.15.98:0/1689859755 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2c3c108c00 con 0x7f2c3c104570 2023-07-14T19:22:15.843 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:15.361+0000 7f2c433e4700 1 -- 172.21.15.98:0/1689859755 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2c3c109020 con 0x7f2c3c104570 2023-07-14T19:22:15.843 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:15.361+0000 7f2c327fc700 1 -- 172.21.15.98:0/1689859755 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7f2c380199c0 con 0x7f2c3c104570 2023-07-14T19:22:15.843 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:15.361+0000 7f2c327fc700 1 -- 172.21.15.98:0/1689859755 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7f2c3800b720 con 0x7f2c3c104570 2023-07-14T19:22:15.843 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:15.361+0000 7f2c433e4700 1 -- 172.21.15.98:0/1689859755 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2c3c061c70 con 0x7f2c3c104570 2023-07-14T19:22:15.843 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:15.364+0000 7f2c327fc700 1 -- 172.21.15.98:0/1689859755 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72446 (secure 0 0 0) 0x7f2c3802a060 con 0x7f2c3c104570 2023-07-14T19:22:15.843 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:15.435+0000 7f2c433e4700 1 -- 172.21.15.98:0/1689859755 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) v1 -- 0x7f2c3c109a30 con 0x7f2c3c104570 2023-07-14T19:22:15.844 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:15.435+0000 7f2c327fc700 1 -- 172.21.15.98:0/1689859755 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) v1 ==== 79+0+1238 (secure 0 0 0) 0x7f2c38019e00 con 0x7f2c3c104570 2023-07-14T19:22:15.844 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:15.436+0000 7f2c433e4700 1 -- 172.21.15.98:0/1689859755 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2c3c104570 msgr2=0x7f2c3c10a5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:22:15.844 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:15.436+0000 7f2c433e4700 1 --2- 172.21.15.98:0/1689859755 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2c3c104570 0x7f2c3c10a5b0 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f2c38009130 tx=0x7f2c38011ad0 comp rx=0 tx=0).stop 2023-07-14T19:22:15.844 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:15.436+0000 7f2c433e4700 1 -- 172.21.15.98:0/1689859755 shutdown_connections 2023-07-14T19:22:15.844 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:15.436+0000 7f2c433e4700 1 --2- 172.21.15.98:0/1689859755 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2c3c104570 0x7f2c3c10a5b0 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:22:15.844 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:15.436+0000 7f2c433e4700 1 -- 172.21.15.98:0/1689859755 >> 172.21.15.98:0/1689859755 conn(0x7f2c3c0ffbc0 msgr2=0x7f2c3c100870 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:22:15.844 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:15.436+0000 7f2c433e4700 1 -- 172.21.15.98:0/1689859755 shutdown_connections 2023-07-14T19:22:15.845 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:15.436+0000 7f2c433e4700 1 -- 172.21.15.98:0/1689859755 wait complete. 2023-07-14T19:22:15.845 INFO:teuthology.orchestra.run.smithi098.stdout:mgr not available, waiting (1/15)... 2023-07-14T19:22:15.998 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Jul 14 19:22:15 smithi098 ceph-9adea648-227b-11ee-9b33-001a4aab830c-mgr-x[127734]: 2023-07-14T19:22:15.685+0000 7f73ba7c8200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-07-14T19:22:15.999 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Jul 14 19:22:15 smithi098 ceph-9adea648-227b-11ee-9b33-001a4aab830c-mgr-x[127734]: 2023-07-14T19:22:15.774+0000 7f73ba7c8200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-07-14T19:22:15.999 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Jul 14 19:22:15 smithi098 ceph-9adea648-227b-11ee-9b33-001a4aab830c-mgr-x[127734]: 2023-07-14T19:22:15.905+0000 7f73ba7c8200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-07-14T19:22:16.281 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Jul 14 19:22:15 smithi098 ceph-9adea648-227b-11ee-9b33-001a4aab830c-mgr-x[127734]: 2023-07-14T19:22:15.997+0000 7f73ba7c8200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-07-14T19:22:16.781 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Jul 14 19:22:16 smithi098 ceph-9adea648-227b-11ee-9b33-001a4aab830c-mgr-x[127734]: 2023-07-14T19:22:16.412+0000 7f73ba7c8200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-07-14T19:22:17.281 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Jul 14 19:22:16 smithi098 ceph-9adea648-227b-11ee-9b33-001a4aab830c-mgr-x[127734]: 2023-07-14T19:22:16.975+0000 7f73ba7c8200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2023-07-14T19:22:17.282 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Jul 14 19:22:17 smithi098 ceph-9adea648-227b-11ee-9b33-001a4aab830c-mgr-x[127734]: 2023-07-14T19:22:17.141+0000 7f73ba7c8200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-07-14T19:22:18.167 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Jul 14 19:22:17 smithi098 ceph-9adea648-227b-11ee-9b33-001a4aab830c-mgr-x[127734]: 2023-07-14T19:22:17.898+0000 7f73ba7c8200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-07-14T19:22:18.167 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Jul 14 19:22:17 smithi098 ceph-9adea648-227b-11ee-9b33-001a4aab830c-mgr-x[127734]: 2023-07-14T19:22:17.989+0000 7f73ba7c8200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-07-14T19:22:18.433 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Jul 14 19:22:18 smithi098 ceph-9adea648-227b-11ee-9b33-001a4aab830c-mgr-x[127734]: 2023-07-14T19:22:18.166+0000 7f73ba7c8200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-07-14T19:22:18.689 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:22:18 smithi098 ceph-mon[127486]: from='client.? 172.21.15.98:0/2935803605' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2023-07-14T19:22:18.690 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Jul 14 19:22:18 smithi098 ceph-9adea648-227b-11ee-9b33-001a4aab830c-mgr-x[127734]: 2023-07-14T19:22:18.453+0000 7f73ba7c8200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2023-07-14T19:22:18.866 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout 2023-07-14T19:22:18.873 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout { 2023-07-14T19:22:18.873 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "fsid": "9adea648-227b-11ee-9b33-001a4aab830c", 2023-07-14T19:22:18.873 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "health": { 2023-07-14T19:22:18.873 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2023-07-14T19:22:18.873 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "checks": {}, 2023-07-14T19:22:18.874 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "mutes": [] 2023-07-14T19:22:18.874 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout }, 2023-07-14T19:22:18.874 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2023-07-14T19:22:18.874 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "quorum": [ 2023-07-14T19:22:18.874 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout 0 2023-07-14T19:22:18.875 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout ], 2023-07-14T19:22:18.875 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2023-07-14T19:22:18.875 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "a" 2023-07-14T19:22:18.875 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout ], 2023-07-14T19:22:18.875 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "quorum_age": 7, 2023-07-14T19:22:18.875 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "monmap": { 2023-07-14T19:22:18.875 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "epoch": 1, 2023-07-14T19:22:18.875 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "reef", 2023-07-14T19:22:18.876 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "num_mons": 1 2023-07-14T19:22:18.876 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout }, 2023-07-14T19:22:18.876 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "osdmap": { 2023-07-14T19:22:18.876 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "epoch": 1, 2023-07-14T19:22:18.876 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2023-07-14T19:22:18.876 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2023-07-14T19:22:18.876 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2023-07-14T19:22:18.877 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2023-07-14T19:22:18.877 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2023-07-14T19:22:18.877 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2023-07-14T19:22:18.877 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout }, 2023-07-14T19:22:18.877 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "pgmap": { 2023-07-14T19:22:18.877 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2023-07-14T19:22:18.877 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2023-07-14T19:22:18.877 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2023-07-14T19:22:18.878 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2023-07-14T19:22:18.878 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2023-07-14T19:22:18.878 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2023-07-14T19:22:18.878 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2023-07-14T19:22:18.878 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2023-07-14T19:22:18.878 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout }, 2023-07-14T19:22:18.878 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "fsmap": { 2023-07-14T19:22:18.878 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "epoch": 1, 2023-07-14T19:22:18.879 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "by_rank": [], 2023-07-14T19:22:18.879 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "up:standby": 0 2023-07-14T19:22:18.879 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout }, 2023-07-14T19:22:18.879 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "mgrmap": { 2023-07-14T19:22:18.879 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "available": false, 2023-07-14T19:22:18.879 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2023-07-14T19:22:18.879 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "modules": [ 2023-07-14T19:22:18.879 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "iostat", 2023-07-14T19:22:18.880 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "nfs", 2023-07-14T19:22:18.880 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "restful" 2023-07-14T19:22:18.880 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout ], 2023-07-14T19:22:18.880 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "services": {} 2023-07-14T19:22:18.880 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout }, 2023-07-14T19:22:18.880 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "servicemap": { 2023-07-14T19:22:18.880 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "epoch": 1, 2023-07-14T19:22:18.881 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "modified": "2023-07-14T19:22:07.179303+0000", 2023-07-14T19:22:18.881 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "services": {} 2023-07-14T19:22:18.881 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout }, 2023-07-14T19:22:18.881 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "progress_events": {} 2023-07-14T19:22:18.881 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout } 2023-07-14T19:22:18.881 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:18.324+0000 7fd1bf9ab700 1 Processor -- start 2023-07-14T19:22:18.881 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:18.324+0000 7fd1bf9ab700 1 -- start start 2023-07-14T19:22:18.881 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:18.325+0000 7fd1bf9ab700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd1b8105660 0x7fd1b8105a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:22:18.882 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:18.325+0000 7fd1bf9ab700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd1b8105fb0 con 0x7fd1b8105660 2023-07-14T19:22:18.882 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:18.325+0000 7fd1bd747700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd1b8105660 0x7fd1b8105a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:22:18.882 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:18.325+0000 7fd1bd747700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd1b8105660 0x7fd1b8105a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:44336/0 (socket says 172.21.15.98:44336) 2023-07-14T19:22:18.882 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:18.325+0000 7fd1bd747700 1 -- 172.21.15.98:0/2499153647 learned_addr learned my addr 172.21.15.98:0/2499153647 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-07-14T19:22:18.882 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:18.325+0000 7fd1bd747700 1 -- 172.21.15.98:0/2499153647 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd1b81060f0 con 0x7fd1b8105660 2023-07-14T19:22:18.882 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:18.325+0000 7fd1bd747700 1 --2- 172.21.15.98:0/2499153647 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd1b8105660 0x7fd1b8105a70 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7fd1b40098a0 tx=0x7fd1b4009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d36a472c7f26fcd server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:22:18.882 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:18.325+0000 7fd1affff700 1 -- 172.21.15.98:0/2499153647 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fd1b4004160 con 0x7fd1b8105660 2023-07-14T19:22:18.883 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:18.326+0000 7fd1affff700 1 -- 172.21.15.98:0/2499153647 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fd1b40042c0 con 0x7fd1b8105660 2023-07-14T19:22:18.883 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:18.326+0000 7fd1affff700 1 -- 172.21.15.98:0/2499153647 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fd1b4011a00 con 0x7fd1b8105660 2023-07-14T19:22:18.883 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:18.326+0000 7fd1bf9ab700 1 -- 172.21.15.98:0/2499153647 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd1b8105660 msgr2=0x7fd1b8105a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:22:18.883 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:18.326+0000 7fd1bf9ab700 1 --2- 172.21.15.98:0/2499153647 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd1b8105660 0x7fd1b8105a70 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7fd1b40098a0 tx=0x7fd1b4009bb0 comp rx=0 tx=0).stop 2023-07-14T19:22:18.883 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:18.326+0000 7fd1bf9ab700 1 -- 172.21.15.98:0/2499153647 shutdown_connections 2023-07-14T19:22:18.883 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:18.326+0000 7fd1bf9ab700 1 --2- 172.21.15.98:0/2499153647 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd1b8105660 0x7fd1b8105a70 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:22:18.883 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:18.326+0000 7fd1bf9ab700 1 -- 172.21.15.98:0/2499153647 >> 172.21.15.98:0/2499153647 conn(0x7fd1b8100cb0 msgr2=0x7fd1b81030e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:22:18.883 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:18.326+0000 7fd1bf9ab700 1 -- 172.21.15.98:0/2499153647 shutdown_connections 2023-07-14T19:22:18.884 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:18.326+0000 7fd1bf9ab700 1 -- 172.21.15.98:0/2499153647 wait complete. 2023-07-14T19:22:18.884 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:18.327+0000 7fd1bf9ab700 1 Processor -- start 2023-07-14T19:22:18.884 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:18.327+0000 7fd1bf9ab700 1 -- start start 2023-07-14T19:22:18.884 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:18.327+0000 7fd1bf9ab700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd1b8105660 0x7fd1b8199480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:22:18.884 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:18.327+0000 7fd1bf9ab700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd1b81999c0 con 0x7fd1b8105660 2023-07-14T19:22:18.884 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:18.328+0000 7fd1bd747700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd1b8105660 0x7fd1b8199480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:22:18.884 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:18.328+0000 7fd1bd747700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd1b8105660 0x7fd1b8199480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:44338/0 (socket says 172.21.15.98:44338) 2023-07-14T19:22:18.885 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:18.328+0000 7fd1bd747700 1 -- 172.21.15.98:0/2935803605 learned_addr learned my addr 172.21.15.98:0/2935803605 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-07-14T19:22:18.885 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:18.328+0000 7fd1bd747700 1 -- 172.21.15.98:0/2935803605 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd1b4009550 con 0x7fd1b8105660 2023-07-14T19:22:18.885 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:18.328+0000 7fd1bd747700 1 --2- 172.21.15.98:0/2935803605 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd1b8105660 0x7fd1b8199480 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7fd1b4009130 tx=0x7fd1b4011ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:22:18.885 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:18.328+0000 7fd1ae7fc700 1 -- 172.21.15.98:0/2935803605 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fd1b4003b70 con 0x7fd1b8105660 2023-07-14T19:22:18.885 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:18.328+0000 7fd1bf9ab700 1 -- 172.21.15.98:0/2935803605 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd1b8199bc0 con 0x7fd1b8105660 2023-07-14T19:22:18.885 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:18.328+0000 7fd1ae7fc700 1 -- 172.21.15.98:0/2935803605 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fd1b4003cd0 con 0x7fd1b8105660 2023-07-14T19:22:18.885 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:18.328+0000 7fd1ae7fc700 1 -- 172.21.15.98:0/2935803605 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fd1b40197d0 con 0x7fd1b8105660 2023-07-14T19:22:18.885 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:18.328+0000 7fd1bf9ab700 1 -- 172.21.15.98:0/2935803605 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd1b8199fe0 con 0x7fd1b8105660 2023-07-14T19:22:18.886 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:18.329+0000 7fd1ae7fc700 1 -- 172.21.15.98:0/2935803605 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7fd1b4019a10 con 0x7fd1b8105660 2023-07-14T19:22:18.886 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:18.329+0000 7fd1ae7fc700 1 -- 172.21.15.98:0/2935803605 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7fd1b400b720 con 0x7fd1b8105660 2023-07-14T19:22:18.886 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:18.329+0000 7fd1bf9ab700 1 -- 172.21.15.98:0/2935803605 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd1b8193430 con 0x7fd1b8105660 2023-07-14T19:22:18.886 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:18.331+0000 7fd1ae7fc700 1 -- 172.21.15.98:0/2935803605 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72446 (secure 0 0 0) 0x7fd1b402a060 con 0x7fd1b8105660 2023-07-14T19:22:18.886 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:18.403+0000 7fd1bf9ab700 1 -- 172.21.15.98:0/2935803605 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) v1 -- 0x7fd1b802cd00 con 0x7fd1b8105660 2023-07-14T19:22:18.886 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:18.404+0000 7fd1ae7fc700 1 -- 172.21.15.98:0/2935803605 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) v1 ==== 79+0+1238 (secure 0 0 0) 0x7fd1b4019e50 con 0x7fd1b8105660 2023-07-14T19:22:18.886 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:18.405+0000 7fd1bf9ab700 1 -- 172.21.15.98:0/2935803605 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd1b8105660 msgr2=0x7fd1b8199480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:22:18.887 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:18.405+0000 7fd1bf9ab700 1 --2- 172.21.15.98:0/2935803605 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd1b8105660 0x7fd1b8199480 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7fd1b4009130 tx=0x7fd1b4011ad0 comp rx=0 tx=0).stop 2023-07-14T19:22:18.887 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:18.405+0000 7fd1bf9ab700 1 -- 172.21.15.98:0/2935803605 shutdown_connections 2023-07-14T19:22:18.887 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:18.405+0000 7fd1bf9ab700 1 --2- 172.21.15.98:0/2935803605 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd1b8105660 0x7fd1b8199480 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:22:18.887 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:18.405+0000 7fd1bf9ab700 1 -- 172.21.15.98:0/2935803605 >> 172.21.15.98:0/2935803605 conn(0x7fd1b8100cb0 msgr2=0x7fd1b8102410 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:22:18.887 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:18.405+0000 7fd1bf9ab700 1 -- 172.21.15.98:0/2935803605 shutdown_connections 2023-07-14T19:22:18.887 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:18.405+0000 7fd1bf9ab700 1 -- 172.21.15.98:0/2935803605 wait complete. 2023-07-14T19:22:18.887 INFO:teuthology.orchestra.run.smithi098.stdout:mgr not available, waiting (2/15)... 2023-07-14T19:22:19.031 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Jul 14 19:22:18 smithi098 ceph-9adea648-227b-11ee-9b33-001a4aab830c-mgr-x[127734]: 2023-07-14T19:22:18.688+0000 7f73ba7c8200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-07-14T19:22:19.032 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Jul 14 19:22:18 smithi098 ceph-9adea648-227b-11ee-9b33-001a4aab830c-mgr-x[127734]: 2023-07-14T19:22:18.839+0000 7f73ba7c8200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-07-14T19:22:19.531 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Jul 14 19:22:19 smithi098 ceph-9adea648-227b-11ee-9b33-001a4aab830c-mgr-x[127734]: 2023-07-14T19:22:19.100+0000 7f73ba7c8200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-07-14T19:22:20.112 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Jul 14 19:22:19 smithi098 ceph-9adea648-227b-11ee-9b33-001a4aab830c-mgr-x[127734]: 2023-07-14T19:22:19.793+0000 7f73ba7c8200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-07-14T19:22:20.113 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Jul 14 19:22:19 smithi098 ceph-9adea648-227b-11ee-9b33-001a4aab830c-mgr-x[127734]: 2023-07-14T19:22:19.922+0000 7f73ba7c8200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-07-14T19:22:20.438 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Jul 14 19:22:20 smithi098 ceph-9adea648-227b-11ee-9b33-001a4aab830c-mgr-x[127734]: 2023-07-14T19:22:20.111+0000 7f73ba7c8200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-07-14T19:22:20.438 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Jul 14 19:22:20 smithi098 ceph-9adea648-227b-11ee-9b33-001a4aab830c-mgr-x[127734]: 2023-07-14T19:22:20.223+0000 7f73ba7c8200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-07-14T19:22:20.781 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Jul 14 19:22:20 smithi098 ceph-9adea648-227b-11ee-9b33-001a4aab830c-mgr-x[127734]: 2023-07-14T19:22:20.437+0000 7f73ba7c8200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-07-14T19:22:21.566 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Jul 14 19:22:21 smithi098 ceph-9adea648-227b-11ee-9b33-001a4aab830c-mgr-x[127734]: 2023-07-14T19:22:21.307+0000 7f73ba7c8200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-07-14T19:22:21.828 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout 2023-07-14T19:22:21.828 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout { 2023-07-14T19:22:21.828 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "fsid": "9adea648-227b-11ee-9b33-001a4aab830c", 2023-07-14T19:22:21.829 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "health": { 2023-07-14T19:22:21.829 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2023-07-14T19:22:21.829 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "checks": {}, 2023-07-14T19:22:21.829 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "mutes": [] 2023-07-14T19:22:21.829 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout }, 2023-07-14T19:22:21.829 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2023-07-14T19:22:21.830 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "quorum": [ 2023-07-14T19:22:21.830 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout 0 2023-07-14T19:22:21.830 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout ], 2023-07-14T19:22:21.830 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2023-07-14T19:22:21.830 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "a" 2023-07-14T19:22:21.830 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout ], 2023-07-14T19:22:21.831 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "quorum_age": 10, 2023-07-14T19:22:21.831 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "monmap": { 2023-07-14T19:22:21.831 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "epoch": 1, 2023-07-14T19:22:21.831 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "reef", 2023-07-14T19:22:21.831 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "num_mons": 1 2023-07-14T19:22:21.831 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout }, 2023-07-14T19:22:21.831 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "osdmap": { 2023-07-14T19:22:21.831 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "epoch": 1, 2023-07-14T19:22:21.832 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2023-07-14T19:22:21.832 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2023-07-14T19:22:21.832 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2023-07-14T19:22:21.832 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2023-07-14T19:22:21.832 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2023-07-14T19:22:21.832 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2023-07-14T19:22:21.832 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout }, 2023-07-14T19:22:21.833 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "pgmap": { 2023-07-14T19:22:21.833 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2023-07-14T19:22:21.833 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2023-07-14T19:22:21.833 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2023-07-14T19:22:21.833 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2023-07-14T19:22:21.833 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2023-07-14T19:22:21.833 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2023-07-14T19:22:21.834 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2023-07-14T19:22:21.834 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2023-07-14T19:22:21.834 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout }, 2023-07-14T19:22:21.834 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "fsmap": { 2023-07-14T19:22:21.835 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "epoch": 1, 2023-07-14T19:22:21.835 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "by_rank": [], 2023-07-14T19:22:21.835 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "up:standby": 0 2023-07-14T19:22:21.836 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout }, 2023-07-14T19:22:21.836 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "mgrmap": { 2023-07-14T19:22:21.836 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "available": false, 2023-07-14T19:22:21.836 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2023-07-14T19:22:21.836 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "modules": [ 2023-07-14T19:22:21.837 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "iostat", 2023-07-14T19:22:21.837 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "nfs", 2023-07-14T19:22:21.837 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "restful" 2023-07-14T19:22:21.838 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout ], 2023-07-14T19:22:21.838 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "services": {} 2023-07-14T19:22:21.838 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout }, 2023-07-14T19:22:21.839 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "servicemap": { 2023-07-14T19:22:21.839 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "epoch": 1, 2023-07-14T19:22:21.839 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "modified": "2023-07-14T19:22:07.179303+0000", 2023-07-14T19:22:21.840 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "services": {} 2023-07-14T19:22:21.840 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout }, 2023-07-14T19:22:21.840 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "progress_events": {} 2023-07-14T19:22:21.840 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout } 2023-07-14T19:22:21.841 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:21.325+0000 7f187c73d700 1 Processor -- start 2023-07-14T19:22:21.841 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:21.325+0000 7f187c73d700 1 -- start start 2023-07-14T19:22:21.841 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:21.325+0000 7f187c73d700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1874103fd0 0x7f18741043e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:22:21.842 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:21.325+0000 7f187c73d700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1874073e80 con 0x7f1874103fd0 2023-07-14T19:22:21.842 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:21.325+0000 7f187a4d9700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1874103fd0 0x7f18741043e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:22:21.842 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:21.325+0000 7f187a4d9700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1874103fd0 0x7f18741043e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:44342/0 (socket says 172.21.15.98:44342) 2023-07-14T19:22:21.842 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:21.325+0000 7f187a4d9700 1 -- 172.21.15.98:0/1822209857 learned_addr learned my addr 172.21.15.98:0/1822209857 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-07-14T19:22:21.843 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:21.326+0000 7f187a4d9700 1 -- 172.21.15.98:0/1822209857 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1874073fc0 con 0x7f1874103fd0 2023-07-14T19:22:21.843 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:21.326+0000 7f187a4d9700 1 --2- 172.21.15.98:0/1822209857 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1874103fd0 0x7f18741043e0 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f18700098a0 tx=0x7f1870009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=436e96d5152867b server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:22:21.843 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:21.326+0000 7f18794d7700 1 -- 172.21.15.98:0/1822209857 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f1870004160 con 0x7f1874103fd0 2023-07-14T19:22:21.843 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:21.326+0000 7f18794d7700 1 -- 172.21.15.98:0/1822209857 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f18700042c0 con 0x7f1874103fd0 2023-07-14T19:22:21.843 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:21.326+0000 7f18794d7700 1 -- 172.21.15.98:0/1822209857 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f1870011a00 con 0x7f1874103fd0 2023-07-14T19:22:21.843 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:21.327+0000 7f187c73d700 1 -- 172.21.15.98:0/1822209857 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1874103fd0 msgr2=0x7f18741043e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:22:21.844 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:21.327+0000 7f187c73d700 1 --2- 172.21.15.98:0/1822209857 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1874103fd0 0x7f18741043e0 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f18700098a0 tx=0x7f1870009bb0 comp rx=0 tx=0).stop 2023-07-14T19:22:21.844 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:21.327+0000 7f187c73d700 1 -- 172.21.15.98:0/1822209857 shutdown_connections 2023-07-14T19:22:21.844 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:21.327+0000 7f187c73d700 1 --2- 172.21.15.98:0/1822209857 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1874103fd0 0x7f18741043e0 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:22:21.844 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:21.327+0000 7f187c73d700 1 -- 172.21.15.98:0/1822209857 >> 172.21.15.98:0/1822209857 conn(0x7f18740ffbc0 msgr2=0x7f1874101ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:22:21.844 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:21.327+0000 7f187c73d700 1 -- 172.21.15.98:0/1822209857 shutdown_connections 2023-07-14T19:22:21.845 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:21.327+0000 7f187c73d700 1 -- 172.21.15.98:0/1822209857 wait complete. 2023-07-14T19:22:21.845 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:21.328+0000 7f187c73d700 1 Processor -- start 2023-07-14T19:22:21.845 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:21.328+0000 7f187c73d700 1 -- start start 2023-07-14T19:22:21.845 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:21.328+0000 7f187c73d700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1874103fd0 0x7f187419b680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:22:21.846 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:21.328+0000 7f187c73d700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f187419bbc0 con 0x7f1874103fd0 2023-07-14T19:22:21.846 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:21.328+0000 7f187a4d9700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1874103fd0 0x7f187419b680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:22:21.846 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:21.328+0000 7f187a4d9700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1874103fd0 0x7f187419b680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:44348/0 (socket says 172.21.15.98:44348) 2023-07-14T19:22:21.846 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:21.328+0000 7f187a4d9700 1 -- 172.21.15.98:0/3279155817 learned_addr learned my addr 172.21.15.98:0/3279155817 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-07-14T19:22:21.847 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:21.328+0000 7f187a4d9700 1 -- 172.21.15.98:0/3279155817 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1870009550 con 0x7f1874103fd0 2023-07-14T19:22:21.847 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:21.329+0000 7f187a4d9700 1 --2- 172.21.15.98:0/3279155817 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1874103fd0 0x7f187419b680 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f1870009130 tx=0x7f1870011ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:22:21.847 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:21.329+0000 7f186b7fe700 1 -- 172.21.15.98:0/3279155817 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f1870003ab0 con 0x7f1874103fd0 2023-07-14T19:22:21.847 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:21.329+0000 7f186b7fe700 1 -- 172.21.15.98:0/3279155817 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f1870003c10 con 0x7f1874103fd0 2023-07-14T19:22:21.847 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:21.329+0000 7f186b7fe700 1 -- 172.21.15.98:0/3279155817 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f18700247d0 con 0x7f1874103fd0 2023-07-14T19:22:21.848 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:21.330+0000 7f187c73d700 1 -- 172.21.15.98:0/3279155817 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f187419bdc0 con 0x7f1874103fd0 2023-07-14T19:22:21.848 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:21.330+0000 7f187c73d700 1 -- 172.21.15.98:0/3279155817 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f187419c1e0 con 0x7f1874103fd0 2023-07-14T19:22:21.848 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:21.331+0000 7f186b7fe700 1 -- 172.21.15.98:0/3279155817 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 2) v1 ==== 45239+0+0 (secure 0 0 0) 0x7f187001abb0 con 0x7f1874103fd0 2023-07-14T19:22:21.848 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:21.331+0000 7f186b7fe700 1 -- 172.21.15.98:0/3279155817 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7f187004c530 con 0x7f1874103fd0 2023-07-14T19:22:21.848 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:21.331+0000 7f187c73d700 1 -- 172.21.15.98:0/3279155817 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1874194ec0 con 0x7f1874103fd0 2023-07-14T19:22:21.848 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:21.333+0000 7f186b7fe700 1 -- 172.21.15.98:0/3279155817 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72446 (secure 0 0 0) 0x7f187001a430 con 0x7f1874103fd0 2023-07-14T19:22:21.849 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:21.410+0000 7f187c73d700 1 -- 172.21.15.98:0/3279155817 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) v1 -- 0x7f1874061c70 con 0x7f1874103fd0 2023-07-14T19:22:21.849 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:21.410+0000 7f186b7fe700 1 -- 172.21.15.98:0/3279155817 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) v1 ==== 79+0+1239 (secure 0 0 0) 0x7f187004a030 con 0x7f1874103fd0 2023-07-14T19:22:21.849 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:21.411+0000 7f187c73d700 1 -- 172.21.15.98:0/3279155817 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1874103fd0 msgr2=0x7f187419b680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:22:21.849 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:21.411+0000 7f187c73d700 1 --2- 172.21.15.98:0/3279155817 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1874103fd0 0x7f187419b680 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f1870009130 tx=0x7f1870011ad0 comp rx=0 tx=0).stop 2023-07-14T19:22:21.850 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:21.411+0000 7f187c73d700 1 -- 172.21.15.98:0/3279155817 shutdown_connections 2023-07-14T19:22:21.850 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:21.411+0000 7f187c73d700 1 --2- 172.21.15.98:0/3279155817 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1874103fd0 0x7f187419b680 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:22:21.850 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:21.411+0000 7f187c73d700 1 -- 172.21.15.98:0/3279155817 >> 172.21.15.98:0/3279155817 conn(0x7f18740ffbc0 msgr2=0x7f187418fb90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:22:21.850 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:21.411+0000 7f187c73d700 1 -- 172.21.15.98:0/3279155817 shutdown_connections 2023-07-14T19:22:21.850 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:21.411+0000 7f187c73d700 1 -- 172.21.15.98:0/3279155817 wait complete. 2023-07-14T19:22:21.850 INFO:teuthology.orchestra.run.smithi098.stdout:mgr not available, waiting (3/15)... 2023-07-14T19:22:22.532 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:22:22 smithi098 ceph-mon[127486]: Activating manager daemon x 2023-07-14T19:22:22.532 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:22:22 smithi098 ceph-mon[127486]: mgrmap e2: x(active, starting, since 0.00291998s) 2023-07-14T19:22:22.532 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:22:22 smithi098 ceph-mon[127486]: from='mgr.14100 172.21.15.98:0/1045717179' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2023-07-14T19:22:22.532 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:22:22 smithi098 ceph-mon[127486]: from='mgr.14100 172.21.15.98:0/1045717179' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2023-07-14T19:22:22.532 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:22:22 smithi098 ceph-mon[127486]: from='mgr.14100 172.21.15.98:0/1045717179' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2023-07-14T19:22:22.533 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:22:22 smithi098 ceph-mon[127486]: from='mgr.14100 172.21.15.98:0/1045717179' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2023-07-14T19:22:22.533 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:22:22 smithi098 ceph-mon[127486]: from='mgr.14100 172.21.15.98:0/1045717179' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2023-07-14T19:22:22.533 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:22:22 smithi098 ceph-mon[127486]: Manager daemon x is now available 2023-07-14T19:22:22.533 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:22:22 smithi098 ceph-mon[127486]: from='mgr.14100 172.21.15.98:0/1045717179' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2023-07-14T19:22:22.533 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:22:22 smithi098 ceph-mon[127486]: from='mgr.14100 172.21.15.98:0/1045717179' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2023-07-14T19:22:22.533 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:22:22 smithi098 ceph-mon[127486]: from='mgr.14100 172.21.15.98:0/1045717179' entity='mgr.x' 2023-07-14T19:22:22.533 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:22:22 smithi098 ceph-mon[127486]: from='mgr.14100 172.21.15.98:0/1045717179' entity='mgr.x' 2023-07-14T19:22:22.534 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:22:22 smithi098 ceph-mon[127486]: from='mgr.14100 172.21.15.98:0/1045717179' entity='mgr.x' 2023-07-14T19:22:22.534 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:22:22 smithi098 ceph-mon[127486]: from='client.? 172.21.15.98:0/3279155817' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2023-07-14T19:22:23.781 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:22:23 smithi098 ceph-mon[127486]: mgrmap e3: x(active, since 1.00593s) 2023-07-14T19:22:24.877 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout 2023-07-14T19:22:24.877 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout { 2023-07-14T19:22:24.877 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "fsid": "9adea648-227b-11ee-9b33-001a4aab830c", 2023-07-14T19:22:24.878 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "health": { 2023-07-14T19:22:24.878 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2023-07-14T19:22:24.878 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "checks": {}, 2023-07-14T19:22:24.878 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "mutes": [] 2023-07-14T19:22:24.878 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout }, 2023-07-14T19:22:24.879 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2023-07-14T19:22:24.879 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "quorum": [ 2023-07-14T19:22:24.879 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout 0 2023-07-14T19:22:24.880 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout ], 2023-07-14T19:22:24.880 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2023-07-14T19:22:24.880 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "a" 2023-07-14T19:22:24.880 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout ], 2023-07-14T19:22:24.880 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "quorum_age": 13, 2023-07-14T19:22:24.881 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "monmap": { 2023-07-14T19:22:24.881 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "epoch": 1, 2023-07-14T19:22:24.881 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "reef", 2023-07-14T19:22:24.881 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "num_mons": 1 2023-07-14T19:22:24.881 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout }, 2023-07-14T19:22:24.881 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "osdmap": { 2023-07-14T19:22:24.882 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "epoch": 1, 2023-07-14T19:22:24.882 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2023-07-14T19:22:24.882 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2023-07-14T19:22:24.882 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2023-07-14T19:22:24.882 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2023-07-14T19:22:24.882 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2023-07-14T19:22:24.882 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2023-07-14T19:22:24.882 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout }, 2023-07-14T19:22:24.883 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "pgmap": { 2023-07-14T19:22:24.883 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2023-07-14T19:22:24.883 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2023-07-14T19:22:24.883 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2023-07-14T19:22:24.883 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2023-07-14T19:22:24.883 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2023-07-14T19:22:24.883 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2023-07-14T19:22:24.884 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2023-07-14T19:22:24.884 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2023-07-14T19:22:24.884 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout }, 2023-07-14T19:22:24.885 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "fsmap": { 2023-07-14T19:22:24.885 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "epoch": 1, 2023-07-14T19:22:24.885 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "by_rank": [], 2023-07-14T19:22:24.886 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "up:standby": 0 2023-07-14T19:22:24.886 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout }, 2023-07-14T19:22:24.886 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "mgrmap": { 2023-07-14T19:22:24.886 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "available": true, 2023-07-14T19:22:24.887 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2023-07-14T19:22:24.887 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "modules": [ 2023-07-14T19:22:24.887 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "iostat", 2023-07-14T19:22:24.888 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "nfs", 2023-07-14T19:22:24.888 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "restful" 2023-07-14T19:22:24.888 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout ], 2023-07-14T19:22:24.888 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "services": {} 2023-07-14T19:22:24.889 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout }, 2023-07-14T19:22:24.889 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "servicemap": { 2023-07-14T19:22:24.889 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "epoch": 1, 2023-07-14T19:22:24.889 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "modified": "2023-07-14T19:22:07.179303+0000", 2023-07-14T19:22:24.889 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "services": {} 2023-07-14T19:22:24.889 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout }, 2023-07-14T19:22:24.890 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "progress_events": {} 2023-07-14T19:22:24.890 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout } 2023-07-14T19:22:24.890 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:24.293+0000 7f6c7477b700 1 Processor -- start 2023-07-14T19:22:24.890 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:24.293+0000 7f6c7477b700 1 -- start start 2023-07-14T19:22:24.890 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:24.293+0000 7f6c7477b700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6c6c1061c0 0x7f6c6c1085a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:22:24.890 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:24.293+0000 7f6c7477b700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6c6c073ea0 con 0x7f6c6c1061c0 2023-07-14T19:22:24.890 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:24.293+0000 7f6c72517700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6c6c1061c0 0x7f6c6c1085a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:22:24.891 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:24.294+0000 7f6c72517700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6c6c1061c0 0x7f6c6c1085a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:44368/0 (socket says 172.21.15.98:44368) 2023-07-14T19:22:24.891 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:24.294+0000 7f6c72517700 1 -- 172.21.15.98:0/535752735 learned_addr learned my addr 172.21.15.98:0/535752735 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-07-14T19:22:24.891 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:24.294+0000 7f6c72517700 1 -- 172.21.15.98:0/535752735 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6c6c073fe0 con 0x7f6c6c1061c0 2023-07-14T19:22:24.891 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:24.294+0000 7f6c72517700 1 --2- 172.21.15.98:0/535752735 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6c6c1061c0 0x7f6c6c1085a0 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f6c680098a0 tx=0x7f6c68009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=54adb4beddb559ab server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:22:24.891 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:24.294+0000 7f6c71515700 1 -- 172.21.15.98:0/535752735 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f6c68004160 con 0x7f6c6c1061c0 2023-07-14T19:22:24.891 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:24.294+0000 7f6c71515700 1 -- 172.21.15.98:0/535752735 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f6c680042c0 con 0x7f6c6c1061c0 2023-07-14T19:22:24.891 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:24.294+0000 7f6c71515700 1 -- 172.21.15.98:0/535752735 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f6c68011a00 con 0x7f6c6c1061c0 2023-07-14T19:22:24.892 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:24.295+0000 7f6c7477b700 1 -- 172.21.15.98:0/535752735 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6c6c1061c0 msgr2=0x7f6c6c1085a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:22:24.892 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:24.295+0000 7f6c7477b700 1 --2- 172.21.15.98:0/535752735 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6c6c1061c0 0x7f6c6c1085a0 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f6c680098a0 tx=0x7f6c68009bb0 comp rx=0 tx=0).stop 2023-07-14T19:22:24.892 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:24.295+0000 7f6c7477b700 1 -- 172.21.15.98:0/535752735 shutdown_connections 2023-07-14T19:22:24.892 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:24.295+0000 7f6c7477b700 1 --2- 172.21.15.98:0/535752735 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6c6c1061c0 0x7f6c6c1085a0 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:22:24.892 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:24.295+0000 7f6c7477b700 1 -- 172.21.15.98:0/535752735 >> 172.21.15.98:0/535752735 conn(0x7f6c6c0ffbc0 msgr2=0x7f6c6c101ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:22:24.892 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:24.295+0000 7f6c7477b700 1 -- 172.21.15.98:0/535752735 shutdown_connections 2023-07-14T19:22:24.893 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:24.295+0000 7f6c7477b700 1 -- 172.21.15.98:0/535752735 wait complete. 2023-07-14T19:22:24.893 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:24.295+0000 7f6c7477b700 1 Processor -- start 2023-07-14T19:22:24.893 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:24.295+0000 7f6c7477b700 1 -- start start 2023-07-14T19:22:24.893 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:24.295+0000 7f6c7477b700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6c6c1061c0 0x7f6c6c199420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:22:24.893 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:24.295+0000 7f6c7477b700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6c6c199960 con 0x7f6c6c1061c0 2023-07-14T19:22:24.893 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:24.296+0000 7f6c72517700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6c6c1061c0 0x7f6c6c199420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:22:24.893 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:24.296+0000 7f6c72517700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6c6c1061c0 0x7f6c6c199420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:44370/0 (socket says 172.21.15.98:44370) 2023-07-14T19:22:24.894 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:24.296+0000 7f6c72517700 1 -- 172.21.15.98:0/790425915 learned_addr learned my addr 172.21.15.98:0/790425915 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-07-14T19:22:24.894 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:24.296+0000 7f6c72517700 1 -- 172.21.15.98:0/790425915 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6c68009550 con 0x7f6c6c1061c0 2023-07-14T19:22:24.894 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:24.296+0000 7f6c72517700 1 --2- 172.21.15.98:0/790425915 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6c6c1061c0 0x7f6c6c199420 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f6c68009fb0 tx=0x7f6c68011ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:22:24.894 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:24.296+0000 7f6c637fe700 1 -- 172.21.15.98:0/790425915 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f6c68003a40 con 0x7f6c6c1061c0 2023-07-14T19:22:24.894 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:24.296+0000 7f6c7477b700 1 -- 172.21.15.98:0/790425915 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6c6c199bc0 con 0x7f6c6c1061c0 2023-07-14T19:22:24.894 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:24.296+0000 7f6c637fe700 1 -- 172.21.15.98:0/790425915 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f6c68003ba0 con 0x7f6c6c1061c0 2023-07-14T19:22:24.894 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:24.296+0000 7f6c637fe700 1 -- 172.21.15.98:0/790425915 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f6c6801a750 con 0x7f6c6c1061c0 2023-07-14T19:22:24.895 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:24.296+0000 7f6c7477b700 1 -- 172.21.15.98:0/790425915 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6c6c19a040 con 0x7f6c6c1061c0 2023-07-14T19:22:24.895 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:24.297+0000 7f6c637fe700 1 -- 172.21.15.98:0/790425915 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 3) v1 ==== 45366+0+0 (secure 0 0 0) 0x7f6c68024430 con 0x7f6c6c1061c0 2023-07-14T19:22:24.895 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:24.297+0000 7f6c7477b700 1 -- 172.21.15.98:0/790425915 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6c6c1933d0 con 0x7f6c6c1061c0 2023-07-14T19:22:24.895 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:24.297+0000 7f6c637fe700 1 --2- 172.21.15.98:0/790425915 >> [v2:172.21.15.98:6800/749234665,v1:172.21.15.98:6801/749234665] conn(0x7f6c58038550 0x7f6c5803aa00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:22:24.895 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:24.297+0000 7f6c637fe700 1 -- 172.21.15.98:0/790425915 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7f6c6804c140 con 0x7f6c6c1061c0 2023-07-14T19:22:24.895 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:24.297+0000 7f6c71d16700 1 --2- 172.21.15.98:0/790425915 >> [v2:172.21.15.98:6800/749234665,v1:172.21.15.98:6801/749234665] conn(0x7f6c58038550 0x7f6c5803aa00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:22:24.895 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:24.298+0000 7f6c71d16700 1 --2- 172.21.15.98:0/790425915 >> [v2:172.21.15.98:6800/749234665,v1:172.21.15.98:6801/749234665] conn(0x7f6c58038550 0x7f6c5803aa00 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f6c5c006fd0 tx=0x7f6c5c006e40 comp rx=0 tx=0).ready entity=mgr.14100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:22:24.896 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:24.302+0000 7f6c637fe700 1 -- 172.21.15.98:0/790425915 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f6c680189a0 con 0x7f6c6c1061c0 2023-07-14T19:22:24.896 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:24.325+0000 7f6c637fe700 1 -- 172.21.15.98:0/790425915 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mgrmap(e 4) v1 ==== 45472+0+0 (secure 0 0 0) 0x7f6c680246e0 con 0x7f6c6c1061c0 2023-07-14T19:22:24.896 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:24.557+0000 7f6c7477b700 1 -- 172.21.15.98:0/790425915 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) v1 -- 0x7f6c6c02c8e0 con 0x7f6c6c1061c0 2023-07-14T19:22:24.896 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:24.557+0000 7f6c637fe700 1 -- 172.21.15.98:0/790425915 <== mon.0 v2:172.21.15.98:3300/0 8 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) v1 ==== 79+0+1238 (secure 0 0 0) 0x7f6c68018d90 con 0x7f6c6c1061c0 2023-07-14T19:22:24.896 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:24.559+0000 7f6c7477b700 1 -- 172.21.15.98:0/790425915 >> [v2:172.21.15.98:6800/749234665,v1:172.21.15.98:6801/749234665] conn(0x7f6c58038550 msgr2=0x7f6c5803aa00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:22:24.896 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:24.559+0000 7f6c7477b700 1 --2- 172.21.15.98:0/790425915 >> [v2:172.21.15.98:6800/749234665,v1:172.21.15.98:6801/749234665] conn(0x7f6c58038550 0x7f6c5803aa00 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f6c5c006fd0 tx=0x7f6c5c006e40 comp rx=0 tx=0).stop 2023-07-14T19:22:24.896 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:24.559+0000 7f6c7477b700 1 -- 172.21.15.98:0/790425915 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6c6c1061c0 msgr2=0x7f6c6c199420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:22:24.897 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:24.559+0000 7f6c7477b700 1 --2- 172.21.15.98:0/790425915 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6c6c1061c0 0x7f6c6c199420 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f6c68009fb0 tx=0x7f6c68011ad0 comp rx=0 tx=0).stop 2023-07-14T19:22:24.897 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:24.559+0000 7f6c7477b700 1 -- 172.21.15.98:0/790425915 shutdown_connections 2023-07-14T19:22:24.897 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:24.559+0000 7f6c7477b700 1 --2- 172.21.15.98:0/790425915 >> [v2:172.21.15.98:6800/749234665,v1:172.21.15.98:6801/749234665] conn(0x7f6c58038550 0x7f6c5803aa00 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:22:24.897 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:24.559+0000 7f6c7477b700 1 --2- 172.21.15.98:0/790425915 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6c6c1061c0 0x7f6c6c199420 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:22:24.897 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:24.559+0000 7f6c7477b700 1 -- 172.21.15.98:0/790425915 >> 172.21.15.98:0/790425915 conn(0x7f6c6c0ffbc0 msgr2=0x7f6c6c101300 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:22:24.897 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:24.559+0000 7f6c7477b700 1 -- 172.21.15.98:0/790425915 shutdown_connections 2023-07-14T19:22:24.897 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:24.559+0000 7f6c7477b700 1 -- 172.21.15.98:0/790425915 wait complete. 2023-07-14T19:22:24.898 INFO:teuthology.orchestra.run.smithi098.stdout:mgr is available 2023-07-14T19:22:25.781 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:22:25 smithi098 ceph-mon[127486]: mgrmap e4: x(active, since 3s) 2023-07-14T19:22:25.782 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:22:25 smithi098 ceph-mon[127486]: from='client.? 172.21.15.98:0/790425915' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2023-07-14T19:22:27.531 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:22:27 smithi098 ceph-mon[127486]: from='client.? 172.21.15.98:0/2442377612' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2023-07-14T19:22:29.932 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout 2023-07-14T19:22:29.932 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout [global] 2023-07-14T19:22:29.932 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout fsid = 9adea648-227b-11ee-9b33-001a4aab830c 2023-07-14T19:22:29.932 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout mon_osd_allow_pg_remap = true 2023-07-14T19:22:29.933 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout mon_osd_allow_primary_affinity = true 2023-07-14T19:22:29.933 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout mon_warn_on_no_sortbitwise = false 2023-07-14T19:22:29.933 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout osd_crush_chooseleaf_type = 0 2023-07-14T19:22:29.933 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout 2023-07-14T19:22:29.933 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout [mgr] 2023-07-14T19:22:29.933 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout mgr/telemetry/nag = false 2023-07-14T19:22:29.933 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout 2023-07-14T19:22:29.933 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout [osd] 2023-07-14T19:22:29.934 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout filestore_flush_min = 0 2023-07-14T19:22:29.934 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout mon_osd_backfillfull_ratio = 0.85 2023-07-14T19:22:29.934 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout mon_osd_full_ratio = 0.9 2023-07-14T19:22:29.934 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout mon_osd_nearfull_ratio = 0.8 2023-07-14T19:22:29.934 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout osd_map_max_advance = 10 2023-07-14T19:22:29.934 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout osd_sloppy_crc = true 2023-07-14T19:22:29.934 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:26.974+0000 7f27f4c90700 1 Processor -- start 2023-07-14T19:22:29.935 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:26.974+0000 7f27f4c90700 1 -- start start 2023-07-14T19:22:29.935 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:26.974+0000 7f27f4c90700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f27f01061c0 0x7f27f01085a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:22:29.935 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:26.974+0000 7f27f4c90700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f27f0073ea0 con 0x7f27f01061c0 2023-07-14T19:22:29.935 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:26.974+0000 7f27ee59c700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f27f01061c0 0x7f27f01085a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:22:29.935 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:26.974+0000 7f27ee59c700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f27f01061c0 0x7f27f01085a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:44374/0 (socket says 172.21.15.98:44374) 2023-07-14T19:22:29.936 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:26.974+0000 7f27ee59c700 1 -- 172.21.15.98:0/903263804 learned_addr learned my addr 172.21.15.98:0/903263804 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-07-14T19:22:29.936 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:26.975+0000 7f27ee59c700 1 -- 172.21.15.98:0/903263804 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f27f0073fe0 con 0x7f27f01061c0 2023-07-14T19:22:29.936 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:26.975+0000 7f27ee59c700 1 --2- 172.21.15.98:0/903263804 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f27f01061c0 0x7f27f01085a0 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f27e00098a0 tx=0x7f27e0009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f090f59a37989569 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:22:29.936 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:26.975+0000 7f27ed59a700 1 -- 172.21.15.98:0/903263804 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f27e0004160 con 0x7f27f01061c0 2023-07-14T19:22:29.936 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:26.975+0000 7f27ed59a700 1 -- 172.21.15.98:0/903263804 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f27e00042c0 con 0x7f27f01061c0 2023-07-14T19:22:29.936 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:26.975+0000 7f27ed59a700 1 -- 172.21.15.98:0/903263804 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f27e0011a00 con 0x7f27f01061c0 2023-07-14T19:22:29.936 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:26.975+0000 7f27f4c90700 1 -- 172.21.15.98:0/903263804 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f27f01061c0 msgr2=0x7f27f01085a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:22:29.937 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:26.975+0000 7f27f4c90700 1 --2- 172.21.15.98:0/903263804 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f27f01061c0 0x7f27f01085a0 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f27e00098a0 tx=0x7f27e0009bb0 comp rx=0 tx=0).stop 2023-07-14T19:22:29.937 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:26.976+0000 7f27f4c90700 1 -- 172.21.15.98:0/903263804 shutdown_connections 2023-07-14T19:22:29.937 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:26.976+0000 7f27f4c90700 1 --2- 172.21.15.98:0/903263804 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f27f01061c0 0x7f27f01085a0 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:22:29.937 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:26.976+0000 7f27f4c90700 1 -- 172.21.15.98:0/903263804 >> 172.21.15.98:0/903263804 conn(0x7f27f00ffbc0 msgr2=0x7f27f0101ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:22:29.937 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:26.976+0000 7f27f4c90700 1 -- 172.21.15.98:0/903263804 shutdown_connections 2023-07-14T19:22:29.937 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:26.976+0000 7f27f4c90700 1 -- 172.21.15.98:0/903263804 wait complete. 2023-07-14T19:22:29.938 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:26.976+0000 7f27f4c90700 1 Processor -- start 2023-07-14T19:22:29.938 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:26.976+0000 7f27f4c90700 1 -- start start 2023-07-14T19:22:29.938 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:26.977+0000 7f27f4c90700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f27f01061c0 0x7f27f0199420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:22:29.938 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:26.977+0000 7f27f4c90700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f27f0199960 con 0x7f27f01061c0 2023-07-14T19:22:29.939 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:26.977+0000 7f27ee59c700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f27f01061c0 0x7f27f0199420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:22:29.939 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:26.977+0000 7f27ee59c700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f27f01061c0 0x7f27f0199420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:44376/0 (socket says 172.21.15.98:44376) 2023-07-14T19:22:29.939 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:26.977+0000 7f27ee59c700 1 -- 172.21.15.98:0/2442377612 learned_addr learned my addr 172.21.15.98:0/2442377612 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-07-14T19:22:29.939 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:26.977+0000 7f27ee59c700 1 -- 172.21.15.98:0/2442377612 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f27e0009550 con 0x7f27f01061c0 2023-07-14T19:22:29.939 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:26.977+0000 7f27ee59c700 1 --2- 172.21.15.98:0/2442377612 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f27f01061c0 0x7f27f0199420 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f27e0009fb0 tx=0x7f27e0011ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:22:29.939 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:26.977+0000 7f27df7fe700 1 -- 172.21.15.98:0/2442377612 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f27e0003a40 con 0x7f27f01061c0 2023-07-14T19:22:29.939 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:26.977+0000 7f27df7fe700 1 -- 172.21.15.98:0/2442377612 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f27e0003ba0 con 0x7f27f01061c0 2023-07-14T19:22:29.940 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:26.977+0000 7f27f4c90700 1 -- 172.21.15.98:0/2442377612 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f27f0199bc0 con 0x7f27f01061c0 2023-07-14T19:22:29.940 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:26.977+0000 7f27df7fe700 1 -- 172.21.15.98:0/2442377612 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f27e001a750 con 0x7f27f01061c0 2023-07-14T19:22:29.940 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:26.977+0000 7f27f4c90700 1 -- 172.21.15.98:0/2442377612 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f27f019a040 con 0x7f27f01061c0 2023-07-14T19:22:29.940 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:26.978+0000 7f27df7fe700 1 -- 172.21.15.98:0/2442377612 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 4) v1 ==== 45472+0+0 (secure 0 0 0) 0x7f27e001a9d0 con 0x7f27f01061c0 2023-07-14T19:22:29.940 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:26.978+0000 7f27df7fe700 1 --2- 172.21.15.98:0/2442377612 >> [v2:172.21.15.98:6800/749234665,v1:172.21.15.98:6801/749234665] conn(0x7f27d8038670 0x7f27d803ab20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:22:29.940 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:26.978+0000 7f27df7fe700 1 -- 172.21.15.98:0/2442377612 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7f27e004c280 con 0x7f27f01061c0 2023-07-14T19:22:29.940 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:26.978+0000 7f27f4c90700 1 -- 172.21.15.98:0/2442377612 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f27f01933d0 con 0x7f27f01061c0 2023-07-14T19:22:29.941 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:26.978+0000 7f27edd9b700 1 --2- 172.21.15.98:0/2442377612 >> [v2:172.21.15.98:6800/749234665,v1:172.21.15.98:6801/749234665] conn(0x7f27d8038670 0x7f27d803ab20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:22:29.941 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:26.979+0000 7f27edd9b700 1 --2- 172.21.15.98:0/2442377612 >> [v2:172.21.15.98:6800/749234665,v1:172.21.15.98:6801/749234665] conn(0x7f27d8038670 0x7f27d803ab20 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f27e4006fd0 tx=0x7f27e4006e40 comp rx=0 tx=0).ready entity=mgr.14100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:22:29.941 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:26.983+0000 7f27df7fe700 1 -- 172.21.15.98:0/2442377612 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f27e0018d90 con 0x7f27f01061c0 2023-07-14T19:22:29.941 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:27.168+0000 7f27f4c90700 1 -- 172.21.15.98:0/2442377612 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "config assimilate-conf"} v 0) v1 -- 0x7f27f002bf30 con 0x7f27f01061c0 2023-07-14T19:22:29.941 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:27.169+0000 7f27df7fe700 1 -- 172.21.15.98:0/2442377612 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "config assimilate-conf"}]=0 v3) v1 ==== 70+0+398 (secure 0 0 0) 0x7f27e0018350 con 0x7f27f01061c0 2023-07-14T19:22:29.941 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:27.171+0000 7f27f4c90700 1 -- 172.21.15.98:0/2442377612 >> [v2:172.21.15.98:6800/749234665,v1:172.21.15.98:6801/749234665] conn(0x7f27d8038670 msgr2=0x7f27d803ab20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:22:29.941 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:27.171+0000 7f27f4c90700 1 --2- 172.21.15.98:0/2442377612 >> [v2:172.21.15.98:6800/749234665,v1:172.21.15.98:6801/749234665] conn(0x7f27d8038670 0x7f27d803ab20 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f27e4006fd0 tx=0x7f27e4006e40 comp rx=0 tx=0).stop 2023-07-14T19:22:29.942 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:27.171+0000 7f27f4c90700 1 -- 172.21.15.98:0/2442377612 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f27f01061c0 msgr2=0x7f27f0199420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:22:29.942 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:27.171+0000 7f27f4c90700 1 --2- 172.21.15.98:0/2442377612 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f27f01061c0 0x7f27f0199420 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f27e0009fb0 tx=0x7f27e0011ad0 comp rx=0 tx=0).stop 2023-07-14T19:22:29.942 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:27.171+0000 7f27f4c90700 1 -- 172.21.15.98:0/2442377612 shutdown_connections 2023-07-14T19:22:29.942 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:27.171+0000 7f27f4c90700 1 --2- 172.21.15.98:0/2442377612 >> [v2:172.21.15.98:6800/749234665,v1:172.21.15.98:6801/749234665] conn(0x7f27d8038670 0x7f27d803ab20 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:22:29.942 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:27.171+0000 7f27f4c90700 1 --2- 172.21.15.98:0/2442377612 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f27f01061c0 0x7f27f0199420 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:22:29.942 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:27.171+0000 7f27f4c90700 1 -- 172.21.15.98:0/2442377612 >> 172.21.15.98:0/2442377612 conn(0x7f27f00ffbc0 msgr2=0x7f27f0101300 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:22:29.942 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:27.171+0000 7f27f4c90700 1 -- 172.21.15.98:0/2442377612 shutdown_connections 2023-07-14T19:22:29.943 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:27.171+0000 7f27f4c90700 1 -- 172.21.15.98:0/2442377612 wait complete. 2023-07-14T19:22:29.943 INFO:teuthology.orchestra.run.smithi098.stdout:Enabling cephadm module... 2023-07-14T19:22:33.237 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:22:32 smithi098 ceph-mon[127486]: from='client.? 172.21.15.98:0/3919851352' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "cephadm"}]: dispatch 2023-07-14T19:22:33.237 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Jul 14 19:22:32 smithi098 ceph-9adea648-227b-11ee-9b33-001a4aab830c-mgr-x[127734]: ignoring --setuser ceph since I am not root 2023-07-14T19:22:33.237 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Jul 14 19:22:32 smithi098 ceph-9adea648-227b-11ee-9b33-001a4aab830c-mgr-x[127734]: ignoring --setgroup ceph since I am not root 2023-07-14T19:22:33.238 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Jul 14 19:22:33 smithi098 ceph-9adea648-227b-11ee-9b33-001a4aab830c-mgr-x[127734]: 2023-07-14T19:22:33.067+0000 7f0f199f7200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-07-14T19:22:33.238 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Jul 14 19:22:33 smithi098 ceph-9adea648-227b-11ee-9b33-001a4aab830c-mgr-x[127734]: 2023-07-14T19:22:33.145+0000 7f0f199f7200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-07-14T19:22:33.504 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Jul 14 19:22:33 smithi098 ceph-9adea648-227b-11ee-9b33-001a4aab830c-mgr-x[127734]: 2023-07-14T19:22:33.235+0000 7f0f199f7200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-07-14T19:22:33.505 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Jul 14 19:22:33 smithi098 ceph-9adea648-227b-11ee-9b33-001a4aab830c-mgr-x[127734]: 2023-07-14T19:22:33.334+0000 7f0f199f7200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-07-14T19:22:33.781 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Jul 14 19:22:33 smithi098 ceph-9adea648-227b-11ee-9b33-001a4aab830c-mgr-x[127734]: 2023-07-14T19:22:33.502+0000 7f0f199f7200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-07-14T19:22:34.123 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:22:33 smithi098 ceph-mon[127486]: from='client.? 172.21.15.98:0/3919851352' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "cephadm"}]': finished 2023-07-14T19:22:34.123 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:22:33 smithi098 ceph-mon[127486]: mgrmap e5: x(active, since 11s) 2023-07-14T19:22:34.124 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Jul 14 19:22:33 smithi098 ceph-9adea648-227b-11ee-9b33-001a4aab830c-mgr-x[127734]: 2023-07-14T19:22:33.809+0000 7f0f199f7200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-07-14T19:22:34.124 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Jul 14 19:22:33 smithi098 ceph-9adea648-227b-11ee-9b33-001a4aab830c-mgr-x[127734]: 2023-07-14T19:22:33.899+0000 7f0f199f7200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-07-14T19:22:34.124 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Jul 14 19:22:34 smithi098 ceph-9adea648-227b-11ee-9b33-001a4aab830c-mgr-x[127734]: 2023-07-14T19:22:34.026+0000 7f0f199f7200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-07-14T19:22:34.535 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Jul 14 19:22:34 smithi098 ceph-9adea648-227b-11ee-9b33-001a4aab830c-mgr-x[127734]: 2023-07-14T19:22:34.121+0000 7f0f199f7200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-07-14T19:22:35.031 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Jul 14 19:22:34 smithi098 ceph-9adea648-227b-11ee-9b33-001a4aab830c-mgr-x[127734]: 2023-07-14T19:22:34.536+0000 7f0f199f7200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-07-14T19:22:35.293 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:32.610+0000 7f91b6739700 1 Processor -- start 2023-07-14T19:22:35.293 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:32.610+0000 7f91b6739700 1 -- start start 2023-07-14T19:22:35.294 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:32.610+0000 7f91b6739700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f91b0103950 0x7f91b0103d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:22:35.294 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:32.610+0000 7f91b6739700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f91b01042a0 con 0x7f91b0103950 2023-07-14T19:22:35.294 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:32.611+0000 7f91affff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f91b0103950 0x7f91b0103d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:22:35.295 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:32.611+0000 7f91affff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f91b0103950 0x7f91b0103d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:44380/0 (socket says 172.21.15.98:44380) 2023-07-14T19:22:35.295 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:32.611+0000 7f91affff700 1 -- 172.21.15.98:0/2941898085 learned_addr learned my addr 172.21.15.98:0/2941898085 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-07-14T19:22:35.295 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:32.611+0000 7f91affff700 1 -- 172.21.15.98:0/2941898085 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f91b01043e0 con 0x7f91b0103950 2023-07-14T19:22:35.295 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:32.611+0000 7f91affff700 1 --2- 172.21.15.98:0/2941898085 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f91b0103950 0x7f91b0103d60 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f91a00098a0 tx=0x7f91a0009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b2d21cc3b5776146 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:22:35.295 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:32.611+0000 7f91aeffd700 1 -- 172.21.15.98:0/2941898085 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f91a0004160 con 0x7f91b0103950 2023-07-14T19:22:35.296 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:32.611+0000 7f91aeffd700 1 -- 172.21.15.98:0/2941898085 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f91a00042c0 con 0x7f91b0103950 2023-07-14T19:22:35.296 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:32.611+0000 7f91aeffd700 1 -- 172.21.15.98:0/2941898085 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f91a0011a00 con 0x7f91b0103950 2023-07-14T19:22:35.296 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:32.612+0000 7f91b6739700 1 -- 172.21.15.98:0/2941898085 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f91b0103950 msgr2=0x7f91b0103d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:22:35.296 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:32.612+0000 7f91b6739700 1 --2- 172.21.15.98:0/2941898085 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f91b0103950 0x7f91b0103d60 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f91a00098a0 tx=0x7f91a0009bb0 comp rx=0 tx=0).stop 2023-07-14T19:22:35.296 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:32.612+0000 7f91b6739700 1 -- 172.21.15.98:0/2941898085 shutdown_connections 2023-07-14T19:22:35.297 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:32.612+0000 7f91b6739700 1 --2- 172.21.15.98:0/2941898085 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f91b0103950 0x7f91b0103d60 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:22:35.297 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:32.612+0000 7f91b6739700 1 -- 172.21.15.98:0/2941898085 >> 172.21.15.98:0/2941898085 conn(0x7f91b00ff330 msgr2=0x7f91b0101760 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:22:35.297 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:32.612+0000 7f91b6739700 1 -- 172.21.15.98:0/2941898085 shutdown_connections 2023-07-14T19:22:35.297 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:32.612+0000 7f91b6739700 1 -- 172.21.15.98:0/2941898085 wait complete. 2023-07-14T19:22:35.297 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:32.613+0000 7f91b6739700 1 Processor -- start 2023-07-14T19:22:35.297 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:32.613+0000 7f91b6739700 1 -- start start 2023-07-14T19:22:35.298 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:32.613+0000 7f91b6739700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f91b0103950 0x7f91b01994d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:22:35.298 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:32.613+0000 7f91b6739700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f91b0199a10 con 0x7f91b0103950 2023-07-14T19:22:35.298 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:32.613+0000 7f91affff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f91b0103950 0x7f91b01994d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:22:35.298 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:32.613+0000 7f91affff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f91b0103950 0x7f91b01994d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:44382/0 (socket says 172.21.15.98:44382) 2023-07-14T19:22:35.298 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:32.613+0000 7f91affff700 1 -- 172.21.15.98:0/3919851352 learned_addr learned my addr 172.21.15.98:0/3919851352 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-07-14T19:22:35.298 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:32.613+0000 7f91affff700 1 -- 172.21.15.98:0/3919851352 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f91a0009550 con 0x7f91b0103950 2023-07-14T19:22:35.298 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:32.613+0000 7f91affff700 1 --2- 172.21.15.98:0/3919851352 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f91b0103950 0x7f91b01994d0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f91a0009130 tx=0x7f91a0011ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:22:35.299 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:32.614+0000 7f91ad7fa700 1 -- 172.21.15.98:0/3919851352 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f91a0003ab0 con 0x7f91b0103950 2023-07-14T19:22:35.299 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:32.614+0000 7f91ad7fa700 1 -- 172.21.15.98:0/3919851352 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f91a0003c10 con 0x7f91b0103950 2023-07-14T19:22:35.299 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:32.614+0000 7f91b6739700 1 -- 172.21.15.98:0/3919851352 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f91b0199c10 con 0x7f91b0103950 2023-07-14T19:22:35.299 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:32.614+0000 7f91ad7fa700 1 -- 172.21.15.98:0/3919851352 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f91a00247d0 con 0x7f91b0103950 2023-07-14T19:22:35.299 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:32.614+0000 7f91b6739700 1 -- 172.21.15.98:0/3919851352 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f91b019a030 con 0x7f91b0103950 2023-07-14T19:22:35.299 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:32.614+0000 7f91ad7fa700 1 -- 172.21.15.98:0/3919851352 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 4) v1 ==== 45472+0+0 (secure 0 0 0) 0x7f91a002d3f0 con 0x7f91b0103950 2023-07-14T19:22:35.299 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:32.615+0000 7f91ad7fa700 1 --2- 172.21.15.98:0/3919851352 >> [v2:172.21.15.98:6800/749234665,v1:172.21.15.98:6801/749234665] conn(0x7f91980386c0 0x7f919803ab70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:22:35.300 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:32.615+0000 7f91ad7fa700 1 -- 172.21.15.98:0/3919851352 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7f91a004c640 con 0x7f91b0103950 2023-07-14T19:22:35.300 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:32.615+0000 7f91b6739700 1 -- 172.21.15.98:0/3919851352 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f91b0193430 con 0x7f91b0103950 2023-07-14T19:22:35.300 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:32.615+0000 7f91af7fe700 1 --2- 172.21.15.98:0/3919851352 >> [v2:172.21.15.98:6800/749234665,v1:172.21.15.98:6801/749234665] conn(0x7f91980386c0 0x7f919803ab70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:22:35.300 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:32.615+0000 7f91af7fe700 1 --2- 172.21.15.98:0/3919851352 >> [v2:172.21.15.98:6800/749234665,v1:172.21.15.98:6801/749234665] conn(0x7f91980386c0 0x7f919803ab70 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f91a4006fd0 tx=0x7f91a4006e40 comp rx=0 tx=0).ready entity=mgr.14100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:22:35.300 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:32.619+0000 7f91ad7fa700 1 -- 172.21.15.98:0/3919851352 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f91a0018b00 con 0x7f91b0103950 2023-07-14T19:22:35.300 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:32.857+0000 7f91b6739700 1 -- 172.21.15.98:0/3919851352 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "cephadm"} v 0) v1 -- 0x7f91b0061250 con 0x7f91b0103950 2023-07-14T19:22:35.300 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:32.910+0000 7f91ad7fa700 1 -- 172.21.15.98:0/3919851352 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mgrmap(e 5) v1 ==== 45483+0+0 (secure 0 0 0) 0x7f91a001a770 con 0x7f91b0103950 2023-07-14T19:22:35.301 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:32.912+0000 7f91ad7fa700 1 -- 172.21.15.98:0/3919851352 <== mon.0 v2:172.21.15.98:3300/0 8 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "cephadm"}]=0 v5) v1 ==== 86+0+0 (secure 0 0 0) 0x7f91a0010eb0 con 0x7f91b0103950 2023-07-14T19:22:35.301 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:32.914+0000 7f91b6739700 1 -- 172.21.15.98:0/3919851352 >> [v2:172.21.15.98:6800/749234665,v1:172.21.15.98:6801/749234665] conn(0x7f91980386c0 msgr2=0x7f919803ab70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:22:35.301 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:32.914+0000 7f91b6739700 1 --2- 172.21.15.98:0/3919851352 >> [v2:172.21.15.98:6800/749234665,v1:172.21.15.98:6801/749234665] conn(0x7f91980386c0 0x7f919803ab70 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f91a4006fd0 tx=0x7f91a4006e40 comp rx=0 tx=0).stop 2023-07-14T19:22:35.301 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:32.914+0000 7f91b6739700 1 -- 172.21.15.98:0/3919851352 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f91b0103950 msgr2=0x7f91b01994d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:22:35.301 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:32.914+0000 7f91b6739700 1 --2- 172.21.15.98:0/3919851352 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f91b0103950 0x7f91b01994d0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f91a0009130 tx=0x7f91a0011ad0 comp rx=0 tx=0).stop 2023-07-14T19:22:35.301 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:32.914+0000 7f91b6739700 1 -- 172.21.15.98:0/3919851352 shutdown_connections 2023-07-14T19:22:35.301 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:32.914+0000 7f91b6739700 1 --2- 172.21.15.98:0/3919851352 >> [v2:172.21.15.98:6800/749234665,v1:172.21.15.98:6801/749234665] conn(0x7f91980386c0 0x7f919803ab70 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:22:35.302 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:32.914+0000 7f91b6739700 1 --2- 172.21.15.98:0/3919851352 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f91b0103950 0x7f91b01994d0 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:22:35.302 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:32.914+0000 7f91b6739700 1 -- 172.21.15.98:0/3919851352 >> 172.21.15.98:0/3919851352 conn(0x7f91b00ff330 msgr2=0x7f91b0101710 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:22:35.302 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:32.914+0000 7f91b6739700 1 -- 172.21.15.98:0/3919851352 shutdown_connections 2023-07-14T19:22:35.302 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:32.914+0000 7f91b6739700 1 -- 172.21.15.98:0/3919851352 wait complete. 2023-07-14T19:22:35.486 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Jul 14 19:22:35 smithi098 ceph-9adea648-227b-11ee-9b33-001a4aab830c-mgr-x[127734]: 2023-07-14T19:22:35.097+0000 7f0f199f7200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2023-07-14T19:22:35.487 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Jul 14 19:22:35 smithi098 ceph-9adea648-227b-11ee-9b33-001a4aab830c-mgr-x[127734]: 2023-07-14T19:22:35.265+0000 7f0f199f7200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-07-14T19:22:36.353 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:22:36 smithi098 ceph-mon[127486]: from='client.? 172.21.15.98:0/1370248233' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2023-07-14T19:22:36.354 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Jul 14 19:22:36 smithi098 ceph-9adea648-227b-11ee-9b33-001a4aab830c-mgr-x[127734]: 2023-07-14T19:22:36.077+0000 7f0f199f7200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-07-14T19:22:36.354 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Jul 14 19:22:36 smithi098 ceph-9adea648-227b-11ee-9b33-001a4aab830c-mgr-x[127734]: 2023-07-14T19:22:36.175+0000 7f0f199f7200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-07-14T19:22:36.429 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout { 2023-07-14T19:22:36.430 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "epoch": 5, 2023-07-14T19:22:36.430 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "available": true, 2023-07-14T19:22:36.431 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "active_name": "x", 2023-07-14T19:22:36.431 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "num_standby": 0 2023-07-14T19:22:36.431 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout } 2023-07-14T19:22:36.431 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:35.790+0000 7fafac46d700 1 Processor -- start 2023-07-14T19:22:36.431 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:35.790+0000 7fafac46d700 1 -- start start 2023-07-14T19:22:36.431 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:35.790+0000 7fafac46d700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fafa4073e60 0x7fafa410a560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:22:36.431 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:35.790+0000 7fafac46d700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fafa4105f50 con 0x7fafa4073e60 2023-07-14T19:22:36.432 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:35.790+0000 7fafaa209700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fafa4073e60 0x7fafa410a560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:22:36.432 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:35.790+0000 7fafaa209700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fafa4073e60 0x7fafa410a560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:44398/0 (socket says 172.21.15.98:44398) 2023-07-14T19:22:36.432 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:35.790+0000 7fafaa209700 1 -- 172.21.15.98:0/2264035055 learned_addr learned my addr 172.21.15.98:0/2264035055 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-07-14T19:22:36.435 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:35.791+0000 7fafaa209700 1 -- 172.21.15.98:0/2264035055 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fafa410aaa0 con 0x7fafa4073e60 2023-07-14T19:22:36.436 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:35.791+0000 7fafaa209700 1 --2- 172.21.15.98:0/2264035055 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fafa4073e60 0x7fafa410a560 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7fafa00098a0 tx=0x7fafa0009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=656de3630d2113e7 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:22:36.436 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:35.791+0000 7fafa9207700 1 -- 172.21.15.98:0/2264035055 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fafa0004160 con 0x7fafa4073e60 2023-07-14T19:22:36.436 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:35.791+0000 7fafa9207700 1 -- 172.21.15.98:0/2264035055 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fafa00042c0 con 0x7fafa4073e60 2023-07-14T19:22:36.437 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:35.791+0000 7fafa9207700 1 -- 172.21.15.98:0/2264035055 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fafa0011a00 con 0x7fafa4073e60 2023-07-14T19:22:36.437 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:35.792+0000 7fafac46d700 1 -- 172.21.15.98:0/2264035055 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fafa4073e60 msgr2=0x7fafa410a560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:22:36.437 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:35.792+0000 7fafac46d700 1 --2- 172.21.15.98:0/2264035055 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fafa4073e60 0x7fafa410a560 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7fafa00098a0 tx=0x7fafa0009bb0 comp rx=0 tx=0).stop 2023-07-14T19:22:36.437 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:35.792+0000 7fafac46d700 1 -- 172.21.15.98:0/2264035055 shutdown_connections 2023-07-14T19:22:36.438 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:35.792+0000 7fafac46d700 1 --2- 172.21.15.98:0/2264035055 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fafa4073e60 0x7fafa410a560 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:22:36.438 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:35.792+0000 7fafac46d700 1 -- 172.21.15.98:0/2264035055 >> 172.21.15.98:0/2264035055 conn(0x7fafa40ffb50 msgr2=0x7fafa4101f60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:22:36.438 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:35.792+0000 7fafac46d700 1 -- 172.21.15.98:0/2264035055 shutdown_connections 2023-07-14T19:22:36.438 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:35.792+0000 7fafac46d700 1 -- 172.21.15.98:0/2264035055 wait complete. 2023-07-14T19:22:36.438 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:35.792+0000 7fafac46d700 1 Processor -- start 2023-07-14T19:22:36.438 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:35.792+0000 7fafac46d700 1 -- start start 2023-07-14T19:22:36.438 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:35.792+0000 7fafac46d700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fafa419b510 0x7fafa419b920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:22:36.439 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:35.792+0000 7fafac46d700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fafa419be60 con 0x7fafa419b510 2023-07-14T19:22:36.439 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:35.793+0000 7fafaa209700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fafa419b510 0x7fafa419b920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:22:36.439 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:35.793+0000 7fafaa209700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fafa419b510 0x7fafa419b920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:44400/0 (socket says 172.21.15.98:44400) 2023-07-14T19:22:36.439 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:35.793+0000 7fafaa209700 1 -- 172.21.15.98:0/1370248233 learned_addr learned my addr 172.21.15.98:0/1370248233 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-07-14T19:22:36.439 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:35.793+0000 7fafaa209700 1 -- 172.21.15.98:0/1370248233 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fafa0009550 con 0x7fafa419b510 2023-07-14T19:22:36.439 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:35.793+0000 7fafaa209700 1 --2- 172.21.15.98:0/1370248233 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fafa419b510 0x7fafa419b920 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7fafa0006af0 tx=0x7fafa0011ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:22:36.439 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:35.793+0000 7faf9b7fe700 1 -- 172.21.15.98:0/1370248233 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fafa0003a40 con 0x7fafa419b510 2023-07-14T19:22:36.440 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:35.793+0000 7fafac46d700 1 -- 172.21.15.98:0/1370248233 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fafa419c060 con 0x7fafa419b510 2023-07-14T19:22:36.440 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:35.793+0000 7faf9b7fe700 1 -- 172.21.15.98:0/1370248233 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fafa0003ba0 con 0x7fafa419b510 2023-07-14T19:22:36.440 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:35.793+0000 7faf9b7fe700 1 -- 172.21.15.98:0/1370248233 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fafa001a700 con 0x7fafa419b510 2023-07-14T19:22:36.440 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:35.793+0000 7fafac46d700 1 -- 172.21.15.98:0/1370248233 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fafa419ecc0 con 0x7fafa419b510 2023-07-14T19:22:36.440 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:35.794+0000 7fafac46d700 1 -- 172.21.15.98:0/1370248233 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fafa4073e60 con 0x7fafa419b510 2023-07-14T19:22:36.440 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:35.794+0000 7faf9b7fe700 1 -- 172.21.15.98:0/1370248233 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) v1 ==== 45483+0+0 (secure 0 0 0) 0x7fafa0024ad0 con 0x7fafa419b510 2023-07-14T19:22:36.440 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:35.794+0000 7faf9b7fe700 1 --2- 172.21.15.98:0/1370248233 >> [v2:172.21.15.98:6800/749234665,v1:172.21.15.98:6801/749234665] conn(0x7faf90038710 0x7faf9003abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:22:36.441 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:35.795+0000 7faf9b7fe700 1 -- 172.21.15.98:0/1370248233 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7fafa004c720 con 0x7fafa419b510 2023-07-14T19:22:36.441 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:35.795+0000 7fafa9a08700 1 -- 172.21.15.98:0/1370248233 >> [v2:172.21.15.98:6800/749234665,v1:172.21.15.98:6801/749234665] conn(0x7faf90038710 msgr2=0x7faf9003abc0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.98:6800/749234665 2023-07-14T19:22:36.441 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:35.795+0000 7fafa9a08700 1 --2- 172.21.15.98:0/1370248233 >> [v2:172.21.15.98:6800/749234665,v1:172.21.15.98:6801/749234665] conn(0x7faf90038710 0x7faf9003abc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-07-14T19:22:36.441 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:35.799+0000 7faf9b7fe700 1 -- 172.21.15.98:0/1370248233 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7fafa0018b80 con 0x7fafa419b510 2023-07-14T19:22:36.441 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:35.996+0000 7fafa9a08700 1 -- 172.21.15.98:0/1370248233 >> [v2:172.21.15.98:6800/749234665,v1:172.21.15.98:6801/749234665] conn(0x7faf90038710 msgr2=0x7faf9003abc0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.98:6800/749234665 2023-07-14T19:22:36.441 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:35.996+0000 7fafa9a08700 1 --2- 172.21.15.98:0/1370248233 >> [v2:172.21.15.98:6800/749234665,v1:172.21.15.98:6801/749234665] conn(0x7faf90038710 0x7faf9003abc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-07-14T19:22:36.441 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:36.045+0000 7fafac46d700 1 -- 172.21.15.98:0/1370248233 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "mgr stat"} v 0) v1 -- 0x7fafa419ee50 con 0x7fafa419b510 2023-07-14T19:22:36.442 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:36.045+0000 7faf9b7fe700 1 -- 172.21.15.98:0/1370248233 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "mgr stat"}]=0 v5) v1 ==== 56+0+88 (secure 0 0 0) 0x7fafa0018350 con 0x7fafa419b510 2023-07-14T19:22:36.442 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:36.047+0000 7fafac46d700 1 -- 172.21.15.98:0/1370248233 >> [v2:172.21.15.98:6800/749234665,v1:172.21.15.98:6801/749234665] conn(0x7faf90038710 msgr2=0x7faf9003abc0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-07-14T19:22:36.442 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:36.047+0000 7fafac46d700 1 --2- 172.21.15.98:0/1370248233 >> [v2:172.21.15.98:6800/749234665,v1:172.21.15.98:6801/749234665] conn(0x7faf90038710 0x7faf9003abc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:22:36.442 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:36.047+0000 7fafac46d700 1 -- 172.21.15.98:0/1370248233 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fafa419b510 msgr2=0x7fafa419b920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:22:36.442 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:36.047+0000 7fafac46d700 1 --2- 172.21.15.98:0/1370248233 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fafa419b510 0x7fafa419b920 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7fafa0006af0 tx=0x7fafa0011ad0 comp rx=0 tx=0).stop 2023-07-14T19:22:36.442 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:36.047+0000 7fafac46d700 1 -- 172.21.15.98:0/1370248233 shutdown_connections 2023-07-14T19:22:36.442 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:36.047+0000 7fafac46d700 1 --2- 172.21.15.98:0/1370248233 >> [v2:172.21.15.98:6800/749234665,v1:172.21.15.98:6801/749234665] conn(0x7faf90038710 0x7faf9003abc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:22:36.442 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:36.047+0000 7fafac46d700 1 --2- 172.21.15.98:0/1370248233 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fafa419b510 0x7fafa419b920 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:22:36.443 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:36.047+0000 7fafac46d700 1 -- 172.21.15.98:0/1370248233 >> 172.21.15.98:0/1370248233 conn(0x7fafa40ffb50 msgr2=0x7fafa4101220 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:22:36.443 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:36.048+0000 7fafac46d700 1 -- 172.21.15.98:0/1370248233 shutdown_connections 2023-07-14T19:22:36.443 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:36.048+0000 7fafac46d700 1 -- 172.21.15.98:0/1370248233 wait complete. 2023-07-14T19:22:36.443 INFO:teuthology.orchestra.run.smithi098.stdout:Waiting for the mgr to restart... 2023-07-14T19:22:36.443 INFO:teuthology.orchestra.run.smithi098.stdout:Waiting for mgr epoch 5... 2023-07-14T19:22:36.643 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Jul 14 19:22:36 smithi098 ceph-9adea648-227b-11ee-9b33-001a4aab830c-mgr-x[127734]: 2023-07-14T19:22:36.351+0000 7f0f199f7200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-07-14T19:22:37.028 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Jul 14 19:22:36 smithi098 ceph-9adea648-227b-11ee-9b33-001a4aab830c-mgr-x[127734]: 2023-07-14T19:22:36.641+0000 7f0f199f7200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2023-07-14T19:22:37.028 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Jul 14 19:22:36 smithi098 ceph-9adea648-227b-11ee-9b33-001a4aab830c-mgr-x[127734]: 2023-07-14T19:22:36.875+0000 7f0f199f7200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-07-14T19:22:37.281 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Jul 14 19:22:37 smithi098 ceph-9adea648-227b-11ee-9b33-001a4aab830c-mgr-x[127734]: 2023-07-14T19:22:37.026+0000 7f0f199f7200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-07-14T19:22:37.781 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Jul 14 19:22:37 smithi098 ceph-9adea648-227b-11ee-9b33-001a4aab830c-mgr-x[127734]: 2023-07-14T19:22:37.285+0000 7f0f199f7200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-07-14T19:22:38.281 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Jul 14 19:22:37 smithi098 ceph-9adea648-227b-11ee-9b33-001a4aab830c-mgr-x[127734]: 2023-07-14T19:22:37.975+0000 7f0f199f7200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-07-14T19:22:38.282 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Jul 14 19:22:38 smithi098 ceph-9adea648-227b-11ee-9b33-001a4aab830c-mgr-x[127734]: 2023-07-14T19:22:38.106+0000 7f0f199f7200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-07-14T19:22:38.627 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Jul 14 19:22:38 smithi098 ceph-9adea648-227b-11ee-9b33-001a4aab830c-mgr-x[127734]: 2023-07-14T19:22:38.297+0000 7f0f199f7200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-07-14T19:22:38.627 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Jul 14 19:22:38 smithi098 ceph-9adea648-227b-11ee-9b33-001a4aab830c-mgr-x[127734]: 2023-07-14T19:22:38.411+0000 7f0f199f7200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-07-14T19:22:39.031 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Jul 14 19:22:38 smithi098 ceph-9adea648-227b-11ee-9b33-001a4aab830c-mgr-x[127734]: 2023-07-14T19:22:38.625+0000 7f0f199f7200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-07-14T19:22:39.781 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Jul 14 19:22:39 smithi098 ceph-9adea648-227b-11ee-9b33-001a4aab830c-mgr-x[127734]: 2023-07-14T19:22:39.467+0000 7f0f199f7200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-07-14T19:22:39.782 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:22:39 smithi098 ceph-mon[127486]: Active manager daemon x restarted 2023-07-14T19:22:39.782 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:22:39 smithi098 ceph-mon[127486]: Activating manager daemon x 2023-07-14T19:22:39.782 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:22:39 smithi098 ceph-mon[127486]: osdmap e2: 0 total, 0 up, 0 in 2023-07-14T19:22:39.782 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:22:39 smithi098 ceph-mon[127486]: mgrmap e6: x(active, starting, since 0.00331353s) 2023-07-14T19:22:39.782 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:22:39 smithi098 ceph-mon[127486]: from='mgr.14120 172.21.15.98:0/3060674050' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2023-07-14T19:22:39.782 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:22:39 smithi098 ceph-mon[127486]: from='mgr.14120 172.21.15.98:0/3060674050' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2023-07-14T19:22:39.783 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:22:39 smithi098 ceph-mon[127486]: from='mgr.14120 172.21.15.98:0/3060674050' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2023-07-14T19:22:39.783 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:22:39 smithi098 ceph-mon[127486]: from='mgr.14120 172.21.15.98:0/3060674050' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2023-07-14T19:22:39.783 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:22:39 smithi098 ceph-mon[127486]: from='mgr.14120 172.21.15.98:0/3060674050' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2023-07-14T19:22:39.783 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:22:39 smithi098 ceph-mon[127486]: Manager daemon x is now available 2023-07-14T19:22:39.783 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:22:39 smithi098 ceph-mon[127486]: from='mgr.14120 172.21.15.98:0/3060674050' entity='mgr.x' 2023-07-14T19:22:39.783 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:22:39 smithi098 ceph-mon[127486]: from='mgr.14120 172.21.15.98:0/3060674050' entity='mgr.x' 2023-07-14T19:22:39.783 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:22:39 smithi098 ceph-mon[127486]: from='mgr.14120 172.21.15.98:0/3060674050' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-14T19:22:39.784 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:22:39 smithi098 ceph-mon[127486]: from='mgr.14120 172.21.15.98:0/3060674050' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-14T19:22:39.784 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:22:39 smithi098 ceph-mon[127486]: from='mgr.14120 172.21.15.98:0/3060674050' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-14T19:22:40.781 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:22:40 smithi098 ceph-mon[127486]: Found migration_current of "None". Setting to last migration. 2023-07-14T19:22:40.781 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:22:40 smithi098 ceph-mon[127486]: from='mgr.14120 172.21.15.98:0/3060674050' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2023-07-14T19:22:40.782 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:22:40 smithi098 ceph-mon[127486]: from='mgr.14120 172.21.15.98:0/3060674050' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2023-07-14T19:22:40.782 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:22:40 smithi098 ceph-mon[127486]: from='mgr.14120 172.21.15.98:0/3060674050' entity='mgr.x' 2023-07-14T19:22:40.782 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:22:40 smithi098 ceph-mon[127486]: from='mgr.14120 172.21.15.98:0/3060674050' entity='mgr.x' 2023-07-14T19:22:40.782 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:22:40 smithi098 ceph-mon[127486]: from='mgr.14120 172.21.15.98:0/3060674050' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-14T19:22:40.782 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:22:40 smithi098 ceph-mon[127486]: mgrmap e7: x(active, since 1.0055s) 2023-07-14T19:22:40.889 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout { 2023-07-14T19:22:40.890 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "mgrmap_epoch": 7, 2023-07-14T19:22:40.890 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "initialized": true 2023-07-14T19:22:40.890 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout } 2023-07-14T19:22:40.890 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:36.933+0000 7fb90702b700 1 Processor -- start 2023-07-14T19:22:40.890 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:36.933+0000 7fb90702b700 1 -- start start 2023-07-14T19:22:40.890 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:36.933+0000 7fb90702b700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb900103950 0x7fb900103d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:22:40.891 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:36.933+0000 7fb90702b700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb9001042a0 con 0x7fb900103950 2023-07-14T19:22:40.891 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:36.934+0000 7fb904dc7700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb900103950 0x7fb900103d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:22:40.891 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:36.934+0000 7fb904dc7700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb900103950 0x7fb900103d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:44408/0 (socket says 172.21.15.98:44408) 2023-07-14T19:22:40.891 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:36.934+0000 7fb904dc7700 1 -- 172.21.15.98:0/208303728 learned_addr learned my addr 172.21.15.98:0/208303728 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-07-14T19:22:40.891 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:36.934+0000 7fb904dc7700 1 -- 172.21.15.98:0/208303728 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb9001043e0 con 0x7fb900103950 2023-07-14T19:22:40.891 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:36.934+0000 7fb904dc7700 1 --2- 172.21.15.98:0/208303728 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb900103950 0x7fb900103d60 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7fb8f40098a0 tx=0x7fb8f4009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b1f6745c8ecb7a6 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:22:40.891 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:36.934+0000 7fb8ff7fe700 1 -- 172.21.15.98:0/208303728 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fb8f4004160 con 0x7fb900103950 2023-07-14T19:22:40.892 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:36.934+0000 7fb8ff7fe700 1 -- 172.21.15.98:0/208303728 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fb8f40042c0 con 0x7fb900103950 2023-07-14T19:22:40.892 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:36.934+0000 7fb8ff7fe700 1 -- 172.21.15.98:0/208303728 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fb8f4011a00 con 0x7fb900103950 2023-07-14T19:22:40.892 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:36.935+0000 7fb90702b700 1 -- 172.21.15.98:0/208303728 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb900103950 msgr2=0x7fb900103d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:22:40.892 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:36.935+0000 7fb90702b700 1 --2- 172.21.15.98:0/208303728 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb900103950 0x7fb900103d60 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7fb8f40098a0 tx=0x7fb8f4009bb0 comp rx=0 tx=0).stop 2023-07-14T19:22:40.892 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:36.935+0000 7fb90702b700 1 -- 172.21.15.98:0/208303728 shutdown_connections 2023-07-14T19:22:40.892 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:36.935+0000 7fb90702b700 1 --2- 172.21.15.98:0/208303728 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb900103950 0x7fb900103d60 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:22:40.893 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:36.935+0000 7fb90702b700 1 -- 172.21.15.98:0/208303728 >> 172.21.15.98:0/208303728 conn(0x7fb9000ff330 msgr2=0x7fb900101760 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:22:40.893 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:36.935+0000 7fb90702b700 1 -- 172.21.15.98:0/208303728 shutdown_connections 2023-07-14T19:22:40.893 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:36.935+0000 7fb90702b700 1 -- 172.21.15.98:0/208303728 wait complete. 2023-07-14T19:22:40.893 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:36.936+0000 7fb90702b700 1 Processor -- start 2023-07-14T19:22:40.894 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:36.936+0000 7fb90702b700 1 -- start start 2023-07-14T19:22:40.894 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:36.936+0000 7fb90702b700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb900103950 0x7fb90019b6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:22:40.894 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:36.936+0000 7fb90702b700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb90019bc00 con 0x7fb900103950 2023-07-14T19:22:40.895 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:36.936+0000 7fb904dc7700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb900103950 0x7fb90019b6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:22:40.895 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:36.936+0000 7fb904dc7700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb900103950 0x7fb90019b6c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:44410/0 (socket says 172.21.15.98:44410) 2023-07-14T19:22:40.895 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:36.936+0000 7fb904dc7700 1 -- 172.21.15.98:0/1523464984 learned_addr learned my addr 172.21.15.98:0/1523464984 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-07-14T19:22:40.896 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:36.936+0000 7fb904dc7700 1 -- 172.21.15.98:0/1523464984 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb8f4009550 con 0x7fb900103950 2023-07-14T19:22:40.896 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:36.936+0000 7fb904dc7700 1 --2- 172.21.15.98:0/1523464984 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb900103950 0x7fb90019b6c0 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7fb8f4009fb0 tx=0x7fb8f4011ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:22:40.896 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:36.936+0000 7fb8fdffb700 1 -- 172.21.15.98:0/1523464984 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fb8f4003a40 con 0x7fb900103950 2023-07-14T19:22:40.896 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:36.936+0000 7fb90702b700 1 -- 172.21.15.98:0/1523464984 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb90019be00 con 0x7fb900103950 2023-07-14T19:22:40.896 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:36.936+0000 7fb8fdffb700 1 -- 172.21.15.98:0/1523464984 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fb8f4003ba0 con 0x7fb900103950 2023-07-14T19:22:40.896 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:36.936+0000 7fb8fdffb700 1 -- 172.21.15.98:0/1523464984 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fb8f401a750 con 0x7fb900103950 2023-07-14T19:22:40.896 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:36.937+0000 7fb90702b700 1 -- 172.21.15.98:0/1523464984 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb90019c220 con 0x7fb900103950 2023-07-14T19:22:40.897 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:36.937+0000 7fb8fdffb700 1 -- 172.21.15.98:0/1523464984 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) v1 ==== 45483+0+0 (secure 0 0 0) 0x7fb8f401a980 con 0x7fb900103950 2023-07-14T19:22:40.897 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:36.938+0000 7fb8fdffb700 1 --2- 172.21.15.98:0/1523464984 >> [v2:172.21.15.98:6800/749234665,v1:172.21.15.98:6801/749234665] conn(0x7fb8f0038710 0x7fb8f003abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:22:40.897 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:36.938+0000 7fb90702b700 1 -- 172.21.15.98:0/1523464984 --> [v2:172.21.15.98:6800/749234665,v1:172.21.15.98:6801/749234665] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x7fb900061c70 con 0x7fb8f0038710 2023-07-14T19:22:40.897 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:36.938+0000 7fb8fdffb700 1 -- 172.21.15.98:0/1523464984 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7fb8f404c350 con 0x7fb900103950 2023-07-14T19:22:40.897 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:36.938+0000 7fb8fffff700 1 -- 172.21.15.98:0/1523464984 >> [v2:172.21.15.98:6800/749234665,v1:172.21.15.98:6801/749234665] conn(0x7fb8f0038710 msgr2=0x7fb8f003abc0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.98:6800/749234665 2023-07-14T19:22:40.897 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:36.938+0000 7fb8fffff700 1 --2- 172.21.15.98:0/1523464984 >> [v2:172.21.15.98:6800/749234665,v1:172.21.15.98:6801/749234665] conn(0x7fb8f0038710 0x7fb8f003abc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-07-14T19:22:40.897 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:37.139+0000 7fb8fffff700 1 -- 172.21.15.98:0/1523464984 >> [v2:172.21.15.98:6800/749234665,v1:172.21.15.98:6801/749234665] conn(0x7fb8f0038710 msgr2=0x7fb8f003abc0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.98:6800/749234665 2023-07-14T19:22:40.898 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:37.139+0000 7fb8fffff700 1 --2- 172.21.15.98:0/1523464984 >> [v2:172.21.15.98:6800/749234665,v1:172.21.15.98:6801/749234665] conn(0x7fb8f0038710 0x7fb8f003abc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-07-14T19:22:40.898 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:37.540+0000 7fb8fffff700 1 -- 172.21.15.98:0/1523464984 >> [v2:172.21.15.98:6800/749234665,v1:172.21.15.98:6801/749234665] conn(0x7fb8f0038710 msgr2=0x7fb8f003abc0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.98:6800/749234665 2023-07-14T19:22:40.898 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:37.540+0000 7fb8fffff700 1 --2- 172.21.15.98:0/1523464984 >> [v2:172.21.15.98:6800/749234665,v1:172.21.15.98:6801/749234665] conn(0x7fb8f0038710 0x7fb8f003abc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-07-14T19:22:40.898 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:38.341+0000 7fb8fffff700 1 -- 172.21.15.98:0/1523464984 >> [v2:172.21.15.98:6800/749234665,v1:172.21.15.98:6801/749234665] conn(0x7fb8f0038710 msgr2=0x7fb8f003abc0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.98:6800/749234665 2023-07-14T19:22:40.898 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:38.341+0000 7fb8fffff700 1 --2- 172.21.15.98:0/1523464984 >> [v2:172.21.15.98:6800/749234665,v1:172.21.15.98:6801/749234665] conn(0x7fb8f0038710 0x7fb8f003abc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-07-14T19:22:40.898 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:39.482+0000 7fb8fdffb700 1 -- 172.21.15.98:0/1523464984 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mgrmap(e 6) v1 ==== 45250+0+0 (secure 0 0 0) 0x7fb8f404b4d0 con 0x7fb900103950 2023-07-14T19:22:40.898 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:39.482+0000 7fb8fdffb700 1 -- 172.21.15.98:0/1523464984 >> [v2:172.21.15.98:6800/749234665,v1:172.21.15.98:6801/749234665] conn(0x7fb8f0038710 msgr2=0x7fb8f003abc0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-07-14T19:22:40.899 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:39.482+0000 7fb8fdffb700 1 --2- 172.21.15.98:0/1523464984 >> [v2:172.21.15.98:6800/749234665,v1:172.21.15.98:6801/749234665] conn(0x7fb8f0038710 0x7fb8f003abc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:22:40.899 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:40.483+0000 7fb8fdffb700 1 -- 172.21.15.98:0/1523464984 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mgrmap(e 7) v1 ==== 45377+0+0 (secure 0 0 0) 0x7fb8f402d3f0 con 0x7fb900103950 2023-07-14T19:22:40.899 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:40.483+0000 7fb8fdffb700 1 --2- 172.21.15.98:0/1523464984 >> [v2:172.21.15.98:6800/1732769087,v1:172.21.15.98:6801/1732769087] conn(0x7fb8f003be30 0x7fb8f003e210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:22:40.899 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:40.483+0000 7fb8fdffb700 1 -- 172.21.15.98:0/1523464984 --> [v2:172.21.15.98:6800/1732769087,v1:172.21.15.98:6801/1732769087] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x7fb900061c70 con 0x7fb8f003be30 2023-07-14T19:22:40.899 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:40.484+0000 7fb8fffff700 1 --2- 172.21.15.98:0/1523464984 >> [v2:172.21.15.98:6800/1732769087,v1:172.21.15.98:6801/1732769087] conn(0x7fb8f003be30 0x7fb8f003e210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:22:40.899 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:40.484+0000 7fb8fffff700 1 --2- 172.21.15.98:0/1523464984 >> [v2:172.21.15.98:6800/1732769087,v1:172.21.15.98:6801/1732769087] conn(0x7fb8f003be30 0x7fb8f003e210 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 crypto rx=0x7fb8ec003a10 tx=0x7fb8ec0092b0 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:22:40.899 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:40.485+0000 7fb8fdffb700 1 -- 172.21.15.98:0/1523464984 <== mgr.14120 v2:172.21.15.98:6800/1732769087 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+6910 (secure 0 0 0) 0x7fb900061c70 con 0x7fb8f003be30 2023-07-14T19:22:40.899 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:40.491+0000 7fb90702b700 1 -- 172.21.15.98:0/1523464984 --> [v2:172.21.15.98:6800/1732769087,v1:172.21.15.98:6801/1732769087] -- command(tid 1: {"prefix": "mgr_status"}) v1 -- 0x7fb90019ca00 con 0x7fb8f003be30 2023-07-14T19:22:40.900 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:40.491+0000 7fb8fdffb700 1 -- 172.21.15.98:0/1523464984 <== mgr.14120 v2:172.21.15.98:6800/1732769087 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+51 (secure 0 0 0) 0x7fb90019ca00 con 0x7fb8f003be30 2023-07-14T19:22:40.900 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:40.492+0000 7fb90702b700 1 -- 172.21.15.98:0/1523464984 >> [v2:172.21.15.98:6800/1732769087,v1:172.21.15.98:6801/1732769087] conn(0x7fb8f003be30 msgr2=0x7fb8f003e210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:22:40.900 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:40.492+0000 7fb90702b700 1 --2- 172.21.15.98:0/1523464984 >> [v2:172.21.15.98:6800/1732769087,v1:172.21.15.98:6801/1732769087] conn(0x7fb8f003be30 0x7fb8f003e210 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 crypto rx=0x7fb8ec003a10 tx=0x7fb8ec0092b0 comp rx=0 tx=0).stop 2023-07-14T19:22:40.900 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:40.492+0000 7fb90702b700 1 -- 172.21.15.98:0/1523464984 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb900103950 msgr2=0x7fb90019b6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:22:40.900 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:40.492+0000 7fb90702b700 1 --2- 172.21.15.98:0/1523464984 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb900103950 0x7fb90019b6c0 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7fb8f4009fb0 tx=0x7fb8f4011ad0 comp rx=0 tx=0).stop 2023-07-14T19:22:40.900 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:40.492+0000 7fb90702b700 1 -- 172.21.15.98:0/1523464984 shutdown_connections 2023-07-14T19:22:40.900 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:40.492+0000 7fb90702b700 1 --2- 172.21.15.98:0/1523464984 >> [v2:172.21.15.98:6800/1732769087,v1:172.21.15.98:6801/1732769087] conn(0x7fb8f003be30 0x7fb8f003e210 unknown :-1 s=CLOSED pgs=3 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:22:40.901 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:40.492+0000 7fb90702b700 1 --2- 172.21.15.98:0/1523464984 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb900103950 0x7fb90019b6c0 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:22:40.901 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:40.492+0000 7fb90702b700 1 --2- 172.21.15.98:0/1523464984 >> [v2:172.21.15.98:6800/749234665,v1:172.21.15.98:6801/749234665] conn(0x7fb8f0038710 0x7fb8f003abc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:22:40.901 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:40.492+0000 7fb90702b700 1 -- 172.21.15.98:0/1523464984 >> 172.21.15.98:0/1523464984 conn(0x7fb9000ff330 msgr2=0x7fb900109100 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:22:40.901 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:40.492+0000 7fb90702b700 1 -- 172.21.15.98:0/1523464984 shutdown_connections 2023-07-14T19:22:40.901 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:40.492+0000 7fb90702b700 1 -- 172.21.15.98:0/1523464984 wait complete. 2023-07-14T19:22:40.901 INFO:teuthology.orchestra.run.smithi098.stdout:mgr epoch 5 is available 2023-07-14T19:22:40.901 INFO:teuthology.orchestra.run.smithi098.stdout:Setting orchestrator backend to cephadm... 2023-07-14T19:22:41.925 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:22:41 smithi098 ceph-mon[127486]: [14/Jul/2023:19:22:40] ENGINE Bus STARTING 2023-07-14T19:22:41.925 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:22:41 smithi098 ceph-mon[127486]: [14/Jul/2023:19:22:40] ENGINE Serving on http://172.21.15.98:8765 2023-07-14T19:22:41.925 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:22:41 smithi098 ceph-mon[127486]: [14/Jul/2023:19:22:40] ENGINE Serving on https://172.21.15.98:7150 2023-07-14T19:22:41.925 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:22:41 smithi098 ceph-mon[127486]: [14/Jul/2023:19:22:40] ENGINE Bus STARTED 2023-07-14T19:22:41.926 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:22:41 smithi098 ceph-mon[127486]: from='client.14124 -' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2023-07-14T19:22:41.926 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:22:41 smithi098 ceph-mon[127486]: from='client.14124 -' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2023-07-14T19:22:41.932 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:41.344+0000 7fcd5af6e700 1 Processor -- start 2023-07-14T19:22:41.932 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:41.344+0000 7fcd5af6e700 1 -- start start 2023-07-14T19:22:41.932 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:41.345+0000 7fcd5af6e700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcd54104570 0x7fcd54104980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:22:41.933 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:41.345+0000 7fcd5af6e700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcd54104ec0 con 0x7fcd54104570 2023-07-14T19:22:41.933 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:41.345+0000 7fcd58d0a700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcd54104570 0x7fcd54104980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:22:41.933 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:41.345+0000 7fcd58d0a700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcd54104570 0x7fcd54104980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:44458/0 (socket says 172.21.15.98:44458) 2023-07-14T19:22:41.933 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:41.345+0000 7fcd58d0a700 1 -- 172.21.15.98:0/2187465628 learned_addr learned my addr 172.21.15.98:0/2187465628 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-07-14T19:22:41.933 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:41.345+0000 7fcd58d0a700 1 -- 172.21.15.98:0/2187465628 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcd54105000 con 0x7fcd54104570 2023-07-14T19:22:41.933 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:41.345+0000 7fcd58d0a700 1 --2- 172.21.15.98:0/2187465628 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcd54104570 0x7fcd54104980 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7fcd4800bb40 tx=0x7fcd4800be50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=58266dd1254f5414 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:22:41.933 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:41.345+0000 7fcd537fe700 1 -- 172.21.15.98:0/2187465628 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fcd4800dbe0 con 0x7fcd54104570 2023-07-14T19:22:41.934 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:41.345+0000 7fcd537fe700 1 -- 172.21.15.98:0/2187465628 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fcd480044d0 con 0x7fcd54104570 2023-07-14T19:22:41.934 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:41.345+0000 7fcd537fe700 1 -- 172.21.15.98:0/2187465628 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fcd48010470 con 0x7fcd54104570 2023-07-14T19:22:41.934 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:41.346+0000 7fcd5af6e700 1 -- 172.21.15.98:0/2187465628 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcd54104570 msgr2=0x7fcd54104980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:22:41.934 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:41.346+0000 7fcd5af6e700 1 --2- 172.21.15.98:0/2187465628 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcd54104570 0x7fcd54104980 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7fcd4800bb40 tx=0x7fcd4800be50 comp rx=0 tx=0).stop 2023-07-14T19:22:41.934 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:41.346+0000 7fcd5af6e700 1 -- 172.21.15.98:0/2187465628 shutdown_connections 2023-07-14T19:22:41.934 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:41.346+0000 7fcd5af6e700 1 --2- 172.21.15.98:0/2187465628 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcd54104570 0x7fcd54104980 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:22:41.934 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:41.346+0000 7fcd5af6e700 1 -- 172.21.15.98:0/2187465628 >> 172.21.15.98:0/2187465628 conn(0x7fcd540ffbc0 msgr2=0x7fcd54101ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:22:41.935 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:41.346+0000 7fcd5af6e700 1 -- 172.21.15.98:0/2187465628 shutdown_connections 2023-07-14T19:22:41.935 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:41.346+0000 7fcd5af6e700 1 -- 172.21.15.98:0/2187465628 wait complete. 2023-07-14T19:22:41.935 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:41.346+0000 7fcd5af6e700 1 Processor -- start 2023-07-14T19:22:41.935 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:41.346+0000 7fcd5af6e700 1 -- start start 2023-07-14T19:22:41.935 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:41.347+0000 7fcd5af6e700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcd54104570 0x7fcd54197280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:22:41.935 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:41.347+0000 7fcd5af6e700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcd541977c0 con 0x7fcd54104570 2023-07-14T19:22:41.935 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:41.347+0000 7fcd58d0a700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcd54104570 0x7fcd54197280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:22:41.936 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:41.347+0000 7fcd58d0a700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcd54104570 0x7fcd54197280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:44460/0 (socket says 172.21.15.98:44460) 2023-07-14T19:22:41.936 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:41.347+0000 7fcd58d0a700 1 -- 172.21.15.98:0/695445730 learned_addr learned my addr 172.21.15.98:0/695445730 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-07-14T19:22:41.936 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:41.347+0000 7fcd58d0a700 1 -- 172.21.15.98:0/695445730 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcd4800b7f0 con 0x7fcd54104570 2023-07-14T19:22:41.936 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:41.347+0000 7fcd58d0a700 1 --2- 172.21.15.98:0/695445730 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcd54104570 0x7fcd54197280 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7fcd4800d650 tx=0x7fcd48003df0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:22:41.936 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:41.347+0000 7fcd51ffb700 1 -- 172.21.15.98:0/695445730 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fcd4801e070 con 0x7fcd54104570 2023-07-14T19:22:41.936 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:41.347+0000 7fcd51ffb700 1 -- 172.21.15.98:0/695445730 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fcd480042a0 con 0x7fcd54104570 2023-07-14T19:22:41.936 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:41.347+0000 7fcd51ffb700 1 -- 172.21.15.98:0/695445730 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fcd48022af0 con 0x7fcd54104570 2023-07-14T19:22:41.936 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:41.347+0000 7fcd5af6e700 1 -- 172.21.15.98:0/695445730 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcd541979c0 con 0x7fcd54104570 2023-07-14T19:22:41.937 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:41.347+0000 7fcd5af6e700 1 -- 172.21.15.98:0/695445730 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcd54197de0 con 0x7fcd54104570 2023-07-14T19:22:41.937 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:41.348+0000 7fcd51ffb700 1 -- 172.21.15.98:0/695445730 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 7) v1 ==== 45377+0+0 (secure 0 0 0) 0x7fcd48010730 con 0x7fcd54104570 2023-07-14T19:22:41.937 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:41.348+0000 7fcd51ffb700 1 --2- 172.21.15.98:0/695445730 >> [v2:172.21.15.98:6800/1732769087,v1:172.21.15.98:6801/1732769087] conn(0x7fcd440385f0 0x7fcd4403aaa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:22:41.938 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:41.348+0000 7fcd5af6e700 1 -- 172.21.15.98:0/695445730 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcd54190ea0 con 0x7fcd54104570 2023-07-14T19:22:41.938 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:41.349+0000 7fcd51ffb700 1 -- 172.21.15.98:0/695445730 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7fcd4804ec70 con 0x7fcd54104570 2023-07-14T19:22:41.938 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:41.349+0000 7fcd53fff700 1 --2- 172.21.15.98:0/695445730 >> [v2:172.21.15.98:6800/1732769087,v1:172.21.15.98:6801/1732769087] conn(0x7fcd440385f0 0x7fcd4403aaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:22:41.938 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:41.349+0000 7fcd53fff700 1 --2- 172.21.15.98:0/695445730 >> [v2:172.21.15.98:6800/1732769087,v1:172.21.15.98:6801/1732769087] conn(0x7fcd440385f0 0x7fcd4403aaa0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7fcd40006fd0 tx=0x7fcd40006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:22:41.938 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:41.353+0000 7fcd51ffb700 1 -- 172.21.15.98:0/695445730 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7fcd48027070 con 0x7fcd54104570 2023-07-14T19:22:41.938 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:41.555+0000 7fcd5af6e700 1 -- 172.21.15.98:0/695445730 --> [v2:172.21.15.98:6800/1732769087,v1:172.21.15.98:6801/1732769087] -- mgr_command(tid 0: {"prefix": "orch set backend", "module_name": "cephadm", "target": ["mon-mgr", ""]}) v1 -- 0x7fcd54060a80 con 0x7fcd440385f0 2023-07-14T19:22:41.938 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:41.561+0000 7fcd51ffb700 1 -- 172.21.15.98:0/695445730 <== mgr.14120 v2:172.21.15.98:6800/1732769087 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7fcd54060a80 con 0x7fcd440385f0 2023-07-14T19:22:41.939 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:41.563+0000 7fcd5af6e700 1 -- 172.21.15.98:0/695445730 >> [v2:172.21.15.98:6800/1732769087,v1:172.21.15.98:6801/1732769087] conn(0x7fcd440385f0 msgr2=0x7fcd4403aaa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:22:41.939 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:41.563+0000 7fcd5af6e700 1 --2- 172.21.15.98:0/695445730 >> [v2:172.21.15.98:6800/1732769087,v1:172.21.15.98:6801/1732769087] conn(0x7fcd440385f0 0x7fcd4403aaa0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7fcd40006fd0 tx=0x7fcd40006e40 comp rx=0 tx=0).stop 2023-07-14T19:22:41.939 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:41.563+0000 7fcd5af6e700 1 -- 172.21.15.98:0/695445730 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcd54104570 msgr2=0x7fcd54197280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:22:41.939 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:41.563+0000 7fcd5af6e700 1 --2- 172.21.15.98:0/695445730 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcd54104570 0x7fcd54197280 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7fcd4800d650 tx=0x7fcd48003df0 comp rx=0 tx=0).stop 2023-07-14T19:22:41.939 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:41.563+0000 7fcd5af6e700 1 -- 172.21.15.98:0/695445730 shutdown_connections 2023-07-14T19:22:41.940 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:41.563+0000 7fcd5af6e700 1 --2- 172.21.15.98:0/695445730 >> [v2:172.21.15.98:6800/1732769087,v1:172.21.15.98:6801/1732769087] conn(0x7fcd440385f0 0x7fcd4403aaa0 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:22:41.940 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:41.563+0000 7fcd5af6e700 1 --2- 172.21.15.98:0/695445730 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcd54104570 0x7fcd54197280 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:22:41.940 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:41.563+0000 7fcd5af6e700 1 -- 172.21.15.98:0/695445730 >> 172.21.15.98:0/695445730 conn(0x7fcd540ffbc0 msgr2=0x7fcd5418dfe0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:22:41.940 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:41.563+0000 7fcd5af6e700 1 -- 172.21.15.98:0/695445730 shutdown_connections 2023-07-14T19:22:41.940 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:41.563+0000 7fcd5af6e700 1 -- 172.21.15.98:0/695445730 wait complete. 2023-07-14T19:22:42.813 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:22:42 smithi098 ceph-mon[127486]: from='client.14132 -' entity='client.admin' cmd=[{"prefix": "orch set backend", "module_name": "cephadm", "target": ["mon-mgr", ""]}]: dispatch 2023-07-14T19:22:42.813 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:22:42 smithi098 ceph-mon[127486]: from='mgr.14120 172.21.15.98:0/3060674050' entity='mgr.x' 2023-07-14T19:22:42.813 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:22:42 smithi098 ceph-mon[127486]: from='mgr.14120 172.21.15.98:0/3060674050' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-14T19:22:42.813 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:22:42 smithi098 ceph-mon[127486]: mgrmap e8: x(active, since 2s) 2023-07-14T19:22:43.075 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout value unchanged 2023-07-14T19:22:43.075 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:42.429+0000 7f983c82e700 1 Processor -- start 2023-07-14T19:22:43.075 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:42.429+0000 7f983c82e700 1 -- start start 2023-07-14T19:22:43.075 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:42.429+0000 7f983c82e700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9838106c80 0x7f9838107090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:22:43.076 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:42.429+0000 7f983c82e700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f98381075d0 con 0x7f9838106c80 2023-07-14T19:22:43.076 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:42.430+0000 7f983659c700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9838106c80 0x7f9838107090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:22:43.076 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:42.430+0000 7f983659c700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9838106c80 0x7f9838107090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:44464/0 (socket says 172.21.15.98:44464) 2023-07-14T19:22:43.076 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:42.430+0000 7f983659c700 1 -- 172.21.15.98:0/2774354282 learned_addr learned my addr 172.21.15.98:0/2774354282 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-07-14T19:22:43.076 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:42.430+0000 7f983659c700 1 -- 172.21.15.98:0/2774354282 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9838107710 con 0x7f9838106c80 2023-07-14T19:22:43.076 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:42.430+0000 7f983659c700 1 --2- 172.21.15.98:0/2774354282 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9838106c80 0x7f9838107090 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f982c0098a0 tx=0x7f982c009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a23e2621337c9c26 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:22:43.076 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:42.430+0000 7f983559a700 1 -- 172.21.15.98:0/2774354282 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f982c004160 con 0x7f9838106c80 2023-07-14T19:22:43.077 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:42.430+0000 7f983559a700 1 -- 172.21.15.98:0/2774354282 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f982c0042c0 con 0x7f9838106c80 2023-07-14T19:22:43.077 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:42.430+0000 7f983559a700 1 -- 172.21.15.98:0/2774354282 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f982c011a00 con 0x7f9838106c80 2023-07-14T19:22:43.077 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:42.431+0000 7f983c82e700 1 -- 172.21.15.98:0/2774354282 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9838106c80 msgr2=0x7f9838107090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:22:43.077 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:42.431+0000 7f983c82e700 1 --2- 172.21.15.98:0/2774354282 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9838106c80 0x7f9838107090 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f982c0098a0 tx=0x7f982c009bb0 comp rx=0 tx=0).stop 2023-07-14T19:22:43.077 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:42.431+0000 7f983c82e700 1 -- 172.21.15.98:0/2774354282 shutdown_connections 2023-07-14T19:22:43.077 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:42.431+0000 7f983c82e700 1 --2- 172.21.15.98:0/2774354282 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9838106c80 0x7f9838107090 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:22:43.077 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:42.431+0000 7f983c82e700 1 -- 172.21.15.98:0/2774354282 >> 172.21.15.98:0/2774354282 conn(0x7f98380ff330 msgr2=0x7f9838101760 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:22:43.077 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:42.431+0000 7f983c82e700 1 -- 172.21.15.98:0/2774354282 shutdown_connections 2023-07-14T19:22:43.078 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:42.431+0000 7f983c82e700 1 -- 172.21.15.98:0/2774354282 wait complete. 2023-07-14T19:22:43.078 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:42.432+0000 7f983c82e700 1 Processor -- start 2023-07-14T19:22:43.078 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:42.432+0000 7f983c82e700 1 -- start start 2023-07-14T19:22:43.078 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:42.432+0000 7f983c82e700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9838106c80 0x7f983819b670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:22:43.078 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:42.432+0000 7f983c82e700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f983819bbb0 con 0x7f9838106c80 2023-07-14T19:22:43.078 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:42.432+0000 7f983659c700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9838106c80 0x7f983819b670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:22:43.078 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:42.432+0000 7f983659c700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9838106c80 0x7f983819b670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:44466/0 (socket says 172.21.15.98:44466) 2023-07-14T19:22:43.079 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:42.432+0000 7f983659c700 1 -- 172.21.15.98:0/1677359949 learned_addr learned my addr 172.21.15.98:0/1677359949 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-07-14T19:22:43.079 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:42.432+0000 7f983659c700 1 -- 172.21.15.98:0/1677359949 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f982c009550 con 0x7f9838106c80 2023-07-14T19:22:43.079 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:42.432+0000 7f983659c700 1 --2- 172.21.15.98:0/1677359949 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9838106c80 0x7f983819b670 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f982c009fb0 tx=0x7f982c011ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:22:43.079 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:42.432+0000 7f98277fe700 1 -- 172.21.15.98:0/1677359949 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f982c003a40 con 0x7f9838106c80 2023-07-14T19:22:43.079 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:42.433+0000 7f98277fe700 1 -- 172.21.15.98:0/1677359949 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f982c003ba0 con 0x7f9838106c80 2023-07-14T19:22:43.079 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:42.433+0000 7f983c82e700 1 -- 172.21.15.98:0/1677359949 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f983819bdb0 con 0x7f9838106c80 2023-07-14T19:22:43.079 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:42.433+0000 7f98277fe700 1 -- 172.21.15.98:0/1677359949 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f982c01a750 con 0x7f9838106c80 2023-07-14T19:22:43.079 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:42.433+0000 7f983c82e700 1 -- 172.21.15.98:0/1677359949 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f983819c1d0 con 0x7f9838106c80 2023-07-14T19:22:43.080 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:42.433+0000 7f98277fe700 1 -- 172.21.15.98:0/1677359949 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 8) v1 ==== 45483+0+0 (secure 0 0 0) 0x7f982c01a9d0 con 0x7f9838106c80 2023-07-14T19:22:43.080 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:42.433+0000 7f983c82e700 1 -- 172.21.15.98:0/1677359949 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9838061c70 con 0x7f9838106c80 2023-07-14T19:22:43.080 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:42.434+0000 7f98277fe700 1 --2- 172.21.15.98:0/1677359949 >> [v2:172.21.15.98:6800/1732769087,v1:172.21.15.98:6801/1732769087] conn(0x7f98200386c0 0x7f982003ab70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:22:43.080 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:42.434+0000 7f98277fe700 1 -- 172.21.15.98:0/1677359949 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7f982c04ca40 con 0x7f9838106c80 2023-07-14T19:22:43.080 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:42.434+0000 7f9835d9b700 1 --2- 172.21.15.98:0/1677359949 >> [v2:172.21.15.98:6800/1732769087,v1:172.21.15.98:6801/1732769087] conn(0x7f98200386c0 0x7f982003ab70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:22:43.080 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:42.434+0000 7f9835d9b700 1 --2- 172.21.15.98:0/1677359949 >> [v2:172.21.15.98:6800/1732769087,v1:172.21.15.98:6801/1732769087] conn(0x7f98200386c0 0x7f982003ab70 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f9828006fd0 tx=0x7f9828006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:22:43.081 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:42.438+0000 7f98277fe700 1 -- 172.21.15.98:0/1677359949 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f982c018d20 con 0x7f9838106c80 2023-07-14T19:22:43.081 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:42.627+0000 7f983c82e700 1 -- 172.21.15.98:0/1677359949 --> [v2:172.21.15.98:6800/1732769087,v1:172.21.15.98:6801/1732769087] -- mgr_command(tid 0: {"prefix": "cephadm set-user", "user": "root", "target": ["mon-mgr", ""]}) v1 -- 0x7f9838101690 con 0x7f98200386c0 2023-07-14T19:22:43.081 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:42.628+0000 7f98277fe700 1 -- 172.21.15.98:0/1677359949 <== mgr.14120 v2:172.21.15.98:6800/1732769087 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+16 (secure 0 0 0) 0x7f9838101690 con 0x7f98200386c0 2023-07-14T19:22:43.081 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:42.630+0000 7f983c82e700 1 -- 172.21.15.98:0/1677359949 >> [v2:172.21.15.98:6800/1732769087,v1:172.21.15.98:6801/1732769087] conn(0x7f98200386c0 msgr2=0x7f982003ab70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:22:43.081 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:42.630+0000 7f983c82e700 1 --2- 172.21.15.98:0/1677359949 >> [v2:172.21.15.98:6800/1732769087,v1:172.21.15.98:6801/1732769087] conn(0x7f98200386c0 0x7f982003ab70 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f9828006fd0 tx=0x7f9828006e40 comp rx=0 tx=0).stop 2023-07-14T19:22:43.081 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:42.630+0000 7f983c82e700 1 -- 172.21.15.98:0/1677359949 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9838106c80 msgr2=0x7f983819b670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:22:43.081 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:42.630+0000 7f983c82e700 1 --2- 172.21.15.98:0/1677359949 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9838106c80 0x7f983819b670 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f982c009fb0 tx=0x7f982c011ad0 comp rx=0 tx=0).stop 2023-07-14T19:22:43.081 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:42.630+0000 7f983c82e700 1 -- 172.21.15.98:0/1677359949 shutdown_connections 2023-07-14T19:22:43.082 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:42.630+0000 7f983c82e700 1 --2- 172.21.15.98:0/1677359949 >> [v2:172.21.15.98:6800/1732769087,v1:172.21.15.98:6801/1732769087] conn(0x7f98200386c0 0x7f982003ab70 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:22:43.082 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:42.630+0000 7f983c82e700 1 --2- 172.21.15.98:0/1677359949 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9838106c80 0x7f983819b670 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:22:43.082 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:42.630+0000 7f983c82e700 1 -- 172.21.15.98:0/1677359949 >> 172.21.15.98:0/1677359949 conn(0x7f98380ff330 msgr2=0x7f9838100a30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:22:43.082 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:42.630+0000 7f983c82e700 1 -- 172.21.15.98:0/1677359949 shutdown_connections 2023-07-14T19:22:43.082 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:42.630+0000 7f983c82e700 1 -- 172.21.15.98:0/1677359949 wait complete. 2023-07-14T19:22:43.082 INFO:teuthology.orchestra.run.smithi098.stdout:Generating ssh key... 2023-07-14T19:22:43.853 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:22:43 smithi098 ceph-mon[127486]: from='client.14134 -' entity='client.admin' cmd=[{"prefix": "cephadm set-user", "user": "root", "target": ["mon-mgr", ""]}]: dispatch 2023-07-14T19:22:44.220 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:43.559+0000 7f31a9aba700 1 Processor -- start 2023-07-14T19:22:44.220 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:43.559+0000 7f31a9aba700 1 -- start start 2023-07-14T19:22:44.221 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:43.560+0000 7f31a9aba700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f31a4104570 0x7f31a4104980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:22:44.221 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:43.560+0000 7f31a9aba700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f31a4104ec0 con 0x7f31a4104570 2023-07-14T19:22:44.221 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:43.560+0000 7f31a37fe700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f31a4104570 0x7f31a4104980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:22:44.221 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:43.560+0000 7f31a37fe700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f31a4104570 0x7f31a4104980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:44470/0 (socket says 172.21.15.98:44470) 2023-07-14T19:22:44.221 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:43.560+0000 7f31a37fe700 1 -- 172.21.15.98:0/1981782810 learned_addr learned my addr 172.21.15.98:0/1981782810 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-07-14T19:22:44.221 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:43.560+0000 7f31a37fe700 1 -- 172.21.15.98:0/1981782810 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f31a4105000 con 0x7f31a4104570 2023-07-14T19:22:44.221 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:43.560+0000 7f31a37fe700 1 --2- 172.21.15.98:0/1981782810 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f31a4104570 0x7f31a4104980 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f3194009e80 tx=0x7f319400b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=81a5951d352d41e9 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:22:44.222 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:43.561+0000 7f31a27fc700 1 -- 172.21.15.98:0/1981782810 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f319400e070 con 0x7f31a4104570 2023-07-14T19:22:44.222 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:43.561+0000 7f31a27fc700 1 -- 172.21.15.98:0/1981782810 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f319400b8c0 con 0x7f31a4104570 2023-07-14T19:22:44.222 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:43.561+0000 7f31a27fc700 1 -- 172.21.15.98:0/1981782810 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f31940044c0 con 0x7f31a4104570 2023-07-14T19:22:44.222 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:43.562+0000 7f31a9aba700 1 -- 172.21.15.98:0/1981782810 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f31a4104570 msgr2=0x7f31a4104980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:22:44.222 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:43.562+0000 7f31a9aba700 1 --2- 172.21.15.98:0/1981782810 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f31a4104570 0x7f31a4104980 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f3194009e80 tx=0x7f319400b330 comp rx=0 tx=0).stop 2023-07-14T19:22:44.222 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:43.562+0000 7f31a9aba700 1 -- 172.21.15.98:0/1981782810 shutdown_connections 2023-07-14T19:22:44.222 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:43.562+0000 7f31a9aba700 1 --2- 172.21.15.98:0/1981782810 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f31a4104570 0x7f31a4104980 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:22:44.223 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:43.562+0000 7f31a9aba700 1 -- 172.21.15.98:0/1981782810 >> 172.21.15.98:0/1981782810 conn(0x7f31a40ffbc0 msgr2=0x7f31a4101ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:22:44.223 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:43.562+0000 7f31a9aba700 1 -- 172.21.15.98:0/1981782810 shutdown_connections 2023-07-14T19:22:44.223 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:43.562+0000 7f31a9aba700 1 -- 172.21.15.98:0/1981782810 wait complete. 2023-07-14T19:22:44.223 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:43.562+0000 7f31a9aba700 1 Processor -- start 2023-07-14T19:22:44.223 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:43.563+0000 7f31a9aba700 1 -- start start 2023-07-14T19:22:44.223 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:43.563+0000 7f31a9aba700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f31a4104570 0x7f31a410a5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:22:44.223 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:43.563+0000 7f31a9aba700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f31a410aaf0 con 0x7f31a4104570 2023-07-14T19:22:44.223 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:43.563+0000 7f31a37fe700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f31a4104570 0x7f31a410a5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:22:44.224 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:43.563+0000 7f31a37fe700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f31a4104570 0x7f31a410a5b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:44472/0 (socket says 172.21.15.98:44472) 2023-07-14T19:22:44.224 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:43.563+0000 7f31a37fe700 1 -- 172.21.15.98:0/4239532666 learned_addr learned my addr 172.21.15.98:0/4239532666 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-07-14T19:22:44.224 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:43.563+0000 7f31a37fe700 1 -- 172.21.15.98:0/4239532666 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3194009b30 con 0x7f31a4104570 2023-07-14T19:22:44.224 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:43.563+0000 7f31a37fe700 1 --2- 172.21.15.98:0/4239532666 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f31a4104570 0x7f31a410a5b0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f3194009fd0 tx=0x7f31940045f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:22:44.224 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:43.563+0000 7f31a0ff9700 1 -- 172.21.15.98:0/4239532666 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f319400e040 con 0x7f31a4104570 2023-07-14T19:22:44.224 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:43.564+0000 7f31a0ff9700 1 -- 172.21.15.98:0/4239532666 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f319400bdf0 con 0x7f31a4104570 2023-07-14T19:22:44.225 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:43.564+0000 7f31a9aba700 1 -- 172.21.15.98:0/4239532666 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f31a4108c00 con 0x7f31a4104570 2023-07-14T19:22:44.225 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:43.564+0000 7f31a0ff9700 1 -- 172.21.15.98:0/4239532666 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f319401c840 con 0x7f31a4104570 2023-07-14T19:22:44.225 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:43.564+0000 7f31a9aba700 1 -- 172.21.15.98:0/4239532666 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f31a4109080 con 0x7f31a4104570 2023-07-14T19:22:44.225 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:43.564+0000 7f31a0ff9700 1 -- 172.21.15.98:0/4239532666 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 8) v1 ==== 45483+0+0 (secure 0 0 0) 0x7f319401ca80 con 0x7f31a4104570 2023-07-14T19:22:44.225 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:43.564+0000 7f31a0ff9700 1 --2- 172.21.15.98:0/4239532666 >> [v2:172.21.15.98:6800/1732769087,v1:172.21.15.98:6801/1732769087] conn(0x7f3190038710 0x7f319003abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:22:44.225 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:43.564+0000 7f31a9aba700 1 -- 172.21.15.98:0/4239532666 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f31a4109a90 con 0x7f31a4104570 2023-07-14T19:22:44.226 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:43.565+0000 7f31a0ff9700 1 -- 172.21.15.98:0/4239532666 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7f319404c610 con 0x7f31a4104570 2023-07-14T19:22:44.226 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:43.565+0000 7f31a2ffd700 1 --2- 172.21.15.98:0/4239532666 >> [v2:172.21.15.98:6800/1732769087,v1:172.21.15.98:6801/1732769087] conn(0x7f3190038710 0x7f319003abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:22:44.226 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:43.565+0000 7f31a2ffd700 1 --2- 172.21.15.98:0/4239532666 >> [v2:172.21.15.98:6800/1732769087,v1:172.21.15.98:6801/1732769087] conn(0x7f3190038710 0x7f319003abc0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f318c006fd0 tx=0x7f318c006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:22:44.226 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:43.569+0000 7f31a0ff9700 1 -- 172.21.15.98:0/4239532666 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f3194011750 con 0x7f31a4104570 2023-07-14T19:22:44.226 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:43.753+0000 7f31a9aba700 1 -- 172.21.15.98:0/4239532666 --> [v2:172.21.15.98:6800/1732769087,v1:172.21.15.98:6801/1732769087] -- mgr_command(tid 0: {"prefix": "cephadm generate-key", "target": ["mon-mgr", ""]}) v1 -- 0x7f31a41092c0 con 0x7f3190038710 2023-07-14T19:22:44.226 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:43.857+0000 7f31a0ff9700 1 -- 172.21.15.98:0/4239532666 <== mgr.14120 v2:172.21.15.98:6800/1732769087 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f31a41092c0 con 0x7f3190038710 2023-07-14T19:22:44.226 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:43.859+0000 7f31a9aba700 1 -- 172.21.15.98:0/4239532666 >> [v2:172.21.15.98:6800/1732769087,v1:172.21.15.98:6801/1732769087] conn(0x7f3190038710 msgr2=0x7f319003abc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:22:44.227 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:43.859+0000 7f31a9aba700 1 --2- 172.21.15.98:0/4239532666 >> [v2:172.21.15.98:6800/1732769087,v1:172.21.15.98:6801/1732769087] conn(0x7f3190038710 0x7f319003abc0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f318c006fd0 tx=0x7f318c006e40 comp rx=0 tx=0).stop 2023-07-14T19:22:44.227 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:43.859+0000 7f31a9aba700 1 -- 172.21.15.98:0/4239532666 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f31a4104570 msgr2=0x7f31a410a5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:22:44.227 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:43.859+0000 7f31a9aba700 1 --2- 172.21.15.98:0/4239532666 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f31a4104570 0x7f31a410a5b0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f3194009fd0 tx=0x7f31940045f0 comp rx=0 tx=0).stop 2023-07-14T19:22:44.227 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:43.859+0000 7f31a9aba700 1 -- 172.21.15.98:0/4239532666 shutdown_connections 2023-07-14T19:22:44.227 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:43.859+0000 7f31a9aba700 1 --2- 172.21.15.98:0/4239532666 >> [v2:172.21.15.98:6800/1732769087,v1:172.21.15.98:6801/1732769087] conn(0x7f3190038710 0x7f319003abc0 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:22:44.227 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:43.859+0000 7f31a9aba700 1 --2- 172.21.15.98:0/4239532666 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f31a4104570 0x7f31a410a5b0 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:22:44.227 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:43.859+0000 7f31a9aba700 1 -- 172.21.15.98:0/4239532666 >> 172.21.15.98:0/4239532666 conn(0x7f31a40ffbc0 msgr2=0x7f31a4100870 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:22:44.228 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:43.860+0000 7f31a9aba700 1 -- 172.21.15.98:0/4239532666 shutdown_connections 2023-07-14T19:22:44.228 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:43.860+0000 7f31a9aba700 1 -- 172.21.15.98:0/4239532666 wait complete. 2023-07-14T19:22:44.251 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Jul 14 19:22:43 smithi098 ceph-9adea648-227b-11ee-9b33-001a4aab830c-mgr-x[127734]: Generating public/private rsa key pair. 2023-07-14T19:22:44.252 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Jul 14 19:22:43 smithi098 ceph-9adea648-227b-11ee-9b33-001a4aab830c-mgr-x[127734]: Your identification has been saved in /tmp/tmp2xwhpw5q/key. 2023-07-14T19:22:44.252 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Jul 14 19:22:43 smithi098 ceph-9adea648-227b-11ee-9b33-001a4aab830c-mgr-x[127734]: Your public key has been saved in /tmp/tmp2xwhpw5q/key.pub. 2023-07-14T19:22:44.252 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Jul 14 19:22:43 smithi098 ceph-9adea648-227b-11ee-9b33-001a4aab830c-mgr-x[127734]: The key fingerprint is: 2023-07-14T19:22:44.252 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Jul 14 19:22:43 smithi098 ceph-9adea648-227b-11ee-9b33-001a4aab830c-mgr-x[127734]: SHA256:GjEpp9S/s4rWhwVTp7sTnp9pcPFKk+BWCYyt/RRa2Zw ceph-9adea648-227b-11ee-9b33-001a4aab830c 2023-07-14T19:22:44.253 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Jul 14 19:22:43 smithi098 ceph-9adea648-227b-11ee-9b33-001a4aab830c-mgr-x[127734]: The key's randomart image is: 2023-07-14T19:22:44.253 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Jul 14 19:22:43 smithi098 ceph-9adea648-227b-11ee-9b33-001a4aab830c-mgr-x[127734]: +---[RSA 3072]----+ 2023-07-14T19:22:44.253 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Jul 14 19:22:43 smithi098 ceph-9adea648-227b-11ee-9b33-001a4aab830c-mgr-x[127734]: | + + . | 2023-07-14T19:22:44.253 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Jul 14 19:22:43 smithi098 ceph-9adea648-227b-11ee-9b33-001a4aab830c-mgr-x[127734]: | ...= = E | 2023-07-14T19:22:44.253 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Jul 14 19:22:43 smithi098 ceph-9adea648-227b-11ee-9b33-001a4aab830c-mgr-x[127734]: | o *+ * o | 2023-07-14T19:22:44.253 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Jul 14 19:22:43 smithi098 ceph-9adea648-227b-11ee-9b33-001a4aab830c-mgr-x[127734]: | . +++= = | 2023-07-14T19:22:44.253 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Jul 14 19:22:43 smithi098 ceph-9adea648-227b-11ee-9b33-001a4aab830c-mgr-x[127734]: | . .+S* + | 2023-07-14T19:22:44.253 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Jul 14 19:22:43 smithi098 ceph-9adea648-227b-11ee-9b33-001a4aab830c-mgr-x[127734]: | oB.* . | 2023-07-14T19:22:44.254 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Jul 14 19:22:43 smithi098 ceph-9adea648-227b-11ee-9b33-001a4aab830c-mgr-x[127734]: | ..=oB o | 2023-07-14T19:22:44.254 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Jul 14 19:22:43 smithi098 ceph-9adea648-227b-11ee-9b33-001a4aab830c-mgr-x[127734]: | ..o *ooo | 2023-07-14T19:22:44.254 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Jul 14 19:22:43 smithi098 ceph-9adea648-227b-11ee-9b33-001a4aab830c-mgr-x[127734]: | .. .o.++ | 2023-07-14T19:22:44.254 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Jul 14 19:22:43 smithi098 ceph-9adea648-227b-11ee-9b33-001a4aab830c-mgr-x[127734]: +----[SHA256]-----+ 2023-07-14T19:22:45.220 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:22:44 smithi098 ceph-mon[127486]: from='client.14136 -' entity='client.admin' cmd=[{"prefix": "cephadm generate-key", "target": ["mon-mgr", ""]}]: dispatch 2023-07-14T19:22:45.221 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:22:44 smithi098 ceph-mon[127486]: Generating ssh key... 2023-07-14T19:22:45.221 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:22:44 smithi098 ceph-mon[127486]: from='mgr.14120 172.21.15.98:0/3060674050' entity='mgr.x' 2023-07-14T19:22:45.221 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:22:44 smithi098 ceph-mon[127486]: from='mgr.14120 172.21.15.98:0/3060674050' entity='mgr.x' 2023-07-14T19:22:45.266 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDp+ClaF7LpEMFYWJOvDu+/3NsZN3Q2vTOLAtxVP/VXDLuI1Nkc+DgD93kbQKb+hjgnmg40GqrfWAPHhO1QF5DTbMKVdcRAocN3LcWnew6S0nCRsD7XBx9foCvlCUBnE7pPXeRZlbEwXdFFaHakHf39wsC/zYUK+1lC+VBEGDPAO7KiSPhS7raP/YlswnDG/be0GMpQEpSjJp/iwDaDa67DHNOtUbkGUBsjZZO+3q5nzadZRHZ8FCX6DQBNxQdWwCtmgCC/F0nv1DRozV9bNubFE9PV9jS7CJ0YrVhPGmhECaO+5xGOpxEUJ5nUVw5jF1/5s2rZwpeWye1Pwp9q9TwGOqevZpDjQICGaZHpa/bF08J6lV+hM7UceEDMeiX386vCmL1rCQDO0+vBpCBKAW+vLfrUd2fB++vy/YvjrRj+cix5y+HkHL1ciscATK9lyGl2/MxfQCxftu2d8wn7l3QKSMr7emoM5bMvMALMB/UMLXUzVtNem6TNliiAH2ZkA58= ceph-9adea648-227b-11ee-9b33-001a4aab830c 2023-07-14T19:22:45.266 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:44.663+0000 7fce7a1a7700 1 Processor -- start 2023-07-14T19:22:45.266 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:44.663+0000 7fce7a1a7700 1 -- start start 2023-07-14T19:22:45.266 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:44.663+0000 7fce7a1a7700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fce74104570 0x7fce74104980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:22:45.267 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:44.663+0000 7fce7a1a7700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fce74104ec0 con 0x7fce74104570 2023-07-14T19:22:45.267 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:44.663+0000 7fce737fe700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fce74104570 0x7fce74104980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:22:45.267 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:44.663+0000 7fce737fe700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fce74104570 0x7fce74104980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:44476/0 (socket says 172.21.15.98:44476) 2023-07-14T19:22:45.267 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:44.663+0000 7fce737fe700 1 -- 172.21.15.98:0/4255233238 learned_addr learned my addr 172.21.15.98:0/4255233238 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-07-14T19:22:45.267 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:44.664+0000 7fce737fe700 1 -- 172.21.15.98:0/4255233238 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fce74105000 con 0x7fce74104570 2023-07-14T19:22:45.267 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:44.664+0000 7fce737fe700 1 --2- 172.21.15.98:0/4255233238 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fce74104570 0x7fce74104980 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7fce680098a0 tx=0x7fce68009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f91e934ba83a49a4 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:22:45.267 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:44.664+0000 7fce727fc700 1 -- 172.21.15.98:0/4255233238 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fce68004160 con 0x7fce74104570 2023-07-14T19:22:45.268 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:44.664+0000 7fce727fc700 1 -- 172.21.15.98:0/4255233238 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fce680042c0 con 0x7fce74104570 2023-07-14T19:22:45.268 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:44.664+0000 7fce727fc700 1 -- 172.21.15.98:0/4255233238 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fce68011a00 con 0x7fce74104570 2023-07-14T19:22:45.268 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:44.664+0000 7fce7a1a7700 1 -- 172.21.15.98:0/4255233238 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fce74104570 msgr2=0x7fce74104980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:22:45.268 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:44.664+0000 7fce7a1a7700 1 --2- 172.21.15.98:0/4255233238 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fce74104570 0x7fce74104980 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7fce680098a0 tx=0x7fce68009bb0 comp rx=0 tx=0).stop 2023-07-14T19:22:45.268 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:44.664+0000 7fce7a1a7700 1 -- 172.21.15.98:0/4255233238 shutdown_connections 2023-07-14T19:22:45.268 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:44.664+0000 7fce7a1a7700 1 --2- 172.21.15.98:0/4255233238 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fce74104570 0x7fce74104980 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:22:45.268 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:44.664+0000 7fce7a1a7700 1 -- 172.21.15.98:0/4255233238 >> 172.21.15.98:0/4255233238 conn(0x7fce740ffbc0 msgr2=0x7fce74101ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:22:45.269 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:44.664+0000 7fce7a1a7700 1 -- 172.21.15.98:0/4255233238 shutdown_connections 2023-07-14T19:22:45.269 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:44.664+0000 7fce7a1a7700 1 -- 172.21.15.98:0/4255233238 wait complete. 2023-07-14T19:22:45.269 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:44.665+0000 7fce7a1a7700 1 Processor -- start 2023-07-14T19:22:45.269 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:44.665+0000 7fce7a1a7700 1 -- start start 2023-07-14T19:22:45.269 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:44.665+0000 7fce7a1a7700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fce74104570 0x7fce74198370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:22:45.269 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:44.665+0000 7fce7a1a7700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fce741988b0 con 0x7fce74104570 2023-07-14T19:22:45.271 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:44.665+0000 7fce737fe700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fce74104570 0x7fce74198370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:22:45.271 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:44.665+0000 7fce737fe700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fce74104570 0x7fce74198370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:44478/0 (socket says 172.21.15.98:44478) 2023-07-14T19:22:45.271 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:44.666+0000 7fce737fe700 1 -- 172.21.15.98:0/1651762672 learned_addr learned my addr 172.21.15.98:0/1651762672 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-07-14T19:22:45.272 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:44.666+0000 7fce737fe700 1 -- 172.21.15.98:0/1651762672 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fce68009550 con 0x7fce74104570 2023-07-14T19:22:45.272 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:44.666+0000 7fce737fe700 1 --2- 172.21.15.98:0/1651762672 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fce74104570 0x7fce74198370 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7fce68009fb0 tx=0x7fce68011ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:22:45.272 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:44.666+0000 7fce70ff9700 1 -- 172.21.15.98:0/1651762672 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fce68003a40 con 0x7fce74104570 2023-07-14T19:22:45.272 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:44.666+0000 7fce70ff9700 1 -- 172.21.15.98:0/1651762672 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fce68003ba0 con 0x7fce74104570 2023-07-14T19:22:45.272 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:44.666+0000 7fce7a1a7700 1 -- 172.21.15.98:0/1651762672 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fce74198ab0 con 0x7fce74104570 2023-07-14T19:22:45.272 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:44.666+0000 7fce70ff9700 1 -- 172.21.15.98:0/1651762672 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fce6801a750 con 0x7fce74104570 2023-07-14T19:22:45.273 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:44.666+0000 7fce7a1a7700 1 -- 172.21.15.98:0/1651762672 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fce74198ed0 con 0x7fce74104570 2023-07-14T19:22:45.273 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:44.667+0000 7fce70ff9700 1 -- 172.21.15.98:0/1651762672 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 8) v1 ==== 45483+0+0 (secure 0 0 0) 0x7fce6801a9d0 con 0x7fce74104570 2023-07-14T19:22:45.273 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:44.667+0000 7fce7a1a7700 1 -- 172.21.15.98:0/1651762672 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fce74192340 con 0x7fce74104570 2023-07-14T19:22:45.273 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:44.667+0000 7fce70ff9700 1 --2- 172.21.15.98:0/1651762672 >> [v2:172.21.15.98:6800/1732769087,v1:172.21.15.98:6801/1732769087] conn(0x7fce5c0386c0 0x7fce5c03ab70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:22:45.273 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:44.667+0000 7fce70ff9700 1 -- 172.21.15.98:0/1651762672 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7fce6804c780 con 0x7fce74104570 2023-07-14T19:22:45.274 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:44.667+0000 7fce72ffd700 1 --2- 172.21.15.98:0/1651762672 >> [v2:172.21.15.98:6800/1732769087,v1:172.21.15.98:6801/1732769087] conn(0x7fce5c0386c0 0x7fce5c03ab70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:22:45.274 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:44.668+0000 7fce72ffd700 1 --2- 172.21.15.98:0/1651762672 >> [v2:172.21.15.98:6800/1732769087,v1:172.21.15.98:6801/1732769087] conn(0x7fce5c0386c0 0x7fce5c03ab70 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7fce64006fd0 tx=0x7fce64006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:22:45.274 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:44.672+0000 7fce70ff9700 1 -- 172.21.15.98:0/1651762672 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7fce68018b80 con 0x7fce74104570 2023-07-14T19:22:45.274 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:44.859+0000 7fce7a1a7700 1 -- 172.21.15.98:0/1651762672 --> [v2:172.21.15.98:6800/1732769087,v1:172.21.15.98:6801/1732769087] -- mgr_command(tid 0: {"prefix": "cephadm get-pub-key", "target": ["mon-mgr", ""]}) v1 -- 0x7fce74060a80 con 0x7fce5c0386c0 2023-07-14T19:22:45.275 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:44.860+0000 7fce70ff9700 1 -- 172.21.15.98:0/1651762672 <== mgr.14120 v2:172.21.15.98:6800/1732769087 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+595 (secure 0 0 0) 0x7fce74060a80 con 0x7fce5c0386c0 2023-07-14T19:22:45.275 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:44.862+0000 7fce7a1a7700 1 -- 172.21.15.98:0/1651762672 >> [v2:172.21.15.98:6800/1732769087,v1:172.21.15.98:6801/1732769087] conn(0x7fce5c0386c0 msgr2=0x7fce5c03ab70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:22:45.275 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:44.862+0000 7fce7a1a7700 1 --2- 172.21.15.98:0/1651762672 >> [v2:172.21.15.98:6800/1732769087,v1:172.21.15.98:6801/1732769087] conn(0x7fce5c0386c0 0x7fce5c03ab70 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7fce64006fd0 tx=0x7fce64006e40 comp rx=0 tx=0).stop 2023-07-14T19:22:45.275 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:44.862+0000 7fce7a1a7700 1 -- 172.21.15.98:0/1651762672 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fce74104570 msgr2=0x7fce74198370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:22:45.275 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:44.862+0000 7fce7a1a7700 1 --2- 172.21.15.98:0/1651762672 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fce74104570 0x7fce74198370 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7fce68009fb0 tx=0x7fce68011ad0 comp rx=0 tx=0).stop 2023-07-14T19:22:45.275 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:44.862+0000 7fce7a1a7700 1 -- 172.21.15.98:0/1651762672 shutdown_connections 2023-07-14T19:22:45.275 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:44.862+0000 7fce7a1a7700 1 --2- 172.21.15.98:0/1651762672 >> [v2:172.21.15.98:6800/1732769087,v1:172.21.15.98:6801/1732769087] conn(0x7fce5c0386c0 0x7fce5c03ab70 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:22:45.276 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:44.862+0000 7fce7a1a7700 1 --2- 172.21.15.98:0/1651762672 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fce74104570 0x7fce74198370 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:22:45.276 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:44.862+0000 7fce7a1a7700 1 -- 172.21.15.98:0/1651762672 >> 172.21.15.98:0/1651762672 conn(0x7fce740ffbc0 msgr2=0x7fce741012e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:22:45.276 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:44.862+0000 7fce7a1a7700 1 -- 172.21.15.98:0/1651762672 shutdown_connections 2023-07-14T19:22:45.276 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:44.862+0000 7fce7a1a7700 1 -- 172.21.15.98:0/1651762672 wait complete. 2023-07-14T19:22:45.276 INFO:teuthology.orchestra.run.smithi098.stdout:Wrote public SSH key to /home/ubuntu/cephtest/ceph.pub 2023-07-14T19:22:45.276 INFO:teuthology.orchestra.run.smithi098.stdout:Adding key to root@localhost authorized_keys... 2023-07-14T19:22:45.277 INFO:teuthology.orchestra.run.smithi098.stdout:Adding host smithi098... 2023-07-14T19:22:46.123 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:22:45 smithi098 ceph-mon[127486]: from='client.14138 -' entity='client.admin' cmd=[{"prefix": "cephadm get-pub-key", "target": ["mon-mgr", ""]}]: dispatch 2023-07-14T19:22:47.531 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:22:47 smithi098 ceph-mon[127486]: from='client.14140 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi098", "addr": "172.21.15.98", "target": ["mon-mgr", ""]}]: dispatch 2023-07-14T19:22:47.532 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:22:47 smithi098 ceph-mon[127486]: Deploying cephadm binary to smithi098 2023-07-14T19:22:48.717 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout Added host 'smithi098' with addr '172.21.15.98' 2023-07-14T19:22:48.718 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:45.815+0000 7f43ae1ce700 1 Processor -- start 2023-07-14T19:22:48.718 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:45.815+0000 7f43ae1ce700 1 -- start start 2023-07-14T19:22:48.718 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:45.815+0000 7f43ae1ce700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f43a8103950 0x7f43a8103d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:22:48.718 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:45.815+0000 7f43ae1ce700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f43a81042a0 con 0x7f43a8103950 2023-07-14T19:22:48.718 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:45.816+0000 7f43a77fe700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f43a8103950 0x7f43a8103d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:22:48.718 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:45.816+0000 7f43a77fe700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f43a8103950 0x7f43a8103d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:44482/0 (socket says 172.21.15.98:44482) 2023-07-14T19:22:48.719 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:45.816+0000 7f43a77fe700 1 -- 172.21.15.98:0/823884334 learned_addr learned my addr 172.21.15.98:0/823884334 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-07-14T19:22:48.719 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:45.816+0000 7f43a77fe700 1 -- 172.21.15.98:0/823884334 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f43a81043e0 con 0x7f43a8103950 2023-07-14T19:22:48.719 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:45.816+0000 7f43a77fe700 1 --2- 172.21.15.98:0/823884334 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f43a8103950 0x7f43a8103d60 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f439c0098a0 tx=0x7f439c009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9ffc22bbc4aaadfe server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:22:48.719 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:45.816+0000 7f43a67fc700 1 -- 172.21.15.98:0/823884334 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f439c004160 con 0x7f43a8103950 2023-07-14T19:22:48.719 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:45.816+0000 7f43a67fc700 1 -- 172.21.15.98:0/823884334 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f439c0042c0 con 0x7f43a8103950 2023-07-14T19:22:48.719 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:45.816+0000 7f43a67fc700 1 -- 172.21.15.98:0/823884334 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f439c011a00 con 0x7f43a8103950 2023-07-14T19:22:48.719 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:45.817+0000 7f43ae1ce700 1 -- 172.21.15.98:0/823884334 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f43a8103950 msgr2=0x7f43a8103d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:22:48.720 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:45.817+0000 7f43ae1ce700 1 --2- 172.21.15.98:0/823884334 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f43a8103950 0x7f43a8103d60 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f439c0098a0 tx=0x7f439c009bb0 comp rx=0 tx=0).stop 2023-07-14T19:22:48.720 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:45.817+0000 7f43ae1ce700 1 -- 172.21.15.98:0/823884334 shutdown_connections 2023-07-14T19:22:48.720 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:45.817+0000 7f43ae1ce700 1 --2- 172.21.15.98:0/823884334 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f43a8103950 0x7f43a8103d60 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:22:48.720 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:45.817+0000 7f43ae1ce700 1 -- 172.21.15.98:0/823884334 >> 172.21.15.98:0/823884334 conn(0x7f43a80ff330 msgr2=0x7f43a8101760 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:22:48.720 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:45.817+0000 7f43ae1ce700 1 -- 172.21.15.98:0/823884334 shutdown_connections 2023-07-14T19:22:48.720 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:45.817+0000 7f43ae1ce700 1 -- 172.21.15.98:0/823884334 wait complete. 2023-07-14T19:22:48.720 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:45.818+0000 7f43ae1ce700 1 Processor -- start 2023-07-14T19:22:48.721 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:45.818+0000 7f43ae1ce700 1 -- start start 2023-07-14T19:22:48.721 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:45.818+0000 7f43ae1ce700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f43a81996c0 0x7f43a8199ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:22:48.721 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:45.818+0000 7f43ae1ce700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f43a819a010 con 0x7f43a81996c0 2023-07-14T19:22:48.721 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:45.818+0000 7f43a77fe700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f43a81996c0 0x7f43a8199ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:22:48.721 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:45.818+0000 7f43a77fe700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f43a81996c0 0x7f43a8199ad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:44484/0 (socket says 172.21.15.98:44484) 2023-07-14T19:22:48.721 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:45.818+0000 7f43a77fe700 1 -- 172.21.15.98:0/2287147102 learned_addr learned my addr 172.21.15.98:0/2287147102 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-07-14T19:22:48.721 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:45.818+0000 7f43a77fe700 1 -- 172.21.15.98:0/2287147102 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f439c009550 con 0x7f43a81996c0 2023-07-14T19:22:48.721 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:45.819+0000 7f43a77fe700 1 --2- 172.21.15.98:0/2287147102 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f43a81996c0 0x7f43a8199ad0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f439c009fb0 tx=0x7f439c012fb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:22:48.722 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:45.819+0000 7f43a4ff9700 1 -- 172.21.15.98:0/2287147102 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f439c0037d0 con 0x7f43a81996c0 2023-07-14T19:22:48.722 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:45.819+0000 7f43a4ff9700 1 -- 172.21.15.98:0/2287147102 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f439c01f070 con 0x7f43a81996c0 2023-07-14T19:22:48.722 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:45.819+0000 7f43ae1ce700 1 -- 172.21.15.98:0/2287147102 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f43a819a210 con 0x7f43a81996c0 2023-07-14T19:22:48.722 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:45.819+0000 7f43a4ff9700 1 -- 172.21.15.98:0/2287147102 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f439c01add0 con 0x7f43a81996c0 2023-07-14T19:22:48.722 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:45.819+0000 7f43ae1ce700 1 -- 172.21.15.98:0/2287147102 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f43a819ce70 con 0x7f43a81996c0 2023-07-14T19:22:48.722 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:45.819+0000 7f43a4ff9700 1 -- 172.21.15.98:0/2287147102 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 8) v1 ==== 45483+0+0 (secure 0 0 0) 0x7f439c01b440 con 0x7f43a81996c0 2023-07-14T19:22:48.722 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:45.820+0000 7f43a4ff9700 1 --2- 172.21.15.98:0/2287147102 >> [v2:172.21.15.98:6800/1732769087,v1:172.21.15.98:6801/1732769087] conn(0x7f4390038710 0x7f439003abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:22:48.723 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:45.820+0000 7f43a4ff9700 1 -- 172.21.15.98:0/2287147102 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7f439c021070 con 0x7f43a81996c0 2023-07-14T19:22:48.723 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:45.820+0000 7f43a6ffd700 1 --2- 172.21.15.98:0/2287147102 >> [v2:172.21.15.98:6800/1732769087,v1:172.21.15.98:6801/1732769087] conn(0x7f4390038710 0x7f439003abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:22:48.723 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:45.820+0000 7f43ae1ce700 1 -- 172.21.15.98:0/2287147102 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f43a81934b0 con 0x7f43a81996c0 2023-07-14T19:22:48.723 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:45.820+0000 7f43a6ffd700 1 --2- 172.21.15.98:0/2287147102 >> [v2:172.21.15.98:6800/1732769087,v1:172.21.15.98:6801/1732769087] conn(0x7f4390038710 0x7f439003abc0 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f4398006fd0 tx=0x7f4398006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:22:48.723 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:45.825+0000 7f43a4ff9700 1 -- 172.21.15.98:0/2287147102 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f439c018920 con 0x7f43a81996c0 2023-07-14T19:22:48.723 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:46.014+0000 7f43ae1ce700 1 -- 172.21.15.98:0/2287147102 --> [v2:172.21.15.98:6800/1732769087,v1:172.21.15.98:6801/1732769087] -- mgr_command(tid 0: {"prefix": "orch host add", "hostname": "smithi098", "addr": "172.21.15.98", "target": ["mon-mgr", ""]}) v1 -- 0x7f43a8060a80 con 0x7f4390038710 2023-07-14T19:22:48.723 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:48.336+0000 7f43a4ff9700 1 -- 172.21.15.98:0/2287147102 <== mgr.14120 v2:172.21.15.98:6800/1732769087 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+48 (secure 0 0 0) 0x7f43a8060a80 con 0x7f4390038710 2023-07-14T19:22:48.724 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:48.339+0000 7f43ae1ce700 1 -- 172.21.15.98:0/2287147102 >> [v2:172.21.15.98:6800/1732769087,v1:172.21.15.98:6801/1732769087] conn(0x7f4390038710 msgr2=0x7f439003abc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:22:48.724 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:48.339+0000 7f43ae1ce700 1 --2- 172.21.15.98:0/2287147102 >> [v2:172.21.15.98:6800/1732769087,v1:172.21.15.98:6801/1732769087] conn(0x7f4390038710 0x7f439003abc0 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f4398006fd0 tx=0x7f4398006e40 comp rx=0 tx=0).stop 2023-07-14T19:22:48.724 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:48.339+0000 7f43ae1ce700 1 -- 172.21.15.98:0/2287147102 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f43a81996c0 msgr2=0x7f43a8199ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:22:48.724 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:48.339+0000 7f43ae1ce700 1 --2- 172.21.15.98:0/2287147102 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f43a81996c0 0x7f43a8199ad0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f439c009fb0 tx=0x7f439c012fb0 comp rx=0 tx=0).stop 2023-07-14T19:22:48.724 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:48.339+0000 7f43ae1ce700 1 -- 172.21.15.98:0/2287147102 shutdown_connections 2023-07-14T19:22:48.724 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:48.339+0000 7f43ae1ce700 1 --2- 172.21.15.98:0/2287147102 >> [v2:172.21.15.98:6800/1732769087,v1:172.21.15.98:6801/1732769087] conn(0x7f4390038710 0x7f439003abc0 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:22:48.724 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:48.339+0000 7f43ae1ce700 1 --2- 172.21.15.98:0/2287147102 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f43a81996c0 0x7f43a8199ad0 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:22:48.725 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:48.339+0000 7f43ae1ce700 1 -- 172.21.15.98:0/2287147102 >> 172.21.15.98:0/2287147102 conn(0x7f43a80ff330 msgr2=0x7f43a8100b20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:22:48.725 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:48.339+0000 7f43ae1ce700 1 -- 172.21.15.98:0/2287147102 shutdown_connections 2023-07-14T19:22:48.725 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:48.339+0000 7f43ae1ce700 1 -- 172.21.15.98:0/2287147102 wait complete. 2023-07-14T19:22:48.725 INFO:teuthology.orchestra.run.smithi098.stdout:Deploying unmanaged mon service... 2023-07-14T19:22:49.758 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout Scheduled mon update... 2023-07-14T19:22:49.759 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:49.202+0000 7f875bc88700 1 Processor -- start 2023-07-14T19:22:49.759 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:49.202+0000 7f875bc88700 1 -- start start 2023-07-14T19:22:49.759 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:49.202+0000 7f875bc88700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8754106c80 0x7f8754107090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:22:49.759 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:49.202+0000 7f875bc88700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f87541075d0 con 0x7f8754106c80 2023-07-14T19:22:49.759 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:49.202+0000 7f8759a24700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8754106c80 0x7f8754107090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:22:49.759 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:49.202+0000 7f8759a24700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8754106c80 0x7f8754107090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:44518/0 (socket says 172.21.15.98:44518) 2023-07-14T19:22:49.760 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:49.202+0000 7f8759a24700 1 -- 172.21.15.98:0/2897359778 learned_addr learned my addr 172.21.15.98:0/2897359778 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-07-14T19:22:49.760 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:49.203+0000 7f8759a24700 1 -- 172.21.15.98:0/2897359778 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8754107710 con 0x7f8754106c80 2023-07-14T19:22:49.760 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:49.203+0000 7f8759a24700 1 --2- 172.21.15.98:0/2897359778 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8754106c80 0x7f8754107090 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f87440098a0 tx=0x7f8744009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c693b09efa15c724 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:22:49.760 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:49.203+0000 7f8758a22700 1 -- 172.21.15.98:0/2897359778 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f8744004160 con 0x7f8754106c80 2023-07-14T19:22:49.760 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:49.203+0000 7f8758a22700 1 -- 172.21.15.98:0/2897359778 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f87440042c0 con 0x7f8754106c80 2023-07-14T19:22:49.761 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:22:49 smithi098 ceph-mon[127486]: from='mgr.14120 172.21.15.98:0/3060674050' entity='mgr.x' 2023-07-14T19:22:49.761 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:22:49 smithi098 ceph-mon[127486]: Added host smithi098 2023-07-14T19:22:49.761 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:22:49 smithi098 ceph-mon[127486]: from='mgr.14120 172.21.15.98:0/3060674050' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-14T19:22:49.764 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:49.203+0000 7f8758a22700 1 -- 172.21.15.98:0/2897359778 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f8744011a00 con 0x7f8754106c80 2023-07-14T19:22:49.764 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:49.203+0000 7f875bc88700 1 -- 172.21.15.98:0/2897359778 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8754106c80 msgr2=0x7f8754107090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:22:49.764 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:49.204+0000 7f875bc88700 1 --2- 172.21.15.98:0/2897359778 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8754106c80 0x7f8754107090 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f87440098a0 tx=0x7f8744009bb0 comp rx=0 tx=0).stop 2023-07-14T19:22:49.765 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:49.204+0000 7f875bc88700 1 -- 172.21.15.98:0/2897359778 shutdown_connections 2023-07-14T19:22:49.765 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:49.204+0000 7f875bc88700 1 --2- 172.21.15.98:0/2897359778 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8754106c80 0x7f8754107090 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:22:49.765 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:49.204+0000 7f875bc88700 1 -- 172.21.15.98:0/2897359778 >> 172.21.15.98:0/2897359778 conn(0x7f87540ff330 msgr2=0x7f8754101760 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:22:49.765 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:49.204+0000 7f875bc88700 1 -- 172.21.15.98:0/2897359778 shutdown_connections 2023-07-14T19:22:49.765 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:49.204+0000 7f875bc88700 1 -- 172.21.15.98:0/2897359778 wait complete. 2023-07-14T19:22:49.765 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:49.204+0000 7f875bc88700 1 Processor -- start 2023-07-14T19:22:49.766 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:49.204+0000 7f875bc88700 1 -- start start 2023-07-14T19:22:49.766 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:49.204+0000 7f875bc88700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8754106c80 0x7f8754199450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:22:49.767 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:49.204+0000 7f875bc88700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8754199990 con 0x7f8754106c80 2023-07-14T19:22:49.767 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:49.205+0000 7f8759a24700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8754106c80 0x7f8754199450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:22:49.767 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:49.205+0000 7f8759a24700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8754106c80 0x7f8754199450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:44520/0 (socket says 172.21.15.98:44520) 2023-07-14T19:22:49.767 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:49.205+0000 7f8759a24700 1 -- 172.21.15.98:0/652414074 learned_addr learned my addr 172.21.15.98:0/652414074 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-07-14T19:22:49.767 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:49.205+0000 7f8759a24700 1 -- 172.21.15.98:0/652414074 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8744009550 con 0x7f8754106c80 2023-07-14T19:22:49.767 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:49.205+0000 7f8759a24700 1 --2- 172.21.15.98:0/652414074 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8754106c80 0x7f8754199450 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f8744009130 tx=0x7f8744011ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:22:49.767 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:49.205+0000 7f874affd700 1 -- 172.21.15.98:0/652414074 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f8744003b70 con 0x7f8754106c80 2023-07-14T19:22:49.768 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:49.205+0000 7f874affd700 1 -- 172.21.15.98:0/652414074 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f8744003cd0 con 0x7f8754106c80 2023-07-14T19:22:49.768 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:49.205+0000 7f875bc88700 1 -- 172.21.15.98:0/652414074 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8754199b90 con 0x7f8754106c80 2023-07-14T19:22:49.768 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:49.205+0000 7f874affd700 1 -- 172.21.15.98:0/652414074 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f87440197d0 con 0x7f8754106c80 2023-07-14T19:22:49.768 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:49.205+0000 7f875bc88700 1 -- 172.21.15.98:0/652414074 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8754199fb0 con 0x7f8754106c80 2023-07-14T19:22:49.768 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:49.206+0000 7f874affd700 1 -- 172.21.15.98:0/652414074 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 8) v1 ==== 45483+0+0 (secure 0 0 0) 0x7f874401abb0 con 0x7f8754106c80 2023-07-14T19:22:49.768 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:49.206+0000 7f875bc88700 1 -- 172.21.15.98:0/652414074 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8754193450 con 0x7f8754106c80 2023-07-14T19:22:49.768 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:49.206+0000 7f874affd700 1 --2- 172.21.15.98:0/652414074 >> [v2:172.21.15.98:6800/1732769087,v1:172.21.15.98:6801/1732769087] conn(0x7f8740038710 0x7f874003abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:22:49.769 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:49.206+0000 7f874affd700 1 -- 172.21.15.98:0/652414074 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7f874404c950 con 0x7f8754106c80 2023-07-14T19:22:49.769 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:49.206+0000 7f8759223700 1 --2- 172.21.15.98:0/652414074 >> [v2:172.21.15.98:6800/1732769087,v1:172.21.15.98:6801/1732769087] conn(0x7f8740038710 0x7f874003abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:22:49.769 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:49.207+0000 7f8759223700 1 --2- 172.21.15.98:0/652414074 >> [v2:172.21.15.98:6800/1732769087,v1:172.21.15.98:6801/1732769087] conn(0x7f8740038710 0x7f874003abc0 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f8750006fd0 tx=0x7f8750006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:22:49.769 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:49.211+0000 7f874affd700 1 -- 172.21.15.98:0/652414074 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f874401a430 con 0x7f8754106c80 2023-07-14T19:22:49.769 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:49.412+0000 7f875bc88700 1 -- 172.21.15.98:0/652414074 --> [v2:172.21.15.98:6800/1732769087,v1:172.21.15.98:6801/1732769087] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mon", "unmanaged": true, "target": ["mon-mgr", ""]}) v1 -- 0x7f8754060a80 con 0x7f8740038710 2023-07-14T19:22:49.769 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:49.417+0000 7f874affd700 1 -- 172.21.15.98:0/652414074 <== mgr.14120 v2:172.21.15.98:6800/1732769087 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+24 (secure 0 0 0) 0x7f8754060a80 con 0x7f8740038710 2023-07-14T19:22:49.769 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:49.419+0000 7f875bc88700 1 -- 172.21.15.98:0/652414074 >> [v2:172.21.15.98:6800/1732769087,v1:172.21.15.98:6801/1732769087] conn(0x7f8740038710 msgr2=0x7f874003abc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:22:49.769 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:49.419+0000 7f875bc88700 1 --2- 172.21.15.98:0/652414074 >> [v2:172.21.15.98:6800/1732769087,v1:172.21.15.98:6801/1732769087] conn(0x7f8740038710 0x7f874003abc0 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f8750006fd0 tx=0x7f8750006e40 comp rx=0 tx=0).stop 2023-07-14T19:22:49.770 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:49.419+0000 7f875bc88700 1 -- 172.21.15.98:0/652414074 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8754106c80 msgr2=0x7f8754199450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:22:49.770 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:49.419+0000 7f875bc88700 1 --2- 172.21.15.98:0/652414074 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8754106c80 0x7f8754199450 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f8744009130 tx=0x7f8744011ad0 comp rx=0 tx=0).stop 2023-07-14T19:22:49.770 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:49.419+0000 7f875bc88700 1 -- 172.21.15.98:0/652414074 shutdown_connections 2023-07-14T19:22:49.770 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:49.419+0000 7f875bc88700 1 --2- 172.21.15.98:0/652414074 >> [v2:172.21.15.98:6800/1732769087,v1:172.21.15.98:6801/1732769087] conn(0x7f8740038710 0x7f874003abc0 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:22:49.770 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:49.419+0000 7f875bc88700 1 --2- 172.21.15.98:0/652414074 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8754106c80 0x7f8754199450 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:22:49.770 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:49.419+0000 7f875bc88700 1 -- 172.21.15.98:0/652414074 >> 172.21.15.98:0/652414074 conn(0x7f87540ff330 msgr2=0x7f8754100ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:22:49.770 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:49.419+0000 7f875bc88700 1 -- 172.21.15.98:0/652414074 shutdown_connections 2023-07-14T19:22:49.771 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:49.419+0000 7f875bc88700 1 -- 172.21.15.98:0/652414074 wait complete. 2023-07-14T19:22:49.771 INFO:teuthology.orchestra.run.smithi098.stdout:Deploying unmanaged mgr service... 2023-07-14T19:22:50.562 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:22:50 smithi098 ceph-mon[127486]: from='client.14142 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "unmanaged": true, "target": ["mon-mgr", ""]}]: dispatch 2023-07-14T19:22:50.562 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:22:50 smithi098 ceph-mon[127486]: Saving service mon spec with placement count:5 2023-07-14T19:22:50.563 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:22:50 smithi098 ceph-mon[127486]: from='mgr.14120 172.21.15.98:0/3060674050' entity='mgr.x' 2023-07-14T19:22:50.982 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout Scheduled mgr update... 2023-07-14T19:22:50.982 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:50.397+0000 7f3bdfdfb700 1 Processor -- start 2023-07-14T19:22:50.982 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:50.398+0000 7f3bdfdfb700 1 -- start start 2023-07-14T19:22:50.983 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:50.398+0000 7f3bdfdfb700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3bd81061a0 0x7f3bd8108580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:22:50.985 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:50.398+0000 7f3bdfdfb700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3bd8079d20 con 0x7f3bd81061a0 2023-07-14T19:22:50.985 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:50.398+0000 7f3bddb97700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3bd81061a0 0x7f3bd8108580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:22:50.985 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:50.398+0000 7f3bddb97700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3bd81061a0 0x7f3bd8108580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:44524/0 (socket says 172.21.15.98:44524) 2023-07-14T19:22:50.985 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:50.398+0000 7f3bddb97700 1 -- 172.21.15.98:0/3894095125 learned_addr learned my addr 172.21.15.98:0/3894095125 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-07-14T19:22:50.985 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:50.398+0000 7f3bddb97700 1 -- 172.21.15.98:0/3894095125 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3bd8079e60 con 0x7f3bd81061a0 2023-07-14T19:22:50.985 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:50.398+0000 7f3bddb97700 1 --2- 172.21.15.98:0/3894095125 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3bd81061a0 0x7f3bd8108580 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f3bd40098a0 tx=0x7f3bd4009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=25790a16c52717bd server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:22:50.986 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:50.399+0000 7f3bdcb95700 1 -- 172.21.15.98:0/3894095125 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f3bd4004160 con 0x7f3bd81061a0 2023-07-14T19:22:50.986 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:50.399+0000 7f3bdcb95700 1 -- 172.21.15.98:0/3894095125 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f3bd40042c0 con 0x7f3bd81061a0 2023-07-14T19:22:50.986 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:50.399+0000 7f3bdcb95700 1 -- 172.21.15.98:0/3894095125 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f3bd4011a00 con 0x7f3bd81061a0 2023-07-14T19:22:50.986 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:50.399+0000 7f3bdfdfb700 1 -- 172.21.15.98:0/3894095125 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3bd81061a0 msgr2=0x7f3bd8108580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:22:50.986 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:50.399+0000 7f3bdfdfb700 1 --2- 172.21.15.98:0/3894095125 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3bd81061a0 0x7f3bd8108580 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f3bd40098a0 tx=0x7f3bd4009bb0 comp rx=0 tx=0).stop 2023-07-14T19:22:50.986 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:50.399+0000 7f3bdfdfb700 1 -- 172.21.15.98:0/3894095125 shutdown_connections 2023-07-14T19:22:50.986 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:50.399+0000 7f3bdfdfb700 1 --2- 172.21.15.98:0/3894095125 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3bd81061a0 0x7f3bd8108580 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:22:50.987 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:50.399+0000 7f3bdfdfb700 1 -- 172.21.15.98:0/3894095125 >> 172.21.15.98:0/3894095125 conn(0x7f3bd80ffba0 msgr2=0x7f3bd8101fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:22:50.987 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:50.399+0000 7f3bdfdfb700 1 -- 172.21.15.98:0/3894095125 shutdown_connections 2023-07-14T19:22:50.987 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:50.399+0000 7f3bdfdfb700 1 -- 172.21.15.98:0/3894095125 wait complete. 2023-07-14T19:22:50.987 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:50.400+0000 7f3bdfdfb700 1 Processor -- start 2023-07-14T19:22:50.987 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:50.400+0000 7f3bdfdfb700 1 -- start start 2023-07-14T19:22:50.987 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:50.400+0000 7f3bdfdfb700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3bd81061a0 0x7f3bd8105b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:22:50.987 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:50.400+0000 7f3bdfdfb700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3bd8106040 con 0x7f3bd81061a0 2023-07-14T19:22:50.987 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:50.400+0000 7f3bddb97700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3bd81061a0 0x7f3bd8105b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:22:50.988 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:50.400+0000 7f3bddb97700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3bd81061a0 0x7f3bd8105b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:44526/0 (socket says 172.21.15.98:44526) 2023-07-14T19:22:50.988 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:50.400+0000 7f3bddb97700 1 -- 172.21.15.98:0/706552337 learned_addr learned my addr 172.21.15.98:0/706552337 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-07-14T19:22:50.988 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:50.401+0000 7f3bddb97700 1 -- 172.21.15.98:0/706552337 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3bd4009550 con 0x7f3bd81061a0 2023-07-14T19:22:50.988 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:50.401+0000 7f3bddb97700 1 --2- 172.21.15.98:0/706552337 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3bd81061a0 0x7f3bd8105b00 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f3bd4009fb0 tx=0x7f3bd4011ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:22:50.988 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:50.401+0000 7f3bceffd700 1 -- 172.21.15.98:0/706552337 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f3bd4003930 con 0x7f3bd81061a0 2023-07-14T19:22:50.988 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:50.401+0000 7f3bceffd700 1 -- 172.21.15.98:0/706552337 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f3bd4003a90 con 0x7f3bd81061a0 2023-07-14T19:22:50.989 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:50.401+0000 7f3bceffd700 1 -- 172.21.15.98:0/706552337 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f3bd40195e0 con 0x7f3bd81061a0 2023-07-14T19:22:50.989 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:50.401+0000 7f3bdfdfb700 1 -- 172.21.15.98:0/706552337 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3bd81041b0 con 0x7f3bd81061a0 2023-07-14T19:22:50.989 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:50.401+0000 7f3bdfdfb700 1 -- 172.21.15.98:0/706552337 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3bd8104630 con 0x7f3bd81061a0 2023-07-14T19:22:50.989 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:50.402+0000 7f3bceffd700 1 -- 172.21.15.98:0/706552337 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 8) v1 ==== 45483+0+0 (secure 0 0 0) 0x7f3bd401a430 con 0x7f3bd81061a0 2023-07-14T19:22:50.989 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:50.402+0000 7f3bdfdfb700 1 -- 172.21.15.98:0/706552337 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3bd81096c0 con 0x7f3bd81061a0 2023-07-14T19:22:50.989 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:50.402+0000 7f3bceffd700 1 --2- 172.21.15.98:0/706552337 >> [v2:172.21.15.98:6800/1732769087,v1:172.21.15.98:6801/1732769087] conn(0x7f3bc40386c0 0x7f3bc403ab70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:22:50.989 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:50.402+0000 7f3bceffd700 1 -- 172.21.15.98:0/706552337 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7f3bd404c790 con 0x7f3bd81061a0 2023-07-14T19:22:50.990 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:50.402+0000 7f3bdd396700 1 --2- 172.21.15.98:0/706552337 >> [v2:172.21.15.98:6800/1732769087,v1:172.21.15.98:6801/1732769087] conn(0x7f3bc40386c0 0x7f3bc403ab70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:22:50.990 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:50.402+0000 7f3bdd396700 1 --2- 172.21.15.98:0/706552337 >> [v2:172.21.15.98:6800/1732769087,v1:172.21.15.98:6801/1732769087] conn(0x7f3bc40386c0 0x7f3bc403ab70 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f3bc8006fd0 tx=0x7f3bc8006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:22:50.990 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:50.407+0000 7f3bceffd700 1 -- 172.21.15.98:0/706552337 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f3bd4023b80 con 0x7f3bd81061a0 2023-07-14T19:22:50.990 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:50.605+0000 7f3bdfdfb700 1 -- 172.21.15.98:0/706552337 --> [v2:172.21.15.98:6800/1732769087,v1:172.21.15.98:6801/1732769087] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mgr", "unmanaged": true, "target": ["mon-mgr", ""]}) v1 -- 0x7f3bd8101f10 con 0x7f3bc40386c0 2023-07-14T19:22:50.990 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:50.610+0000 7f3bceffd700 1 -- 172.21.15.98:0/706552337 <== mgr.14120 v2:172.21.15.98:6800/1732769087 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+24 (secure 0 0 0) 0x7f3bd8101f10 con 0x7f3bc40386c0 2023-07-14T19:22:50.990 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:50.612+0000 7f3bdfdfb700 1 -- 172.21.15.98:0/706552337 >> [v2:172.21.15.98:6800/1732769087,v1:172.21.15.98:6801/1732769087] conn(0x7f3bc40386c0 msgr2=0x7f3bc403ab70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:22:50.990 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:50.612+0000 7f3bdfdfb700 1 --2- 172.21.15.98:0/706552337 >> [v2:172.21.15.98:6800/1732769087,v1:172.21.15.98:6801/1732769087] conn(0x7f3bc40386c0 0x7f3bc403ab70 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f3bc8006fd0 tx=0x7f3bc8006e40 comp rx=0 tx=0).stop 2023-07-14T19:22:50.991 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:50.612+0000 7f3bdfdfb700 1 -- 172.21.15.98:0/706552337 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3bd81061a0 msgr2=0x7f3bd8105b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:22:50.991 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:50.612+0000 7f3bdfdfb700 1 --2- 172.21.15.98:0/706552337 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3bd81061a0 0x7f3bd8105b00 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f3bd4009fb0 tx=0x7f3bd4011ad0 comp rx=0 tx=0).stop 2023-07-14T19:22:50.991 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:50.612+0000 7f3bdfdfb700 1 -- 172.21.15.98:0/706552337 shutdown_connections 2023-07-14T19:22:50.991 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:50.612+0000 7f3bdfdfb700 1 --2- 172.21.15.98:0/706552337 >> [v2:172.21.15.98:6800/1732769087,v1:172.21.15.98:6801/1732769087] conn(0x7f3bc40386c0 0x7f3bc403ab70 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:22:50.991 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:50.612+0000 7f3bdfdfb700 1 --2- 172.21.15.98:0/706552337 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3bd81061a0 0x7f3bd8105b00 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:22:50.991 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:50.612+0000 7f3bdfdfb700 1 -- 172.21.15.98:0/706552337 >> 172.21.15.98:0/706552337 conn(0x7f3bd80ffba0 msgr2=0x7f3bd81012f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:22:50.991 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:50.612+0000 7f3bdfdfb700 1 -- 172.21.15.98:0/706552337 shutdown_connections 2023-07-14T19:22:50.991 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:50.612+0000 7f3bdfdfb700 1 -- 172.21.15.98:0/706552337 wait complete. 2023-07-14T19:22:51.870 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:22:51 smithi098 ceph-mon[127486]: from='client.14144 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "unmanaged": true, "target": ["mon-mgr", ""]}]: dispatch 2023-07-14T19:22:51.871 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:22:51 smithi098 ceph-mon[127486]: Saving service mgr spec with placement count:2 2023-07-14T19:22:51.871 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:22:51 smithi098 ceph-mon[127486]: from='mgr.14120 172.21.15.98:0/3060674050' entity='mgr.x' 2023-07-14T19:22:51.871 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:22:51 smithi098 ceph-mon[127486]: from='mgr.14120 172.21.15.98:0/3060674050' entity='mgr.x' 2023-07-14T19:22:51.871 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:22:51 smithi098 ceph-mon[127486]: from='mgr.14120 172.21.15.98:0/3060674050' entity='mgr.x' 2023-07-14T19:22:52.125 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:51.484+0000 7ff4a7e2d700 1 Processor -- start 2023-07-14T19:22:52.126 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:51.484+0000 7ff4a7e2d700 1 -- start start 2023-07-14T19:22:52.126 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:51.484+0000 7ff4a7e2d700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff4a0105bf0 0x7ff4a0106000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:22:52.126 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:51.484+0000 7ff4a7e2d700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff4a0106540 con 0x7ff4a0105bf0 2023-07-14T19:22:52.126 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:51.484+0000 7ff4a5bc9700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff4a0105bf0 0x7ff4a0106000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:22:52.126 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:51.484+0000 7ff4a5bc9700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff4a0105bf0 0x7ff4a0106000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:44530/0 (socket says 172.21.15.98:44530) 2023-07-14T19:22:52.127 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:51.484+0000 7ff4a5bc9700 1 -- 172.21.15.98:0/2246293859 learned_addr learned my addr 172.21.15.98:0/2246293859 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-07-14T19:22:52.127 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:51.485+0000 7ff4a5bc9700 1 -- 172.21.15.98:0/2246293859 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff4a0106680 con 0x7ff4a0105bf0 2023-07-14T19:22:52.127 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:51.485+0000 7ff4a5bc9700 1 --2- 172.21.15.98:0/2246293859 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff4a0105bf0 0x7ff4a0106000 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7ff4900098a0 tx=0x7ff490009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ec4c01d6b2ed1005 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:22:52.127 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:51.485+0000 7ff4a4bc7700 1 -- 172.21.15.98:0/2246293859 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7ff490004160 con 0x7ff4a0105bf0 2023-07-14T19:22:52.127 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:51.485+0000 7ff4a4bc7700 1 -- 172.21.15.98:0/2246293859 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7ff4900042c0 con 0x7ff4a0105bf0 2023-07-14T19:22:52.127 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:51.485+0000 7ff4a4bc7700 1 -- 172.21.15.98:0/2246293859 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7ff4900119b0 con 0x7ff4a0105bf0 2023-07-14T19:22:52.127 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:51.486+0000 7ff4a7e2d700 1 -- 172.21.15.98:0/2246293859 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff4a0105bf0 msgr2=0x7ff4a0106000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:22:52.128 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:51.486+0000 7ff4a7e2d700 1 --2- 172.21.15.98:0/2246293859 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff4a0105bf0 0x7ff4a0106000 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7ff4900098a0 tx=0x7ff490009bb0 comp rx=0 tx=0).stop 2023-07-14T19:22:52.128 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:51.486+0000 7ff4a7e2d700 1 -- 172.21.15.98:0/2246293859 shutdown_connections 2023-07-14T19:22:52.128 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:51.486+0000 7ff4a7e2d700 1 --2- 172.21.15.98:0/2246293859 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff4a0105bf0 0x7ff4a0106000 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:22:52.128 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:51.486+0000 7ff4a7e2d700 1 -- 172.21.15.98:0/2246293859 >> 172.21.15.98:0/2246293859 conn(0x7ff4a00ff330 msgr2=0x7ff4a0101760 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:22:52.128 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:51.486+0000 7ff4a7e2d700 1 -- 172.21.15.98:0/2246293859 shutdown_connections 2023-07-14T19:22:52.128 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:51.486+0000 7ff4a7e2d700 1 -- 172.21.15.98:0/2246293859 wait complete. 2023-07-14T19:22:52.129 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:51.486+0000 7ff4a7e2d700 1 Processor -- start 2023-07-14T19:22:52.129 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:51.486+0000 7ff4a7e2d700 1 -- start start 2023-07-14T19:22:52.129 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:51.486+0000 7ff4a7e2d700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff4a0105bf0 0x7ff4a010a3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:22:52.129 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:51.486+0000 7ff4a7e2d700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff4a010a910 con 0x7ff4a0105bf0 2023-07-14T19:22:52.129 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:51.487+0000 7ff4a5bc9700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff4a0105bf0 0x7ff4a010a3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:22:52.129 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:51.487+0000 7ff4a5bc9700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff4a0105bf0 0x7ff4a010a3d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:44532/0 (socket says 172.21.15.98:44532) 2023-07-14T19:22:52.129 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:51.487+0000 7ff4a5bc9700 1 -- 172.21.15.98:0/3833649767 learned_addr learned my addr 172.21.15.98:0/3833649767 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-07-14T19:22:52.130 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:51.487+0000 7ff4a5bc9700 1 -- 172.21.15.98:0/3833649767 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff490009550 con 0x7ff4a0105bf0 2023-07-14T19:22:52.130 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:51.487+0000 7ff4a5bc9700 1 --2- 172.21.15.98:0/3833649767 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff4a0105bf0 0x7ff4a010a3d0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7ff490006af0 tx=0x7ff490011a10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:22:52.130 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:51.487+0000 7ff496ffd700 1 -- 172.21.15.98:0/3833649767 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7ff490003a40 con 0x7ff4a0105bf0 2023-07-14T19:22:52.130 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:51.487+0000 7ff496ffd700 1 -- 172.21.15.98:0/3833649767 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7ff490003ba0 con 0x7ff4a0105bf0 2023-07-14T19:22:52.130 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:51.487+0000 7ff4a7e2d700 1 -- 172.21.15.98:0/3833649767 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff4a010ab10 con 0x7ff4a0105bf0 2023-07-14T19:22:52.130 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:51.487+0000 7ff496ffd700 1 -- 172.21.15.98:0/3833649767 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7ff4900196c0 con 0x7ff4a0105bf0 2023-07-14T19:22:52.130 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:51.487+0000 7ff4a7e2d700 1 -- 172.21.15.98:0/3833649767 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff4a0107000 con 0x7ff4a0105bf0 2023-07-14T19:22:52.131 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:51.488+0000 7ff496ffd700 1 -- 172.21.15.98:0/3833649767 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 8) v1 ==== 45483+0+0 (secure 0 0 0) 0x7ff49001aa80 con 0x7ff4a0105bf0 2023-07-14T19:22:52.131 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:51.488+0000 7ff4a7e2d700 1 -- 172.21.15.98:0/3833649767 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff4a010aca0 con 0x7ff4a0105bf0 2023-07-14T19:22:52.131 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:51.488+0000 7ff496ffd700 1 --2- 172.21.15.98:0/3833649767 >> [v2:172.21.15.98:6800/1732769087,v1:172.21.15.98:6801/1732769087] conn(0x7ff48c038710 0x7ff48c03abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:22:52.132 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:51.488+0000 7ff496ffd700 1 -- 172.21.15.98:0/3833649767 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7ff49004c740 con 0x7ff4a0105bf0 2023-07-14T19:22:52.132 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:51.488+0000 7ff4a53c8700 1 --2- 172.21.15.98:0/3833649767 >> [v2:172.21.15.98:6800/1732769087,v1:172.21.15.98:6801/1732769087] conn(0x7ff48c038710 0x7ff48c03abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:22:52.132 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:51.489+0000 7ff4a53c8700 1 --2- 172.21.15.98:0/3833649767 >> [v2:172.21.15.98:6800/1732769087,v1:172.21.15.98:6801/1732769087] conn(0x7ff48c038710 0x7ff48c03abc0 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7ff49c006fd0 tx=0x7ff49c006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:22:52.133 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:51.493+0000 7ff496ffd700 1 -- 172.21.15.98:0/3833649767 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7ff49001a430 con 0x7ff4a0105bf0 2023-07-14T19:22:52.133 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:51.693+0000 7ff4a7e2d700 1 -- 172.21.15.98:0/3833649767 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command([{prefix=config set, name=mgr/cephadm/container_init}] v 0) v1 -- 0x7ff4a0061cd0 con 0x7ff4a0105bf0 2023-07-14T19:22:52.133 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:51.699+0000 7ff496ffd700 1 -- 172.21.15.98:0/3833649767 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/cephadm/container_init}]=0 v7)=0 v7) v1 ==== 142+0+0 (secure 0 0 0) 0x7ff49000b3e0 con 0x7ff4a0105bf0 2023-07-14T19:22:52.133 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:51.702+0000 7ff4a7e2d700 1 -- 172.21.15.98:0/3833649767 >> [v2:172.21.15.98:6800/1732769087,v1:172.21.15.98:6801/1732769087] conn(0x7ff48c038710 msgr2=0x7ff48c03abc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:22:52.133 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:51.702+0000 7ff4a7e2d700 1 --2- 172.21.15.98:0/3833649767 >> [v2:172.21.15.98:6800/1732769087,v1:172.21.15.98:6801/1732769087] conn(0x7ff48c038710 0x7ff48c03abc0 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7ff49c006fd0 tx=0x7ff49c006e40 comp rx=0 tx=0).stop 2023-07-14T19:22:52.133 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:51.702+0000 7ff4a7e2d700 1 -- 172.21.15.98:0/3833649767 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff4a0105bf0 msgr2=0x7ff4a010a3d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:22:52.133 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:51.702+0000 7ff4a7e2d700 1 --2- 172.21.15.98:0/3833649767 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff4a0105bf0 0x7ff4a010a3d0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7ff490006af0 tx=0x7ff490011a10 comp rx=0 tx=0).stop 2023-07-14T19:22:52.134 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:51.702+0000 7ff4a7e2d700 1 -- 172.21.15.98:0/3833649767 shutdown_connections 2023-07-14T19:22:52.134 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:51.702+0000 7ff4a7e2d700 1 --2- 172.21.15.98:0/3833649767 >> [v2:172.21.15.98:6800/1732769087,v1:172.21.15.98:6801/1732769087] conn(0x7ff48c038710 0x7ff48c03abc0 secure :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7ff49c006fd0 tx=0x7ff49c006e40 comp rx=0 tx=0).stop 2023-07-14T19:22:52.134 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:51.702+0000 7ff4a7e2d700 1 --2- 172.21.15.98:0/3833649767 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff4a0105bf0 0x7ff4a010a3d0 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:22:52.134 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:51.702+0000 7ff4a7e2d700 1 -- 172.21.15.98:0/3833649767 >> 172.21.15.98:0/3833649767 conn(0x7ff4a00ff330 msgr2=0x7ff4a0100ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:22:52.134 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:51.703+0000 7ff4a7e2d700 1 -- 172.21.15.98:0/3833649767 shutdown_connections 2023-07-14T19:22:52.134 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:51.703+0000 7ff4a7e2d700 1 -- 172.21.15.98:0/3833649767 wait complete. 2023-07-14T19:22:52.997 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:22:52 smithi098 ceph-mon[127486]: from='client.? 172.21.15.98:0/3833649767' entity='client.admin' 2023-07-14T19:22:53.369 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:52.741+0000 7fe2e261d700 1 Processor -- start 2023-07-14T19:22:53.370 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:52.741+0000 7fe2e261d700 1 -- start start 2023-07-14T19:22:53.370 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:52.741+0000 7fe2e261d700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe2dc073e60 0x7fe2dc108210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:22:53.370 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:52.741+0000 7fe2e261d700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe2dc108750 con 0x7fe2dc073e60 2023-07-14T19:22:53.370 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:52.742+0000 7fe2dbfff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe2dc073e60 0x7fe2dc108210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:22:53.370 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:52.742+0000 7fe2dbfff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe2dc073e60 0x7fe2dc108210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:44536/0 (socket says 172.21.15.98:44536) 2023-07-14T19:22:53.370 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:52.742+0000 7fe2dbfff700 1 -- 172.21.15.98:0/330002305 learned_addr learned my addr 172.21.15.98:0/330002305 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-07-14T19:22:53.371 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:52.742+0000 7fe2dbfff700 1 -- 172.21.15.98:0/330002305 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe2dc108890 con 0x7fe2dc073e60 2023-07-14T19:22:53.371 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:52.742+0000 7fe2dbfff700 1 --2- 172.21.15.98:0/330002305 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe2dc073e60 0x7fe2dc108210 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7fe2cc0098a0 tx=0x7fe2cc009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=85b1008eb875695e server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:22:53.372 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:52.742+0000 7fe2daffd700 1 -- 172.21.15.98:0/330002305 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fe2cc004160 con 0x7fe2dc073e60 2023-07-14T19:22:53.372 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:52.742+0000 7fe2daffd700 1 -- 172.21.15.98:0/330002305 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fe2cc0042c0 con 0x7fe2dc073e60 2023-07-14T19:22:53.372 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:52.742+0000 7fe2daffd700 1 -- 172.21.15.98:0/330002305 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fe2cc0119b0 con 0x7fe2dc073e60 2023-07-14T19:22:53.373 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:52.743+0000 7fe2e261d700 1 -- 172.21.15.98:0/330002305 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe2dc073e60 msgr2=0x7fe2dc108210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:22:53.373 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:52.743+0000 7fe2e261d700 1 --2- 172.21.15.98:0/330002305 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe2dc073e60 0x7fe2dc108210 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7fe2cc0098a0 tx=0x7fe2cc009bb0 comp rx=0 tx=0).stop 2023-07-14T19:22:53.373 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:52.743+0000 7fe2e261d700 1 -- 172.21.15.98:0/330002305 shutdown_connections 2023-07-14T19:22:53.373 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:52.743+0000 7fe2e261d700 1 --2- 172.21.15.98:0/330002305 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe2dc073e60 0x7fe2dc108210 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:22:53.373 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:52.743+0000 7fe2e261d700 1 -- 172.21.15.98:0/330002305 >> 172.21.15.98:0/330002305 conn(0x7fe2dc0ffbc0 msgr2=0x7fe2dc101ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:22:53.373 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:52.743+0000 7fe2e261d700 1 -- 172.21.15.98:0/330002305 shutdown_connections 2023-07-14T19:22:53.373 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:52.743+0000 7fe2e261d700 1 -- 172.21.15.98:0/330002305 wait complete. 2023-07-14T19:22:53.374 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:52.743+0000 7fe2e261d700 1 Processor -- start 2023-07-14T19:22:53.374 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:52.744+0000 7fe2e261d700 1 -- start start 2023-07-14T19:22:53.374 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:52.744+0000 7fe2e261d700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe2dc19b690 0x7fe2dc19baa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:22:53.374 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:52.744+0000 7fe2e261d700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe2dc19bfe0 con 0x7fe2dc19b690 2023-07-14T19:22:53.374 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:52.744+0000 7fe2dbfff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe2dc19b690 0x7fe2dc19baa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:22:53.374 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:52.744+0000 7fe2dbfff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe2dc19b690 0x7fe2dc19baa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:44538/0 (socket says 172.21.15.98:44538) 2023-07-14T19:22:53.374 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:52.744+0000 7fe2dbfff700 1 -- 172.21.15.98:0/893684156 learned_addr learned my addr 172.21.15.98:0/893684156 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-07-14T19:22:53.375 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:52.744+0000 7fe2dbfff700 1 -- 172.21.15.98:0/893684156 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe2cc009550 con 0x7fe2dc19b690 2023-07-14T19:22:53.375 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:52.744+0000 7fe2dbfff700 1 --2- 172.21.15.98:0/893684156 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe2dc19b690 0x7fe2dc19baa0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7fe2cc003e60 tx=0x7fe2cc011a30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:22:53.375 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:52.744+0000 7fe2d97fa700 1 -- 172.21.15.98:0/893684156 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fe2cc003a60 con 0x7fe2dc19b690 2023-07-14T19:22:53.375 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:52.744+0000 7fe2d97fa700 1 -- 172.21.15.98:0/893684156 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fe2cc003bc0 con 0x7fe2dc19b690 2023-07-14T19:22:53.376 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:52.744+0000 7fe2d97fa700 1 -- 172.21.15.98:0/893684156 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fe2cc0196c0 con 0x7fe2dc19b690 2023-07-14T19:22:53.376 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:52.745+0000 7fe2e261d700 1 -- 172.21.15.98:0/893684156 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe2dc19c1e0 con 0x7fe2dc19b690 2023-07-14T19:22:53.376 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:52.745+0000 7fe2e261d700 1 -- 172.21.15.98:0/893684156 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe2dc19ee40 con 0x7fe2dc19b690 2023-07-14T19:22:53.376 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:52.745+0000 7fe2d97fa700 1 -- 172.21.15.98:0/893684156 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 8) v1 ==== 45483+0+0 (secure 0 0 0) 0x7fe2cc019930 con 0x7fe2dc19b690 2023-07-14T19:22:53.376 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:52.745+0000 7fe2e261d700 1 -- 172.21.15.98:0/893684156 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe2dc195860 con 0x7fe2dc19b690 2023-07-14T19:22:53.376 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:52.746+0000 7fe2d97fa700 1 --2- 172.21.15.98:0/893684156 >> [v2:172.21.15.98:6800/1732769087,v1:172.21.15.98:6801/1732769087] conn(0x7fe2c4038740 0x7fe2c403abf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:22:53.376 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:52.746+0000 7fe2d97fa700 1 -- 172.21.15.98:0/893684156 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7fe2cc04c8b0 con 0x7fe2dc19b690 2023-07-14T19:22:53.377 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:52.746+0000 7fe2db7fe700 1 --2- 172.21.15.98:0/893684156 >> [v2:172.21.15.98:6800/1732769087,v1:172.21.15.98:6801/1732769087] conn(0x7fe2c4038740 0x7fe2c403abf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:22:53.377 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:52.746+0000 7fe2db7fe700 1 --2- 172.21.15.98:0/893684156 >> [v2:172.21.15.98:6800/1732769087,v1:172.21.15.98:6801/1732769087] conn(0x7fe2c4038740 0x7fe2c403abf0 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7fe2d0006fd0 tx=0x7fe2d0006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:22:53.377 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:52.750+0000 7fe2d97fa700 1 -- 172.21.15.98:0/893684156 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7fe2cc0189a0 con 0x7fe2dc19b690 2023-07-14T19:22:53.377 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:52.964+0000 7fe2e261d700 1 -- 172.21.15.98:0/893684156 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command([{prefix=config set, name=mgr/dashboard/ssl_server_port}] v 0) v1 -- 0x7fe2dc073e60 con 0x7fe2dc19b690 2023-07-14T19:22:53.377 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:52.970+0000 7fe2d97fa700 1 -- 172.21.15.98:0/893684156 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/dashboard/ssl_server_port}]=0 v8)=0 v8) v1 ==== 130+0+0 (secure 0 0 0) 0x7fe2cc018350 con 0x7fe2dc19b690 2023-07-14T19:22:53.377 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:52.973+0000 7fe2e261d700 1 -- 172.21.15.98:0/893684156 >> [v2:172.21.15.98:6800/1732769087,v1:172.21.15.98:6801/1732769087] conn(0x7fe2c4038740 msgr2=0x7fe2c403abf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:22:53.377 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:52.973+0000 7fe2e261d700 1 --2- 172.21.15.98:0/893684156 >> [v2:172.21.15.98:6800/1732769087,v1:172.21.15.98:6801/1732769087] conn(0x7fe2c4038740 0x7fe2c403abf0 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7fe2d0006fd0 tx=0x7fe2d0006e40 comp rx=0 tx=0).stop 2023-07-14T19:22:53.378 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:52.973+0000 7fe2e261d700 1 -- 172.21.15.98:0/893684156 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe2dc19b690 msgr2=0x7fe2dc19baa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:22:53.378 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:52.973+0000 7fe2e261d700 1 --2- 172.21.15.98:0/893684156 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe2dc19b690 0x7fe2dc19baa0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7fe2cc003e60 tx=0x7fe2cc011a30 comp rx=0 tx=0).stop 2023-07-14T19:22:53.378 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:52.973+0000 7fe2e261d700 1 -- 172.21.15.98:0/893684156 shutdown_connections 2023-07-14T19:22:53.378 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:52.973+0000 7fe2e261d700 1 --2- 172.21.15.98:0/893684156 >> [v2:172.21.15.98:6800/1732769087,v1:172.21.15.98:6801/1732769087] conn(0x7fe2c4038740 0x7fe2c403abf0 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:22:53.378 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:52.973+0000 7fe2e261d700 1 --2- 172.21.15.98:0/893684156 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe2dc19b690 0x7fe2dc19baa0 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:22:53.378 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:52.973+0000 7fe2e261d700 1 -- 172.21.15.98:0/893684156 >> 172.21.15.98:0/893684156 conn(0x7fe2dc0ffbc0 msgr2=0x7fe2dc101320 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:22:53.378 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:52.973+0000 7fe2e261d700 1 -- 172.21.15.98:0/893684156 shutdown_connections 2023-07-14T19:22:53.378 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:52.973+0000 7fe2e261d700 1 -- 172.21.15.98:0/893684156 wait complete. 2023-07-14T19:22:53.379 INFO:teuthology.orchestra.run.smithi098.stdout:Enabling the dashboard module... 2023-07-14T19:22:54.281 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:22:53 smithi098 ceph-mon[127486]: from='mgr.14120 172.21.15.98:0/3060674050' entity='mgr.x' 2023-07-14T19:22:54.282 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:22:53 smithi098 ceph-mon[127486]: from='client.? 172.21.15.98:0/893684156' entity='client.admin' 2023-07-14T19:22:54.282 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:22:53 smithi098 ceph-mon[127486]: from='mgr.14120 172.21.15.98:0/3060674050' entity='mgr.x' 2023-07-14T19:22:54.907 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:22:54 smithi098 ceph-mon[127486]: from='client.? 172.21.15.98:0/1194116075' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "dashboard"}]: dispatch 2023-07-14T19:22:55.227 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Jul 14 19:22:54 smithi098 ceph-9adea648-227b-11ee-9b33-001a4aab830c-mgr-x[127734]: ignoring --setuser ceph since I am not root 2023-07-14T19:22:55.228 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Jul 14 19:22:54 smithi098 ceph-9adea648-227b-11ee-9b33-001a4aab830c-mgr-x[127734]: ignoring --setgroup ceph since I am not root 2023-07-14T19:22:55.228 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Jul 14 19:22:55 smithi098 ceph-9adea648-227b-11ee-9b33-001a4aab830c-mgr-x[127734]: 2023-07-14T19:22:55.059+0000 7f24f3568200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-07-14T19:22:55.228 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Jul 14 19:22:55 smithi098 ceph-9adea648-227b-11ee-9b33-001a4aab830c-mgr-x[127734]: 2023-07-14T19:22:55.139+0000 7f24f3568200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-07-14T19:22:55.423 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:53.832+0000 7feeeca81700 1 Processor -- start 2023-07-14T19:22:55.424 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:53.832+0000 7feeeca81700 1 -- start start 2023-07-14T19:22:55.424 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:53.832+0000 7feeeca81700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7feee8105b60 0x7feee8105f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:22:55.424 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:53.832+0000 7feeeca81700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feee81064b0 con 0x7feee8105b60 2023-07-14T19:22:55.424 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:53.833+0000 7feee659c700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7feee8105b60 0x7feee8105f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:22:55.424 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:53.833+0000 7feee659c700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7feee8105b60 0x7feee8105f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:44542/0 (socket says 172.21.15.98:44542) 2023-07-14T19:22:55.424 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:53.833+0000 7feee659c700 1 -- 172.21.15.98:0/2571003427 learned_addr learned my addr 172.21.15.98:0/2571003427 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-07-14T19:22:55.425 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:53.833+0000 7feee659c700 1 -- 172.21.15.98:0/2571003427 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feee81065f0 con 0x7feee8105b60 2023-07-14T19:22:55.425 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:53.833+0000 7feee659c700 1 --2- 172.21.15.98:0/2571003427 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7feee8105b60 0x7feee8105f70 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7feedc0098a0 tx=0x7feedc009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=738a12c28d84b27b server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:22:55.425 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:53.833+0000 7feee559a700 1 -- 172.21.15.98:0/2571003427 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7feedc004160 con 0x7feee8105b60 2023-07-14T19:22:55.425 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:53.833+0000 7feee559a700 1 -- 172.21.15.98:0/2571003427 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7feedc0042c0 con 0x7feee8105b60 2023-07-14T19:22:55.425 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:53.833+0000 7feee559a700 1 -- 172.21.15.98:0/2571003427 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7feedc0119b0 con 0x7feee8105b60 2023-07-14T19:22:55.425 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:53.834+0000 7feeeca81700 1 -- 172.21.15.98:0/2571003427 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7feee8105b60 msgr2=0x7feee8105f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:22:55.425 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:53.834+0000 7feeeca81700 1 --2- 172.21.15.98:0/2571003427 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7feee8105b60 0x7feee8105f70 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7feedc0098a0 tx=0x7feedc009bb0 comp rx=0 tx=0).stop 2023-07-14T19:22:55.426 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:53.834+0000 7feeeca81700 1 -- 172.21.15.98:0/2571003427 shutdown_connections 2023-07-14T19:22:55.426 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:53.834+0000 7feeeca81700 1 --2- 172.21.15.98:0/2571003427 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7feee8105b60 0x7feee8105f70 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:22:55.426 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:53.834+0000 7feeeca81700 1 -- 172.21.15.98:0/2571003427 >> 172.21.15.98:0/2571003427 conn(0x7feee80ff330 msgr2=0x7feee8101760 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:22:55.426 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:53.834+0000 7feeeca81700 1 -- 172.21.15.98:0/2571003427 shutdown_connections 2023-07-14T19:22:55.426 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:53.834+0000 7feeeca81700 1 -- 172.21.15.98:0/2571003427 wait complete. 2023-07-14T19:22:55.426 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:53.835+0000 7feeeca81700 1 Processor -- start 2023-07-14T19:22:55.426 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:53.835+0000 7feeeca81700 1 -- start start 2023-07-14T19:22:55.426 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:53.835+0000 7feeeca81700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7feee8105b60 0x7feee81983f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:22:55.427 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:53.835+0000 7feeeca81700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feee8198930 con 0x7feee8105b60 2023-07-14T19:22:55.427 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:53.835+0000 7feee659c700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7feee8105b60 0x7feee81983f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:22:55.427 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:53.835+0000 7feee659c700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7feee8105b60 0x7feee81983f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:44544/0 (socket says 172.21.15.98:44544) 2023-07-14T19:22:55.427 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:53.835+0000 7feee659c700 1 -- 172.21.15.98:0/1194116075 learned_addr learned my addr 172.21.15.98:0/1194116075 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-07-14T19:22:55.427 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:53.835+0000 7feee659c700 1 -- 172.21.15.98:0/1194116075 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feedc009550 con 0x7feee8105b60 2023-07-14T19:22:55.427 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:53.835+0000 7feee659c700 1 --2- 172.21.15.98:0/1194116075 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7feee8105b60 0x7feee81983f0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7feedc009130 tx=0x7feedc011a30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:22:55.427 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:53.836+0000 7feed77fe700 1 -- 172.21.15.98:0/1194116075 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7feedc003aa0 con 0x7feee8105b60 2023-07-14T19:22:55.428 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:53.836+0000 7feeeca81700 1 -- 172.21.15.98:0/1194116075 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7feee8198b30 con 0x7feee8105b60 2023-07-14T19:22:55.428 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:53.836+0000 7feed77fe700 1 -- 172.21.15.98:0/1194116075 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7feedc003c00 con 0x7feee8105b60 2023-07-14T19:22:55.428 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:53.836+0000 7feed77fe700 1 -- 172.21.15.98:0/1194116075 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7feedc0247a0 con 0x7feee8105b60 2023-07-14T19:22:55.428 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:53.836+0000 7feeeca81700 1 -- 172.21.15.98:0/1194116075 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7feee8198f50 con 0x7feee8105b60 2023-07-14T19:22:55.428 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:53.836+0000 7feed77fe700 1 -- 172.21.15.98:0/1194116075 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 8) v1 ==== 45483+0+0 (secure 0 0 0) 0x7feedc024940 con 0x7feee8105b60 2023-07-14T19:22:55.428 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:53.837+0000 7feeeca81700 1 -- 172.21.15.98:0/1194116075 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7feee8192370 con 0x7feee8105b60 2023-07-14T19:22:55.428 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:53.837+0000 7feed77fe700 1 --2- 172.21.15.98:0/1194116075 >> [v2:172.21.15.98:6800/1732769087,v1:172.21.15.98:6801/1732769087] conn(0x7feed0038710 0x7feed003abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:22:55.428 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:53.837+0000 7feed77fe700 1 -- 172.21.15.98:0/1194116075 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7feedc04ca10 con 0x7feee8105b60 2023-07-14T19:22:55.429 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:53.837+0000 7feee5d9b700 1 --2- 172.21.15.98:0/1194116075 >> [v2:172.21.15.98:6800/1732769087,v1:172.21.15.98:6801/1732769087] conn(0x7feed0038710 0x7feed003abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:22:55.429 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:53.837+0000 7feee5d9b700 1 --2- 172.21.15.98:0/1194116075 >> [v2:172.21.15.98:6800/1732769087,v1:172.21.15.98:6801/1732769087] conn(0x7feed0038710 0x7feed003abc0 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7feed8006fd0 tx=0x7feed8006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:22:55.429 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:53.841+0000 7feed77fe700 1 -- 172.21.15.98:0/1194116075 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7feedc024bf0 con 0x7feee8105b60 2023-07-14T19:22:55.429 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:54.097+0000 7feeeca81700 1 -- 172.21.15.98:0/1194116075 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "dashboard"} v 0) v1 -- 0x7feee8061250 con 0x7feee8105b60 2023-07-14T19:22:55.429 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:54.876+0000 7feed77fe700 1 -- 172.21.15.98:0/1194116075 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mgrmap(e 9) v1 ==== 45496+0+0 (secure 0 0 0) 0x7feedc02d9b0 con 0x7feee8105b60 2023-07-14T19:22:55.429 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:54.878+0000 7feed77fe700 1 -- 172.21.15.98:0/1194116075 <== mon.0 v2:172.21.15.98:3300/0 8 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "dashboard"}]=0 v9) v1 ==== 88+0+0 (secure 0 0 0) 0x7feedc02c780 con 0x7feee8105b60 2023-07-14T19:22:55.430 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:54.880+0000 7feeeca81700 1 -- 172.21.15.98:0/1194116075 >> [v2:172.21.15.98:6800/1732769087,v1:172.21.15.98:6801/1732769087] conn(0x7feed0038710 msgr2=0x7feed003abc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:22:55.430 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:54.880+0000 7feeeca81700 1 --2- 172.21.15.98:0/1194116075 >> [v2:172.21.15.98:6800/1732769087,v1:172.21.15.98:6801/1732769087] conn(0x7feed0038710 0x7feed003abc0 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7feed8006fd0 tx=0x7feed8006e40 comp rx=0 tx=0).stop 2023-07-14T19:22:55.430 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:54.880+0000 7feeeca81700 1 -- 172.21.15.98:0/1194116075 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7feee8105b60 msgr2=0x7feee81983f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:22:55.430 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:54.880+0000 7feeeca81700 1 --2- 172.21.15.98:0/1194116075 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7feee8105b60 0x7feee81983f0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7feedc009130 tx=0x7feedc011a30 comp rx=0 tx=0).stop 2023-07-14T19:22:55.430 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:54.880+0000 7feeeca81700 1 -- 172.21.15.98:0/1194116075 shutdown_connections 2023-07-14T19:22:55.430 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:54.880+0000 7feeeca81700 1 --2- 172.21.15.98:0/1194116075 >> [v2:172.21.15.98:6800/1732769087,v1:172.21.15.98:6801/1732769087] conn(0x7feed0038710 0x7feed003abc0 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:22:55.430 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:54.880+0000 7feeeca81700 1 --2- 172.21.15.98:0/1194116075 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7feee8105b60 0x7feee81983f0 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:22:55.430 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:54.880+0000 7feeeca81700 1 -- 172.21.15.98:0/1194116075 >> 172.21.15.98:0/1194116075 conn(0x7feee80ff330 msgr2=0x7feee8100a70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:22:55.431 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:54.882+0000 7feeeca81700 1 -- 172.21.15.98:0/1194116075 shutdown_connections 2023-07-14T19:22:55.431 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:54.882+0000 7feeeca81700 1 -- 172.21.15.98:0/1194116075 wait complete. 2023-07-14T19:22:55.508 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Jul 14 19:22:55 smithi098 ceph-9adea648-227b-11ee-9b33-001a4aab830c-mgr-x[127734]: 2023-07-14T19:22:55.234+0000 7f24f3568200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-07-14T19:22:55.509 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Jul 14 19:22:55 smithi098 ceph-9adea648-227b-11ee-9b33-001a4aab830c-mgr-x[127734]: 2023-07-14T19:22:55.327+0000 7f24f3568200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-07-14T19:22:55.781 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Jul 14 19:22:55 smithi098 ceph-9adea648-227b-11ee-9b33-001a4aab830c-mgr-x[127734]: 2023-07-14T19:22:55.506+0000 7f24f3568200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-07-14T19:22:56.152 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:22:55 smithi098 ceph-mon[127486]: from='client.? 172.21.15.98:0/1194116075' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "dashboard"}]': finished 2023-07-14T19:22:56.153 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:22:55 smithi098 ceph-mon[127486]: mgrmap e9: x(active, since 15s) 2023-07-14T19:22:56.153 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Jul 14 19:22:55 smithi098 ceph-9adea648-227b-11ee-9b33-001a4aab830c-mgr-x[127734]: 2023-07-14T19:22:55.825+0000 7f24f3568200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-07-14T19:22:56.153 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Jul 14 19:22:55 smithi098 ceph-9adea648-227b-11ee-9b33-001a4aab830c-mgr-x[127734]: 2023-07-14T19:22:55.919+0000 7f24f3568200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-07-14T19:22:56.153 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Jul 14 19:22:56 smithi098 ceph-9adea648-227b-11ee-9b33-001a4aab830c-mgr-x[127734]: 2023-07-14T19:22:56.054+0000 7f24f3568200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-07-14T19:22:56.531 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Jul 14 19:22:56 smithi098 ceph-9adea648-227b-11ee-9b33-001a4aab830c-mgr-x[127734]: 2023-07-14T19:22:56.150+0000 7f24f3568200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-07-14T19:22:56.569 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout { 2023-07-14T19:22:56.570 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "epoch": 9, 2023-07-14T19:22:56.570 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "available": true, 2023-07-14T19:22:56.570 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "active_name": "x", 2023-07-14T19:22:56.570 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "num_standby": 0 2023-07-14T19:22:56.570 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout } 2023-07-14T19:22:56.571 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:55.957+0000 7f2d5f48c700 1 Processor -- start 2023-07-14T19:22:56.571 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:55.958+0000 7f2d5f48c700 1 -- start start 2023-07-14T19:22:56.571 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:55.958+0000 7f2d5f48c700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2d58073e60 0x7f2d5810a610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:22:56.571 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:55.958+0000 7f2d5f48c700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2d58106000 con 0x7f2d58073e60 2023-07-14T19:22:56.571 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:55.958+0000 7f2d5d228700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2d58073e60 0x7f2d5810a610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:22:56.571 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:55.958+0000 7f2d5d228700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2d58073e60 0x7f2d5810a610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:44558/0 (socket says 172.21.15.98:44558) 2023-07-14T19:22:56.571 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:55.958+0000 7f2d5d228700 1 -- 172.21.15.98:0/1250705349 learned_addr learned my addr 172.21.15.98:0/1250705349 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-07-14T19:22:56.572 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:55.958+0000 7f2d5d228700 1 -- 172.21.15.98:0/1250705349 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2d5810ab50 con 0x7f2d58073e60 2023-07-14T19:22:56.572 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:55.958+0000 7f2d5d228700 1 --2- 172.21.15.98:0/1250705349 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2d58073e60 0x7f2d5810a610 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f2d540098a0 tx=0x7f2d54009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=21742340999cd7c2 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:22:56.572 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:55.959+0000 7f2d4ffff700 1 -- 172.21.15.98:0/1250705349 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f2d54004160 con 0x7f2d58073e60 2023-07-14T19:22:56.572 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:55.959+0000 7f2d4ffff700 1 -- 172.21.15.98:0/1250705349 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f2d540042c0 con 0x7f2d58073e60 2023-07-14T19:22:56.572 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:55.959+0000 7f2d4ffff700 1 -- 172.21.15.98:0/1250705349 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f2d540119b0 con 0x7f2d58073e60 2023-07-14T19:22:56.572 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:55.959+0000 7f2d5f48c700 1 -- 172.21.15.98:0/1250705349 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2d58073e60 msgr2=0x7f2d5810a610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:22:56.572 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:55.959+0000 7f2d5f48c700 1 --2- 172.21.15.98:0/1250705349 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2d58073e60 0x7f2d5810a610 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f2d540098a0 tx=0x7f2d54009bb0 comp rx=0 tx=0).stop 2023-07-14T19:22:56.572 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:55.959+0000 7f2d5f48c700 1 -- 172.21.15.98:0/1250705349 shutdown_connections 2023-07-14T19:22:56.573 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:55.959+0000 7f2d5f48c700 1 --2- 172.21.15.98:0/1250705349 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2d58073e60 0x7f2d5810a610 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:22:56.573 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:55.959+0000 7f2d5f48c700 1 -- 172.21.15.98:0/1250705349 >> 172.21.15.98:0/1250705349 conn(0x7f2d580ffbc0 msgr2=0x7f2d58101ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:22:56.573 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:55.960+0000 7f2d5f48c700 1 -- 172.21.15.98:0/1250705349 shutdown_connections 2023-07-14T19:22:56.573 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:55.960+0000 7f2d5f48c700 1 -- 172.21.15.98:0/1250705349 wait complete. 2023-07-14T19:22:56.573 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:55.960+0000 7f2d5f48c700 1 Processor -- start 2023-07-14T19:22:56.573 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:55.960+0000 7f2d5f48c700 1 -- start start 2023-07-14T19:22:56.573 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:55.960+0000 7f2d5f48c700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2d5819b670 0x7f2d5819ba80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:22:56.574 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:55.960+0000 7f2d5f48c700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2d5819bfc0 con 0x7f2d5819b670 2023-07-14T19:22:56.574 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:55.960+0000 7f2d5d228700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2d5819b670 0x7f2d5819ba80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:22:56.574 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:55.960+0000 7f2d5d228700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2d5819b670 0x7f2d5819ba80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:44560/0 (socket says 172.21.15.98:44560) 2023-07-14T19:22:56.574 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:55.960+0000 7f2d5d228700 1 -- 172.21.15.98:0/4059899310 learned_addr learned my addr 172.21.15.98:0/4059899310 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-07-14T19:22:56.574 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:55.961+0000 7f2d5d228700 1 -- 172.21.15.98:0/4059899310 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2d54009550 con 0x7f2d5819b670 2023-07-14T19:22:56.574 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:55.961+0000 7f2d5d228700 1 --2- 172.21.15.98:0/4059899310 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2d5819b670 0x7f2d5819ba80 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f2d54006af0 tx=0x7f2d54011a10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:22:56.574 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:55.961+0000 7f2d4e7fc700 1 -- 172.21.15.98:0/4059899310 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f2d54003a40 con 0x7f2d5819b670 2023-07-14T19:22:56.574 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:55.961+0000 7f2d4e7fc700 1 -- 172.21.15.98:0/4059899310 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f2d54003ba0 con 0x7f2d5819b670 2023-07-14T19:22:56.575 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:55.961+0000 7f2d5f48c700 1 -- 172.21.15.98:0/4059899310 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2d5819c1c0 con 0x7f2d5819b670 2023-07-14T19:22:56.575 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:55.961+0000 7f2d4e7fc700 1 -- 172.21.15.98:0/4059899310 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f2d54019690 con 0x7f2d5819b670 2023-07-14T19:22:56.575 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:55.961+0000 7f2d5f48c700 1 -- 172.21.15.98:0/4059899310 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2d5819ee20 con 0x7f2d5819b670 2023-07-14T19:22:56.575 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:55.962+0000 7f2d4e7fc700 1 -- 172.21.15.98:0/4059899310 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 9) v1 ==== 45496+0+0 (secure 0 0 0) 0x7f2d54019950 con 0x7f2d5819b670 2023-07-14T19:22:56.575 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:55.962+0000 7f2d4e7fc700 1 --2- 172.21.15.98:0/4059899310 >> [v2:172.21.15.98:6800/1732769087,v1:172.21.15.98:6801/1732769087] conn(0x7f2d44038710 0x7f2d4403abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:22:56.575 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:55.962+0000 7f2d4e7fc700 1 -- 172.21.15.98:0/4059899310 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7f2d5404c6d0 con 0x7f2d5819b670 2023-07-14T19:22:56.575 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:55.962+0000 7f2d5ca27700 1 -- 172.21.15.98:0/4059899310 >> [v2:172.21.15.98:6800/1732769087,v1:172.21.15.98:6801/1732769087] conn(0x7f2d44038710 msgr2=0x7f2d4403abc0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.98:6800/1732769087 2023-07-14T19:22:56.576 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:55.962+0000 7f2d5f48c700 1 -- 172.21.15.98:0/4059899310 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2d58195660 con 0x7f2d5819b670 2023-07-14T19:22:56.576 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:55.962+0000 7f2d5ca27700 1 --2- 172.21.15.98:0/4059899310 >> [v2:172.21.15.98:6800/1732769087,v1:172.21.15.98:6801/1732769087] conn(0x7f2d44038710 0x7f2d4403abc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-07-14T19:22:56.576 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:55.967+0000 7f2d4e7fc700 1 -- 172.21.15.98:0/4059899310 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f2d540243f0 con 0x7f2d5819b670 2023-07-14T19:22:56.576 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:56.163+0000 7f2d5ca27700 1 -- 172.21.15.98:0/4059899310 >> [v2:172.21.15.98:6800/1732769087,v1:172.21.15.98:6801/1732769087] conn(0x7f2d44038710 msgr2=0x7f2d4403abc0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.98:6800/1732769087 2023-07-14T19:22:56.576 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:56.163+0000 7f2d5ca27700 1 --2- 172.21.15.98:0/4059899310 >> [v2:172.21.15.98:6800/1732769087,v1:172.21.15.98:6801/1732769087] conn(0x7f2d44038710 0x7f2d4403abc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-07-14T19:22:56.576 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:56.214+0000 7f2d5f48c700 1 -- 172.21.15.98:0/4059899310 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "mgr stat"} v 0) v1 -- 0x7f2d5802c910 con 0x7f2d5819b670 2023-07-14T19:22:56.576 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:56.214+0000 7f2d4e7fc700 1 -- 172.21.15.98:0/4059899310 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "mgr stat"}]=0 v9) v1 ==== 56+0+88 (secure 0 0 0) 0x7f2d5404a030 con 0x7f2d5819b670 2023-07-14T19:22:56.576 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:56.216+0000 7f2d5f48c700 1 -- 172.21.15.98:0/4059899310 >> [v2:172.21.15.98:6800/1732769087,v1:172.21.15.98:6801/1732769087] conn(0x7f2d44038710 msgr2=0x7f2d4403abc0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-07-14T19:22:56.577 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:56.216+0000 7f2d5f48c700 1 --2- 172.21.15.98:0/4059899310 >> [v2:172.21.15.98:6800/1732769087,v1:172.21.15.98:6801/1732769087] conn(0x7f2d44038710 0x7f2d4403abc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:22:56.577 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:56.216+0000 7f2d5f48c700 1 -- 172.21.15.98:0/4059899310 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2d5819b670 msgr2=0x7f2d5819ba80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:22:56.577 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:56.216+0000 7f2d5f48c700 1 --2- 172.21.15.98:0/4059899310 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2d5819b670 0x7f2d5819ba80 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f2d54006af0 tx=0x7f2d54011a10 comp rx=0 tx=0).stop 2023-07-14T19:22:56.577 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:56.216+0000 7f2d5f48c700 1 -- 172.21.15.98:0/4059899310 shutdown_connections 2023-07-14T19:22:56.577 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:56.216+0000 7f2d5f48c700 1 --2- 172.21.15.98:0/4059899310 >> [v2:172.21.15.98:6800/1732769087,v1:172.21.15.98:6801/1732769087] conn(0x7f2d44038710 0x7f2d4403abc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:22:56.577 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:56.216+0000 7f2d5f48c700 1 --2- 172.21.15.98:0/4059899310 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2d5819b670 0x7f2d5819ba80 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:22:56.577 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:56.216+0000 7f2d5f48c700 1 -- 172.21.15.98:0/4059899310 >> 172.21.15.98:0/4059899310 conn(0x7f2d580ffbc0 msgr2=0x7f2d58101300 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:22:56.578 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:56.217+0000 7f2d5f48c700 1 -- 172.21.15.98:0/4059899310 shutdown_connections 2023-07-14T19:22:56.578 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:56.217+0000 7f2d5f48c700 1 -- 172.21.15.98:0/4059899310 wait complete. 2023-07-14T19:22:56.578 INFO:teuthology.orchestra.run.smithi098.stdout:Waiting for the mgr to restart... 2023-07-14T19:22:56.578 INFO:teuthology.orchestra.run.smithi098.stdout:Waiting for mgr epoch 9... 2023-07-14T19:22:57.031 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Jul 14 19:22:56 smithi098 ceph-9adea648-227b-11ee-9b33-001a4aab830c-mgr-x[127734]: 2023-07-14T19:22:56.587+0000 7f24f3568200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-07-14T19:22:57.531 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:22:57 smithi098 ceph-mon[127486]: from='client.? 172.21.15.98:0/4059899310' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2023-07-14T19:22:57.532 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Jul 14 19:22:57 smithi098 ceph-9adea648-227b-11ee-9b33-001a4aab830c-mgr-x[127734]: 2023-07-14T19:22:57.172+0000 7f24f3568200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2023-07-14T19:22:57.532 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Jul 14 19:22:57 smithi098 ceph-9adea648-227b-11ee-9b33-001a4aab830c-mgr-x[127734]: 2023-07-14T19:22:57.344+0000 7f24f3568200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-07-14T19:22:58.363 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Jul 14 19:22:58 smithi098 ceph-9adea648-227b-11ee-9b33-001a4aab830c-mgr-x[127734]: 2023-07-14T19:22:58.099+0000 7f24f3568200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-07-14T19:22:58.364 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Jul 14 19:22:58 smithi098 ceph-9adea648-227b-11ee-9b33-001a4aab830c-mgr-x[127734]: 2023-07-14T19:22:58.190+0000 7f24f3568200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-07-14T19:22:58.364 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Jul 14 19:22:58 smithi098 ceph-9adea648-227b-11ee-9b33-001a4aab830c-mgr-x[127734]: 2023-07-14T19:22:58.361+0000 7f24f3568200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-07-14T19:22:59.031 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Jul 14 19:22:58 smithi098 ceph-9adea648-227b-11ee-9b33-001a4aab830c-mgr-x[127734]: 2023-07-14T19:22:58.638+0000 7f24f3568200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2023-07-14T19:22:59.031 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Jul 14 19:22:58 smithi098 ceph-9adea648-227b-11ee-9b33-001a4aab830c-mgr-x[127734]: 2023-07-14T19:22:58.880+0000 7f24f3568200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-07-14T19:22:59.292 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Jul 14 19:22:59 smithi098 ceph-9adea648-227b-11ee-9b33-001a4aab830c-mgr-x[127734]: 2023-07-14T19:22:59.029+0000 7f24f3568200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-07-14T19:22:59.781 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Jul 14 19:22:59 smithi098 ceph-9adea648-227b-11ee-9b33-001a4aab830c-mgr-x[127734]: 2023-07-14T19:22:59.290+0000 7f24f3568200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-07-14T19:23:00.281 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Jul 14 19:23:00 smithi098 ceph-9adea648-227b-11ee-9b33-001a4aab830c-mgr-x[127734]: 2023-07-14T19:23:00.008+0000 7f24f3568200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-07-14T19:23:00.282 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Jul 14 19:23:00 smithi098 ceph-9adea648-227b-11ee-9b33-001a4aab830c-mgr-x[127734]: 2023-07-14T19:23:00.139+0000 7f24f3568200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-07-14T19:23:00.658 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Jul 14 19:23:00 smithi098 ceph-9adea648-227b-11ee-9b33-001a4aab830c-mgr-x[127734]: 2023-07-14T19:23:00.329+0000 7f24f3568200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-07-14T19:23:00.658 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Jul 14 19:23:00 smithi098 ceph-9adea648-227b-11ee-9b33-001a4aab830c-mgr-x[127734]: 2023-07-14T19:23:00.442+0000 7f24f3568200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-07-14T19:23:01.031 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Jul 14 19:23:00 smithi098 ceph-9adea648-227b-11ee-9b33-001a4aab830c-mgr-x[127734]: 2023-07-14T19:23:00.656+0000 7f24f3568200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-07-14T19:23:01.781 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Jul 14 19:23:01 smithi098 ceph-9adea648-227b-11ee-9b33-001a4aab830c-mgr-x[127734]: 2023-07-14T19:23:01.505+0000 7f24f3568200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-07-14T19:23:02.531 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:23:02 smithi098 ceph-mon[127486]: Active manager daemon x restarted 2023-07-14T19:23:02.532 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:23:02 smithi098 ceph-mon[127486]: Activating manager daemon x 2023-07-14T19:23:02.532 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:23:02 smithi098 ceph-mon[127486]: osdmap e3: 0 total, 0 up, 0 in 2023-07-14T19:23:02.532 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:23:02 smithi098 ceph-mon[127486]: mgrmap e10: x(active, starting, since 0.00335827s) 2023-07-14T19:23:02.532 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:23:02 smithi098 ceph-mon[127486]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2023-07-14T19:23:02.532 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:23:02 smithi098 ceph-mon[127486]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2023-07-14T19:23:02.533 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:23:02 smithi098 ceph-mon[127486]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2023-07-14T19:23:02.533 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:23:02 smithi098 ceph-mon[127486]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2023-07-14T19:23:02.533 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:23:02 smithi098 ceph-mon[127486]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2023-07-14T19:23:02.533 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:23:02 smithi098 ceph-mon[127486]: Manager daemon x is now available 2023-07-14T19:23:02.534 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:23:02 smithi098 ceph-mon[127486]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-14T19:23:02.534 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:23:02 smithi098 ceph-mon[127486]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-14T19:23:02.534 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:23:02 smithi098 ceph-mon[127486]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2023-07-14T19:23:02.534 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:23:02 smithi098 ceph-mon[127486]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2023-07-14T19:23:02.535 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:23:02 smithi098 ceph-mon[127486]: [14/Jul/2023:19:23:01] ENGINE Bus STARTING 2023-07-14T19:23:03.228 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout { 2023-07-14T19:23:03.229 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "mgrmap_epoch": 11, 2023-07-14T19:23:03.229 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "initialized": true 2023-07-14T19:23:03.229 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout } 2023-07-14T19:23:03.229 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:57.062+0000 7fb765ace700 1 Processor -- start 2023-07-14T19:23:03.229 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:57.062+0000 7fb765ace700 1 -- start start 2023-07-14T19:23:03.230 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:57.062+0000 7fb765ace700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb760105af0 0x7fb760105f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:23:03.230 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:57.062+0000 7fb765ace700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb760106440 con 0x7fb760105af0 2023-07-14T19:23:03.230 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:57.062+0000 7fb75f7fe700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb760105af0 0x7fb760105f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:23:03.230 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:57.062+0000 7fb75f7fe700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb760105af0 0x7fb760105f00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:44568/0 (socket says 172.21.15.98:44568) 2023-07-14T19:23:03.230 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:57.062+0000 7fb75f7fe700 1 -- 172.21.15.98:0/1692427902 learned_addr learned my addr 172.21.15.98:0/1692427902 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-07-14T19:23:03.230 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:57.063+0000 7fb75f7fe700 1 -- 172.21.15.98:0/1692427902 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb760106580 con 0x7fb760105af0 2023-07-14T19:23:03.230 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:57.063+0000 7fb75f7fe700 1 --2- 172.21.15.98:0/1692427902 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb760105af0 0x7fb760105f00 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7fb75000bb40 tx=0x7fb75000be50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6b0f0d3c26213d67 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:23:03.231 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:57.063+0000 7fb75e7fc700 1 -- 172.21.15.98:0/1692427902 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fb75000dbe0 con 0x7fb760105af0 2023-07-14T19:23:03.231 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:57.063+0000 7fb75e7fc700 1 -- 172.21.15.98:0/1692427902 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fb7500044f0 con 0x7fb760105af0 2023-07-14T19:23:03.231 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:57.063+0000 7fb75e7fc700 1 -- 172.21.15.98:0/1692427902 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fb750003e60 con 0x7fb760105af0 2023-07-14T19:23:03.231 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:57.064+0000 7fb765ace700 1 -- 172.21.15.98:0/1692427902 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb760105af0 msgr2=0x7fb760105f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:23:03.231 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:57.064+0000 7fb765ace700 1 --2- 172.21.15.98:0/1692427902 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb760105af0 0x7fb760105f00 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7fb75000bb40 tx=0x7fb75000be50 comp rx=0 tx=0).stop 2023-07-14T19:23:03.231 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:57.064+0000 7fb765ace700 1 -- 172.21.15.98:0/1692427902 shutdown_connections 2023-07-14T19:23:03.231 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:57.064+0000 7fb765ace700 1 --2- 172.21.15.98:0/1692427902 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb760105af0 0x7fb760105f00 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:23:03.231 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:57.064+0000 7fb765ace700 1 -- 172.21.15.98:0/1692427902 >> 172.21.15.98:0/1692427902 conn(0x7fb7600ff330 msgr2=0x7fb760101740 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:23:03.232 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:57.065+0000 7fb765ace700 1 -- 172.21.15.98:0/1692427902 shutdown_connections 2023-07-14T19:23:03.232 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:57.065+0000 7fb765ace700 1 -- 172.21.15.98:0/1692427902 wait complete. 2023-07-14T19:23:03.232 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:57.065+0000 7fb765ace700 1 Processor -- start 2023-07-14T19:23:03.232 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:57.066+0000 7fb765ace700 1 -- start start 2023-07-14T19:23:03.232 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:57.066+0000 7fb765ace700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb760105af0 0x7fb760197260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:23:03.232 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:57.066+0000 7fb765ace700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7601977a0 con 0x7fb760105af0 2023-07-14T19:23:03.232 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:57.066+0000 7fb75f7fe700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb760105af0 0x7fb760197260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:23:03.233 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:57.066+0000 7fb75f7fe700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb760105af0 0x7fb760197260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:44570/0 (socket says 172.21.15.98:44570) 2023-07-14T19:23:03.233 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:57.066+0000 7fb75f7fe700 1 -- 172.21.15.98:0/578772079 learned_addr learned my addr 172.21.15.98:0/578772079 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-07-14T19:23:03.233 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:57.066+0000 7fb75f7fe700 1 -- 172.21.15.98:0/578772079 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb75000b7f0 con 0x7fb760105af0 2023-07-14T19:23:03.233 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:57.066+0000 7fb75f7fe700 1 --2- 172.21.15.98:0/578772079 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb760105af0 0x7fb760197260 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7fb750006fa0 tx=0x7fb750003fb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:23:03.233 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:57.067+0000 7fb75cff9700 1 -- 172.21.15.98:0/578772079 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fb750010630 con 0x7fb760105af0 2023-07-14T19:23:03.233 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:57.067+0000 7fb75cff9700 1 -- 172.21.15.98:0/578772079 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fb750010c50 con 0x7fb760105af0 2023-07-14T19:23:03.233 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:57.067+0000 7fb75cff9700 1 -- 172.21.15.98:0/578772079 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fb750019ae0 con 0x7fb760105af0 2023-07-14T19:23:03.234 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:57.067+0000 7fb765ace700 1 -- 172.21.15.98:0/578772079 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb7601979a0 con 0x7fb760105af0 2023-07-14T19:23:03.234 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:57.067+0000 7fb765ace700 1 -- 172.21.15.98:0/578772079 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb760197d60 con 0x7fb760105af0 2023-07-14T19:23:03.234 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:57.069+0000 7fb75cff9700 1 -- 172.21.15.98:0/578772079 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 9) v1 ==== 45496+0+0 (secure 0 0 0) 0x7fb750019c40 con 0x7fb760105af0 2023-07-14T19:23:03.234 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:57.069+0000 7fb75cff9700 1 --2- 172.21.15.98:0/578772079 >> [v2:172.21.15.98:6800/1732769087,v1:172.21.15.98:6801/1732769087] conn(0x7fb748038800 0x7fb74803acb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:23:03.234 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:57.069+0000 7fb75cff9700 1 -- 172.21.15.98:0/578772079 --> [v2:172.21.15.98:6800/1732769087,v1:172.21.15.98:6801/1732769087] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x7fb74803b3c0 con 0x7fb748038800 2023-07-14T19:23:03.234 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:57.069+0000 7fb75cff9700 1 -- 172.21.15.98:0/578772079 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7fb75004cac0 con 0x7fb760105af0 2023-07-14T19:23:03.234 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:57.069+0000 7fb75effd700 1 -- 172.21.15.98:0/578772079 >> [v2:172.21.15.98:6800/1732769087,v1:172.21.15.98:6801/1732769087] conn(0x7fb748038800 msgr2=0x7fb74803acb0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.98:6800/1732769087 2023-07-14T19:23:03.235 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:57.069+0000 7fb75effd700 1 --2- 172.21.15.98:0/578772079 >> [v2:172.21.15.98:6800/1732769087,v1:172.21.15.98:6801/1732769087] conn(0x7fb748038800 0x7fb74803acb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-07-14T19:23:03.235 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:57.270+0000 7fb75effd700 1 -- 172.21.15.98:0/578772079 >> [v2:172.21.15.98:6800/1732769087,v1:172.21.15.98:6801/1732769087] conn(0x7fb748038800 msgr2=0x7fb74803acb0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.98:6800/1732769087 2023-07-14T19:23:03.235 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:57.270+0000 7fb75effd700 1 --2- 172.21.15.98:0/578772079 >> [v2:172.21.15.98:6800/1732769087,v1:172.21.15.98:6801/1732769087] conn(0x7fb748038800 0x7fb74803acb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-07-14T19:23:03.235 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:57.671+0000 7fb75effd700 1 -- 172.21.15.98:0/578772079 >> [v2:172.21.15.98:6800/1732769087,v1:172.21.15.98:6801/1732769087] conn(0x7fb748038800 msgr2=0x7fb74803acb0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.98:6800/1732769087 2023-07-14T19:23:03.235 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:57.671+0000 7fb75effd700 1 --2- 172.21.15.98:0/578772079 >> [v2:172.21.15.98:6800/1732769087,v1:172.21.15.98:6801/1732769087] conn(0x7fb748038800 0x7fb74803acb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-07-14T19:23:03.235 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:58.472+0000 7fb75effd700 1 -- 172.21.15.98:0/578772079 >> [v2:172.21.15.98:6800/1732769087,v1:172.21.15.98:6801/1732769087] conn(0x7fb748038800 msgr2=0x7fb74803acb0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.98:6800/1732769087 2023-07-14T19:23:03.235 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:22:58.472+0000 7fb75effd700 1 --2- 172.21.15.98:0/578772079 >> [v2:172.21.15.98:6800/1732769087,v1:172.21.15.98:6801/1732769087] conn(0x7fb748038800 0x7fb74803acb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-07-14T19:23:03.236 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:23:00.073+0000 7fb75effd700 1 -- 172.21.15.98:0/578772079 >> [v2:172.21.15.98:6800/1732769087,v1:172.21.15.98:6801/1732769087] conn(0x7fb748038800 msgr2=0x7fb74803acb0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.98:6800/1732769087 2023-07-14T19:23:03.236 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:23:00.073+0000 7fb75effd700 1 --2- 172.21.15.98:0/578772079 >> [v2:172.21.15.98:6800/1732769087,v1:172.21.15.98:6801/1732769087] conn(0x7fb748038800 0x7fb74803acb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 3.200000 2023-07-14T19:23:03.236 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:23:01.521+0000 7fb75cff9700 1 -- 172.21.15.98:0/578772079 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mgrmap(e 10) v1 ==== 45263+0+0 (secure 0 0 0) 0x7fb75002a450 con 0x7fb760105af0 2023-07-14T19:23:03.236 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:23:01.521+0000 7fb75cff9700 1 -- 172.21.15.98:0/578772079 >> [v2:172.21.15.98:6800/1732769087,v1:172.21.15.98:6801/1732769087] conn(0x7fb748038800 msgr2=0x7fb74803acb0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-07-14T19:23:03.236 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:23:01.521+0000 7fb75cff9700 1 --2- 172.21.15.98:0/578772079 >> [v2:172.21.15.98:6800/1732769087,v1:172.21.15.98:6801/1732769087] conn(0x7fb748038800 0x7fb74803acb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:23:03.236 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:23:02.525+0000 7fb75cff9700 1 -- 172.21.15.98:0/578772079 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mgrmap(e 11) v1 ==== 45390+0+0 (secure 0 0 0) 0x7fb750029d00 con 0x7fb760105af0 2023-07-14T19:23:03.236 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:23:02.525+0000 7fb75cff9700 1 --2- 172.21.15.98:0/578772079 >> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] conn(0x7fb74803bff0 0x7fb74803e3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:23:03.236 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:23:02.525+0000 7fb75cff9700 1 -- 172.21.15.98:0/578772079 --> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x7fb74803b3c0 con 0x7fb74803bff0 2023-07-14T19:23:03.237 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:23:02.525+0000 7fb75effd700 1 --2- 172.21.15.98:0/578772079 >> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] conn(0x7fb74803bff0 0x7fb74803e3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:23:03.237 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:23:02.525+0000 7fb75effd700 1 --2- 172.21.15.98:0/578772079 >> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] conn(0x7fb74803bff0 0x7fb74803e3d0 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7fb754003a10 tx=0x7fb7540092b0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:23:03.237 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:23:02.526+0000 7fb75cff9700 1 -- 172.21.15.98:0/578772079 <== mgr.14152 v2:172.21.15.98:6800/452042164 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+6910 (secure 0 0 0) 0x7fb74803b3c0 con 0x7fb74803bff0 2023-07-14T19:23:03.237 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:23:02.533+0000 7fb765ace700 1 -- 172.21.15.98:0/578772079 --> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] -- command(tid 1: {"prefix": "mgr_status"}) v1 -- 0x7fb7601a95d0 con 0x7fb74803bff0 2023-07-14T19:23:03.237 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:23:02.533+0000 7fb75cff9700 1 -- 172.21.15.98:0/578772079 <== mgr.14152 v2:172.21.15.98:6800/452042164 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+52 (secure 0 0 0) 0x7fb7601a95d0 con 0x7fb74803bff0 2023-07-14T19:23:03.237 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:23:02.533+0000 7fb765ace700 1 -- 172.21.15.98:0/578772079 >> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] conn(0x7fb74803bff0 msgr2=0x7fb74803e3d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:23:03.237 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:23:02.533+0000 7fb765ace700 1 --2- 172.21.15.98:0/578772079 >> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] conn(0x7fb74803bff0 0x7fb74803e3d0 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7fb754003a10 tx=0x7fb7540092b0 comp rx=0 tx=0).stop 2023-07-14T19:23:03.238 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:23:02.533+0000 7fb765ace700 1 -- 172.21.15.98:0/578772079 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb760105af0 msgr2=0x7fb760197260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:23:03.238 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:23:02.533+0000 7fb765ace700 1 --2- 172.21.15.98:0/578772079 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb760105af0 0x7fb760197260 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7fb750006fa0 tx=0x7fb750003fb0 comp rx=0 tx=0).stop 2023-07-14T19:23:03.238 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:23:02.533+0000 7fb765ace700 1 -- 172.21.15.98:0/578772079 shutdown_connections 2023-07-14T19:23:03.238 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:23:02.533+0000 7fb765ace700 1 --2- 172.21.15.98:0/578772079 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb760105af0 0x7fb760197260 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:23:03.238 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:23:02.533+0000 7fb765ace700 1 --2- 172.21.15.98:0/578772079 >> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] conn(0x7fb74803bff0 0x7fb74803e3d0 unknown :-1 s=CLOSED pgs=1 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:23:03.238 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:23:02.533+0000 7fb765ace700 1 --2- 172.21.15.98:0/578772079 >> [v2:172.21.15.98:6800/1732769087,v1:172.21.15.98:6801/1732769087] conn(0x7fb748038800 0x7fb74803acb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:23:03.238 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:23:02.533+0000 7fb765ace700 1 -- 172.21.15.98:0/578772079 >> 172.21.15.98:0/578772079 conn(0x7fb7600ff330 msgr2=0x7fb76018dfd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:23:03.239 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:23:02.533+0000 7fb765ace700 1 -- 172.21.15.98:0/578772079 shutdown_connections 2023-07-14T19:23:03.239 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:23:02.533+0000 7fb765ace700 1 -- 172.21.15.98:0/578772079 wait complete. 2023-07-14T19:23:03.239 INFO:teuthology.orchestra.run.smithi098.stdout:mgr epoch 9 is available 2023-07-14T19:23:03.239 INFO:teuthology.orchestra.run.smithi098.stdout:Generating a dashboard self-signed certificate... 2023-07-14T19:23:03.781 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:23:03 smithi098 ceph-mon[127486]: [14/Jul/2023:19:23:02] ENGINE Serving on http://172.21.15.98:8765 2023-07-14T19:23:03.782 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:23:03 smithi098 ceph-mon[127486]: [14/Jul/2023:19:23:02] ENGINE Serving on https://172.21.15.98:7150 2023-07-14T19:23:03.782 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:23:03 smithi098 ceph-mon[127486]: [14/Jul/2023:19:23:02] ENGINE Bus STARTED 2023-07-14T19:23:03.782 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:23:03 smithi098 ceph-mon[127486]: from='client.14156 -' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2023-07-14T19:23:03.782 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:23:03 smithi098 ceph-mon[127486]: mgrmap e11: x(active, since 1.00796s) 2023-07-14T19:23:03.782 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:23:03 smithi098 ceph-mon[127486]: from='client.14156 -' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2023-07-14T19:23:03.782 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:23:03 smithi098 ceph-mon[127486]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' 2023-07-14T19:23:04.374 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout Self-signed certificate created 2023-07-14T19:23:04.375 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:23:03.673+0000 7fa272610700 1 Processor -- start 2023-07-14T19:23:04.375 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:23:03.673+0000 7fa272610700 1 -- start start 2023-07-14T19:23:04.375 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:23:03.673+0000 7fa272610700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa26c105660 0x7fa26c105a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:23:04.375 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:23:03.673+0000 7fa272610700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa26c105fb0 con 0x7fa26c105660 2023-07-14T19:23:04.375 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:23:03.674+0000 7fa26bfff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa26c105660 0x7fa26c105a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:23:04.375 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:23:03.674+0000 7fa26bfff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa26c105660 0x7fa26c105a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:44624/0 (socket says 172.21.15.98:44624) 2023-07-14T19:23:04.376 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:23:03.674+0000 7fa26bfff700 1 -- 172.21.15.98:0/3190116096 learned_addr learned my addr 172.21.15.98:0/3190116096 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-07-14T19:23:04.376 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:23:03.674+0000 7fa26bfff700 1 -- 172.21.15.98:0/3190116096 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa26c1060f0 con 0x7fa26c105660 2023-07-14T19:23:04.376 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:23:03.674+0000 7fa26bfff700 1 --2- 172.21.15.98:0/3190116096 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa26c105660 0x7fa26c105a70 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7fa25c0098a0 tx=0x7fa25c009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=784a92cfeeb0a3c2 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:23:04.376 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:23:03.674+0000 7fa26affd700 1 -- 172.21.15.98:0/3190116096 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fa25c004160 con 0x7fa26c105660 2023-07-14T19:23:04.376 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:23:03.674+0000 7fa26affd700 1 -- 172.21.15.98:0/3190116096 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fa25c0042c0 con 0x7fa26c105660 2023-07-14T19:23:04.376 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:23:03.674+0000 7fa26affd700 1 -- 172.21.15.98:0/3190116096 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fa25c0119b0 con 0x7fa26c105660 2023-07-14T19:23:04.377 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:23:03.675+0000 7fa272610700 1 -- 172.21.15.98:0/3190116096 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa26c105660 msgr2=0x7fa26c105a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:23:04.377 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:23:03.675+0000 7fa272610700 1 --2- 172.21.15.98:0/3190116096 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa26c105660 0x7fa26c105a70 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7fa25c0098a0 tx=0x7fa25c009bb0 comp rx=0 tx=0).stop 2023-07-14T19:23:04.377 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:23:03.675+0000 7fa272610700 1 -- 172.21.15.98:0/3190116096 shutdown_connections 2023-07-14T19:23:04.377 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:23:03.675+0000 7fa272610700 1 --2- 172.21.15.98:0/3190116096 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa26c105660 0x7fa26c105a70 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:23:04.377 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:23:03.675+0000 7fa272610700 1 -- 172.21.15.98:0/3190116096 >> 172.21.15.98:0/3190116096 conn(0x7fa26c100cb0 msgr2=0x7fa26c1030e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:23:04.377 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:23:03.675+0000 7fa272610700 1 -- 172.21.15.98:0/3190116096 shutdown_connections 2023-07-14T19:23:04.378 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:23:03.675+0000 7fa272610700 1 -- 172.21.15.98:0/3190116096 wait complete. 2023-07-14T19:23:04.378 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:23:03.676+0000 7fa272610700 1 Processor -- start 2023-07-14T19:23:04.378 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:23:03.676+0000 7fa272610700 1 -- start start 2023-07-14T19:23:04.378 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:23:03.676+0000 7fa272610700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa26c105660 0x7fa26c1994a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:23:04.378 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:23:03.676+0000 7fa272610700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa26c1999e0 con 0x7fa26c105660 2023-07-14T19:23:04.378 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:23:03.676+0000 7fa26bfff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa26c105660 0x7fa26c1994a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:23:04.378 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:23:03.676+0000 7fa26bfff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa26c105660 0x7fa26c1994a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:44626/0 (socket says 172.21.15.98:44626) 2023-07-14T19:23:04.379 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:23:03.676+0000 7fa26bfff700 1 -- 172.21.15.98:0/3218016823 learned_addr learned my addr 172.21.15.98:0/3218016823 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-07-14T19:23:04.379 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:23:03.677+0000 7fa26bfff700 1 -- 172.21.15.98:0/3218016823 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa25c009550 con 0x7fa26c105660 2023-07-14T19:23:04.379 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:23:03.677+0000 7fa26bfff700 1 --2- 172.21.15.98:0/3218016823 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa26c105660 0x7fa26c1994a0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7fa25c003f00 tx=0x7fa25c011a30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:23:04.379 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:23:03.677+0000 7fa2697fa700 1 -- 172.21.15.98:0/3218016823 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fa25c003b20 con 0x7fa26c105660 2023-07-14T19:23:04.379 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:23:03.677+0000 7fa272610700 1 -- 172.21.15.98:0/3218016823 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa26c199be0 con 0x7fa26c105660 2023-07-14T19:23:04.379 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:23:03.677+0000 7fa2697fa700 1 -- 172.21.15.98:0/3218016823 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fa25c003c80 con 0x7fa26c105660 2023-07-14T19:23:04.379 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:23:03.677+0000 7fa2697fa700 1 -- 172.21.15.98:0/3218016823 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fa25c019770 con 0x7fa26c105660 2023-07-14T19:23:04.379 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:23:03.677+0000 7fa272610700 1 -- 172.21.15.98:0/3218016823 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa26c19a000 con 0x7fa26c105660 2023-07-14T19:23:04.380 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:23:03.678+0000 7fa2697fa700 1 -- 172.21.15.98:0/3218016823 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 11) v1 ==== 45390+0+0 (secure 0 0 0) 0x7fa25c019a30 con 0x7fa26c105660 2023-07-14T19:23:04.380 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:23:03.678+0000 7fa272610700 1 -- 172.21.15.98:0/3218016823 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa26c193450 con 0x7fa26c105660 2023-07-14T19:23:04.380 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:23:03.678+0000 7fa2697fa700 1 --2- 172.21.15.98:0/3218016823 >> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] conn(0x7fa254038630 0x7fa25403aae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:23:04.380 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:23:03.678+0000 7fa2697fa700 1 -- 172.21.15.98:0/3218016823 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7fa25c04c900 con 0x7fa26c105660 2023-07-14T19:23:04.380 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:23:03.678+0000 7fa26b7fe700 1 --2- 172.21.15.98:0/3218016823 >> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] conn(0x7fa254038630 0x7fa25403aae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:23:04.380 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:23:03.679+0000 7fa26b7fe700 1 --2- 172.21.15.98:0/3218016823 >> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] conn(0x7fa254038630 0x7fa25403aae0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7fa260006fd0 tx=0x7fa260006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:23:04.380 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:23:03.683+0000 7fa2697fa700 1 -- 172.21.15.98:0/3218016823 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7fa25c00b9a0 con 0x7fa26c105660 2023-07-14T19:23:04.381 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:23:03.883+0000 7fa272610700 1 -- 172.21.15.98:0/3218016823 --> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] -- mgr_command(tid 0: {"prefix": "dashboard create-self-signed-cert", "target": ["mon-mgr", ""]}) v1 -- 0x7fa26c060a80 con 0x7fa254038630 2023-07-14T19:23:04.381 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:23:04.011+0000 7fa2697fa700 1 -- 172.21.15.98:0/3218016823 <== mgr.14152 v2:172.21.15.98:6800/452042164 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+32 (secure 0 0 0) 0x7fa26c060a80 con 0x7fa254038630 2023-07-14T19:23:04.381 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:23:04.013+0000 7fa272610700 1 -- 172.21.15.98:0/3218016823 >> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] conn(0x7fa254038630 msgr2=0x7fa25403aae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:23:04.381 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:23:04.013+0000 7fa272610700 1 --2- 172.21.15.98:0/3218016823 >> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] conn(0x7fa254038630 0x7fa25403aae0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7fa260006fd0 tx=0x7fa260006e40 comp rx=0 tx=0).stop 2023-07-14T19:23:04.381 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:23:04.013+0000 7fa272610700 1 -- 172.21.15.98:0/3218016823 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa26c105660 msgr2=0x7fa26c1994a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:23:04.381 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:23:04.013+0000 7fa272610700 1 --2- 172.21.15.98:0/3218016823 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa26c105660 0x7fa26c1994a0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7fa25c003f00 tx=0x7fa25c011a30 comp rx=0 tx=0).stop 2023-07-14T19:23:04.381 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:23:04.013+0000 7fa272610700 1 -- 172.21.15.98:0/3218016823 shutdown_connections 2023-07-14T19:23:04.382 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:23:04.013+0000 7fa272610700 1 --2- 172.21.15.98:0/3218016823 >> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] conn(0x7fa254038630 0x7fa25403aae0 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:23:04.382 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:23:04.013+0000 7fa272610700 1 --2- 172.21.15.98:0/3218016823 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa26c105660 0x7fa26c1994a0 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:23:04.382 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:23:04.013+0000 7fa272610700 1 -- 172.21.15.98:0/3218016823 >> 172.21.15.98:0/3218016823 conn(0x7fa26c100cb0 msgr2=0x7fa26c102410 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:23:04.382 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:23:04.014+0000 7fa272610700 1 -- 172.21.15.98:0/3218016823 shutdown_connections 2023-07-14T19:23:04.382 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:23:04.014+0000 7fa272610700 1 -- 172.21.15.98:0/3218016823 wait complete. 2023-07-14T19:23:04.382 INFO:teuthology.orchestra.run.smithi098.stdout:Creating initial admin user... 2023-07-14T19:23:04.781 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:23:04 smithi098 ceph-mon[127486]: from='client.14164 -' entity='client.admin' cmd=[{"prefix": "dashboard create-self-signed-cert", "target": ["mon-mgr", ""]}]: dispatch 2023-07-14T19:23:04.782 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:23:04 smithi098 ceph-mon[127486]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' 2023-07-14T19:23:04.782 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:23:04 smithi098 ceph-mon[127486]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' 2023-07-14T19:23:04.782 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:23:04 smithi098 ceph-mon[127486]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' 2023-07-14T19:23:04.782 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:23:04 smithi098 ceph-mon[127486]: mgrmap e12: x(active, since 2s) 2023-07-14T19:23:05.614 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout {"username": "admin", "password": "$2b$12$svxIKVdxfNGR1Zuz9eX4Cuipjg2WBRuSlfz/0iBMN7Nl10oRo/p3a", "roles": ["administrator"], "name": null, "email": null, "lastUpdate": 1689362585, "enabled": true, "pwdExpirationDate": null, "pwdUpdateRequired": true} 2023-07-14T19:23:05.614 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:23:04.857+0000 7fc1ca09b700 1 Processor -- start 2023-07-14T19:23:05.615 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:23:04.857+0000 7fc1ca09b700 1 -- start start 2023-07-14T19:23:05.615 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:23:04.857+0000 7fc1ca09b700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc1c4073e60 0x7fc1c4108210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:23:05.615 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:23:04.857+0000 7fc1ca09b700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc1c4108750 con 0x7fc1c4073e60 2023-07-14T19:23:05.615 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:23:04.858+0000 7fc1c37fe700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc1c4073e60 0x7fc1c4108210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:23:05.615 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:23:04.858+0000 7fc1c37fe700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc1c4073e60 0x7fc1c4108210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:44630/0 (socket says 172.21.15.98:44630) 2023-07-14T19:23:05.615 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:23:04.858+0000 7fc1c37fe700 1 -- 172.21.15.98:0/866833329 learned_addr learned my addr 172.21.15.98:0/866833329 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-07-14T19:23:05.615 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:23:04.858+0000 7fc1c37fe700 1 -- 172.21.15.98:0/866833329 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc1c4108890 con 0x7fc1c4073e60 2023-07-14T19:23:05.616 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:23:04.858+0000 7fc1c37fe700 1 --2- 172.21.15.98:0/866833329 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc1c4073e60 0x7fc1c4108210 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7fc1ac0098a0 tx=0x7fc1ac009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2b8aa9550abc2188 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:23:05.616 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:23:04.858+0000 7fc1c27fc700 1 -- 172.21.15.98:0/866833329 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fc1ac004160 con 0x7fc1c4073e60 2023-07-14T19:23:05.616 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:23:04.858+0000 7fc1c27fc700 1 -- 172.21.15.98:0/866833329 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fc1ac0042c0 con 0x7fc1c4073e60 2023-07-14T19:23:05.616 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:23:04.858+0000 7fc1c27fc700 1 -- 172.21.15.98:0/866833329 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fc1ac0119b0 con 0x7fc1c4073e60 2023-07-14T19:23:05.616 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:23:04.859+0000 7fc1ca09b700 1 -- 172.21.15.98:0/866833329 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc1c4073e60 msgr2=0x7fc1c4108210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:23:05.616 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:23:04.859+0000 7fc1ca09b700 1 --2- 172.21.15.98:0/866833329 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc1c4073e60 0x7fc1c4108210 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7fc1ac0098a0 tx=0x7fc1ac009bb0 comp rx=0 tx=0).stop 2023-07-14T19:23:05.616 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:23:04.859+0000 7fc1ca09b700 1 -- 172.21.15.98:0/866833329 shutdown_connections 2023-07-14T19:23:05.617 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:23:04.859+0000 7fc1ca09b700 1 --2- 172.21.15.98:0/866833329 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc1c4073e60 0x7fc1c4108210 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:23:05.617 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:23:04.859+0000 7fc1ca09b700 1 -- 172.21.15.98:0/866833329 >> 172.21.15.98:0/866833329 conn(0x7fc1c40ffbc0 msgr2=0x7fc1c4101ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:23:05.617 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:23:04.859+0000 7fc1ca09b700 1 -- 172.21.15.98:0/866833329 shutdown_connections 2023-07-14T19:23:05.617 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:23:04.859+0000 7fc1ca09b700 1 -- 172.21.15.98:0/866833329 wait complete. 2023-07-14T19:23:05.617 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:23:04.859+0000 7fc1ca09b700 1 Processor -- start 2023-07-14T19:23:05.617 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:23:04.860+0000 7fc1ca09b700 1 -- start start 2023-07-14T19:23:05.617 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:23:04.860+0000 7fc1ca09b700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc1c41994c0 0x7fc1c41998d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:23:05.617 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:23:04.860+0000 7fc1ca09b700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc1c4199e10 con 0x7fc1c41994c0 2023-07-14T19:23:05.618 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:23:04.860+0000 7fc1c37fe700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc1c41994c0 0x7fc1c41998d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:23:05.618 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:23:04.860+0000 7fc1c37fe700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc1c41994c0 0x7fc1c41998d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:44632/0 (socket says 172.21.15.98:44632) 2023-07-14T19:23:05.618 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:23:04.860+0000 7fc1c37fe700 1 -- 172.21.15.98:0/1445579430 learned_addr learned my addr 172.21.15.98:0/1445579430 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-07-14T19:23:05.618 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:23:04.860+0000 7fc1c37fe700 1 -- 172.21.15.98:0/1445579430 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc1ac009550 con 0x7fc1c41994c0 2023-07-14T19:23:05.618 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:23:04.860+0000 7fc1c37fe700 1 --2- 172.21.15.98:0/1445579430 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc1c41994c0 0x7fc1c41998d0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7fc1ac003e60 tx=0x7fc1ac011a30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:23:05.618 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:23:04.860+0000 7fc1c0ff9700 1 -- 172.21.15.98:0/1445579430 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fc1ac003a60 con 0x7fc1c41994c0 2023-07-14T19:23:05.618 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:23:04.860+0000 7fc1c0ff9700 1 -- 172.21.15.98:0/1445579430 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fc1ac003bc0 con 0x7fc1c41994c0 2023-07-14T19:23:05.619 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:23:04.860+0000 7fc1ca09b700 1 -- 172.21.15.98:0/1445579430 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc1c419a010 con 0x7fc1c41994c0 2023-07-14T19:23:05.619 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:23:04.860+0000 7fc1c0ff9700 1 -- 172.21.15.98:0/1445579430 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fc1ac0196c0 con 0x7fc1c41994c0 2023-07-14T19:23:05.619 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:23:04.861+0000 7fc1ca09b700 1 -- 172.21.15.98:0/1445579430 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc1c419cc70 con 0x7fc1c41994c0 2023-07-14T19:23:05.619 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:23:04.861+0000 7fc1c0ff9700 1 -- 172.21.15.98:0/1445579430 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 12) v1 ==== 45496+0+0 (secure 0 0 0) 0x7fc1ac019980 con 0x7fc1c41994c0 2023-07-14T19:23:05.619 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:23:04.861+0000 7fc1ca09b700 1 -- 172.21.15.98:0/1445579430 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc1c4193460 con 0x7fc1c41994c0 2023-07-14T19:23:05.619 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:23:04.862+0000 7fc1c0ff9700 1 --2- 172.21.15.98:0/1445579430 >> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] conn(0x7fc1b0038740 0x7fc1b003abf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:23:05.619 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:23:04.862+0000 7fc1c0ff9700 1 -- 172.21.15.98:0/1445579430 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7fc1ac04ca80 con 0x7fc1c41994c0 2023-07-14T19:23:05.619 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:23:04.862+0000 7fc1c2ffd700 1 --2- 172.21.15.98:0/1445579430 >> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] conn(0x7fc1b0038740 0x7fc1b003abf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:23:05.620 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:23:04.862+0000 7fc1c2ffd700 1 --2- 172.21.15.98:0/1445579430 >> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] conn(0x7fc1b0038740 0x7fc1b003abf0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7fc1b4006fd0 tx=0x7fc1b4006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:23:05.620 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:23:04.866+0000 7fc1c0ff9700 1 -- 172.21.15.98:0/1445579430 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7fc1ac023370 con 0x7fc1c41994c0 2023-07-14T19:23:05.620 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:23:05.075+0000 7fc1ca09b700 1 -- 172.21.15.98:0/1445579430 --> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] -- mgr_command(tid 0: {"prefix": "dashboard ac-user-create", "username": "admin", "rolename": "administrator", "force_password": true, "pwd_update_required": true, "target": ["mon-mgr", ""]}) v1 -- 0x7fc1c4073e60 con 0x7fc1b0038740 2023-07-14T19:23:05.620 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:23:05.301+0000 7fc1c0ff9700 1 -- 172.21.15.98:0/1445579430 <== mgr.14152 v2:172.21.15.98:6800/452042164 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+252 (secure 0 0 0) 0x7fc1c4073e60 con 0x7fc1b0038740 2023-07-14T19:23:05.620 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:23:05.304+0000 7fc1ca09b700 1 -- 172.21.15.98:0/1445579430 >> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] conn(0x7fc1b0038740 msgr2=0x7fc1b003abf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:23:05.620 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:23:05.304+0000 7fc1ca09b700 1 --2- 172.21.15.98:0/1445579430 >> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] conn(0x7fc1b0038740 0x7fc1b003abf0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7fc1b4006fd0 tx=0x7fc1b4006e40 comp rx=0 tx=0).stop 2023-07-14T19:23:05.620 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:23:05.304+0000 7fc1ca09b700 1 -- 172.21.15.98:0/1445579430 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc1c41994c0 msgr2=0x7fc1c41998d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:23:05.621 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:23:05.304+0000 7fc1ca09b700 1 --2- 172.21.15.98:0/1445579430 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc1c41994c0 0x7fc1c41998d0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7fc1ac003e60 tx=0x7fc1ac011a30 comp rx=0 tx=0).stop 2023-07-14T19:23:05.621 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:23:05.304+0000 7fc1ca09b700 1 -- 172.21.15.98:0/1445579430 shutdown_connections 2023-07-14T19:23:05.621 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:23:05.304+0000 7fc1ca09b700 1 --2- 172.21.15.98:0/1445579430 >> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] conn(0x7fc1b0038740 0x7fc1b003abf0 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:23:05.621 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:23:05.304+0000 7fc1ca09b700 1 --2- 172.21.15.98:0/1445579430 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc1c41994c0 0x7fc1c41998d0 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:23:05.621 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:23:05.304+0000 7fc1ca09b700 1 -- 172.21.15.98:0/1445579430 >> 172.21.15.98:0/1445579430 conn(0x7fc1c40ffbc0 msgr2=0x7fc1c4101fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:23:05.621 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:23:05.304+0000 7fc1ca09b700 1 -- 172.21.15.98:0/1445579430 shutdown_connections 2023-07-14T19:23:05.621 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:23:05.304+0000 7fc1ca09b700 1 -- 172.21.15.98:0/1445579430 wait complete. 2023-07-14T19:23:05.622 INFO:teuthology.orchestra.run.smithi098.stdout:Fetching dashboard port number... 2023-07-14T19:23:06.781 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:23:06 smithi098 ceph-mon[127486]: from='client.14166 -' entity='client.admin' cmd=[{"prefix": "dashboard ac-user-create", "username": "admin", "rolename": "administrator", "force_password": true, "pwd_update_required": true, "target": ["mon-mgr", ""]}]: dispatch 2023-07-14T19:23:06.782 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:23:06 smithi098 ceph-mon[127486]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' 2023-07-14T19:23:07.282 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout 8443 2023-07-14T19:23:07.282 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:23:06.615+0000 7f39ebd4c700 1 Processor -- start 2023-07-14T19:23:07.283 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:23:06.616+0000 7f39ebd4c700 1 -- start start 2023-07-14T19:23:07.283 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:23:06.616+0000 7f39ebd4c700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f39e41061e0 0x7f39e41085c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:23:07.283 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:23:06.616+0000 7f39ebd4c700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f39e4073e80 con 0x7f39e41061e0 2023-07-14T19:23:07.283 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:23:06.616+0000 7f39e9ae8700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f39e41061e0 0x7f39e41085c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:23:07.284 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:23:06.616+0000 7f39e9ae8700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f39e41061e0 0x7f39e41085c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:44636/0 (socket says 172.21.15.98:44636) 2023-07-14T19:23:07.284 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:23:06.616+0000 7f39e9ae8700 1 -- 172.21.15.98:0/298032702 learned_addr learned my addr 172.21.15.98:0/298032702 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-07-14T19:23:07.284 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:23:06.616+0000 7f39e9ae8700 1 -- 172.21.15.98:0/298032702 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f39e4073fc0 con 0x7f39e41061e0 2023-07-14T19:23:07.285 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:23:06.616+0000 7f39e9ae8700 1 --2- 172.21.15.98:0/298032702 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f39e41061e0 0x7f39e41085c0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f39e00098a0 tx=0x7f39e0009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a34b7981d301189f server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:23:07.285 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:23:06.616+0000 7f39e8ae6700 1 -- 172.21.15.98:0/298032702 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f39e0004160 con 0x7f39e41061e0 2023-07-14T19:23:07.285 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:23:06.617+0000 7f39e8ae6700 1 -- 172.21.15.98:0/298032702 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f39e00042c0 con 0x7f39e41061e0 2023-07-14T19:23:07.286 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:23:06.617+0000 7f39e8ae6700 1 -- 172.21.15.98:0/298032702 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f39e00119b0 con 0x7f39e41061e0 2023-07-14T19:23:07.286 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:23:06.617+0000 7f39ebd4c700 1 -- 172.21.15.98:0/298032702 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f39e41061e0 msgr2=0x7f39e41085c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:23:07.286 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:23:06.617+0000 7f39ebd4c700 1 --2- 172.21.15.98:0/298032702 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f39e41061e0 0x7f39e41085c0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f39e00098a0 tx=0x7f39e0009bb0 comp rx=0 tx=0).stop 2023-07-14T19:23:07.286 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:23:06.617+0000 7f39ebd4c700 1 -- 172.21.15.98:0/298032702 shutdown_connections 2023-07-14T19:23:07.287 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:23:06.617+0000 7f39ebd4c700 1 --2- 172.21.15.98:0/298032702 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f39e41061e0 0x7f39e41085c0 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:23:07.287 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:23:06.617+0000 7f39ebd4c700 1 -- 172.21.15.98:0/298032702 >> 172.21.15.98:0/298032702 conn(0x7f39e40ffbc0 msgr2=0x7f39e4101ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:23:07.287 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:23:06.617+0000 7f39ebd4c700 1 -- 172.21.15.98:0/298032702 shutdown_connections 2023-07-14T19:23:07.287 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:23:06.617+0000 7f39ebd4c700 1 -- 172.21.15.98:0/298032702 wait complete. 2023-07-14T19:23:07.287 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:23:06.618+0000 7f39ebd4c700 1 Processor -- start 2023-07-14T19:23:07.287 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:23:06.618+0000 7f39ebd4c700 1 -- start start 2023-07-14T19:23:07.287 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:23:06.618+0000 7f39ebd4c700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f39e41061e0 0x7f39e41994d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:23:07.288 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:23:06.618+0000 7f39ebd4c700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f39e4199a10 con 0x7f39e41061e0 2023-07-14T19:23:07.288 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:23:06.618+0000 7f39e9ae8700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f39e41061e0 0x7f39e41994d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:23:07.288 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:23:06.618+0000 7f39e9ae8700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f39e41061e0 0x7f39e41994d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:44638/0 (socket says 172.21.15.98:44638) 2023-07-14T19:23:07.288 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:23:06.618+0000 7f39e9ae8700 1 -- 172.21.15.98:0/2664794887 learned_addr learned my addr 172.21.15.98:0/2664794887 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-07-14T19:23:07.288 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:23:06.618+0000 7f39e9ae8700 1 -- 172.21.15.98:0/2664794887 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f39e0009550 con 0x7f39e41061e0 2023-07-14T19:23:07.288 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:23:06.619+0000 7f39e9ae8700 1 --2- 172.21.15.98:0/2664794887 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f39e41061e0 0x7f39e41994d0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f39e0006af0 tx=0x7f39e0011a10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:23:07.288 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:23:06.619+0000 7f39daffd700 1 -- 172.21.15.98:0/2664794887 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f39e0003a40 con 0x7f39e41061e0 2023-07-14T19:23:07.289 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:23:06.619+0000 7f39ebd4c700 1 -- 172.21.15.98:0/2664794887 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f39e4199c10 con 0x7f39e41061e0 2023-07-14T19:23:07.289 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:23:06.619+0000 7f39daffd700 1 -- 172.21.15.98:0/2664794887 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f39e0003ba0 con 0x7f39e41061e0 2023-07-14T19:23:07.289 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:23:06.619+0000 7f39daffd700 1 -- 172.21.15.98:0/2664794887 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f39e0019690 con 0x7f39e41061e0 2023-07-14T19:23:07.289 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:23:06.619+0000 7f39ebd4c700 1 -- 172.21.15.98:0/2664794887 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f39e419a030 con 0x7f39e41061e0 2023-07-14T19:23:07.290 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:23:06.620+0000 7f39daffd700 1 -- 172.21.15.98:0/2664794887 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 12) v1 ==== 45496+0+0 (secure 0 0 0) 0x7f39e0019950 con 0x7f39e41061e0 2023-07-14T19:23:07.290 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:23:06.620+0000 7f39daffd700 1 --2- 172.21.15.98:0/2664794887 >> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] conn(0x7f39d0038710 0x7f39d003abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:23:07.290 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:23:06.620+0000 7f39daffd700 1 -- 172.21.15.98:0/2664794887 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f39e004caf0 con 0x7f39e41061e0 2023-07-14T19:23:07.290 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:23:06.620+0000 7f39e92e7700 1 --2- 172.21.15.98:0/2664794887 >> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] conn(0x7f39d0038710 0x7f39d003abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:23:07.291 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:23:06.620+0000 7f39ebd4c700 1 -- 172.21.15.98:0/2664794887 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f39e4193450 con 0x7f39e41061e0 2023-07-14T19:23:07.291 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:23:06.620+0000 7f39e92e7700 1 --2- 172.21.15.98:0/2664794887 >> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] conn(0x7f39d0038710 0x7f39d003abc0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f39d4006fd0 tx=0x7f39d4006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:23:07.291 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:23:06.625+0000 7f39daffd700 1 -- 172.21.15.98:0/2664794887 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f39e00243f0 con 0x7f39e41061e0 2023-07-14T19:23:07.291 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:23:06.839+0000 7f39ebd4c700 1 -- 172.21.15.98:0/2664794887 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"} v 0) v1 -- 0x7f39e402cb90 con 0x7f39e41061e0 2023-07-14T19:23:07.291 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:23:06.839+0000 7f39daffd700 1 -- 172.21.15.98:0/2664794887 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"}]=0 v8) v1 ==== 112+0+5 (secure 0 0 0) 0x7f39e004a020 con 0x7f39e41061e0 2023-07-14T19:23:07.291 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:23:06.841+0000 7f39ebd4c700 1 -- 172.21.15.98:0/2664794887 >> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] conn(0x7f39d0038710 msgr2=0x7f39d003abc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:23:07.291 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:23:06.842+0000 7f39ebd4c700 1 --2- 172.21.15.98:0/2664794887 >> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] conn(0x7f39d0038710 0x7f39d003abc0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f39d4006fd0 tx=0x7f39d4006e40 comp rx=0 tx=0).stop 2023-07-14T19:23:07.292 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:23:06.842+0000 7f39ebd4c700 1 -- 172.21.15.98:0/2664794887 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f39e41061e0 msgr2=0x7f39e41994d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:23:07.292 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:23:06.842+0000 7f39ebd4c700 1 --2- 172.21.15.98:0/2664794887 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f39e41061e0 0x7f39e41994d0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f39e0006af0 tx=0x7f39e0011a10 comp rx=0 tx=0).stop 2023-07-14T19:23:07.292 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:23:06.842+0000 7f39ebd4c700 1 -- 172.21.15.98:0/2664794887 shutdown_connections 2023-07-14T19:23:07.292 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:23:06.842+0000 7f39ebd4c700 1 --2- 172.21.15.98:0/2664794887 >> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] conn(0x7f39d0038710 0x7f39d003abc0 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:23:07.292 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:23:06.842+0000 7f39ebd4c700 1 --2- 172.21.15.98:0/2664794887 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f39e41061e0 0x7f39e41994d0 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:23:07.292 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:23:06.842+0000 7f39ebd4c700 1 -- 172.21.15.98:0/2664794887 >> 172.21.15.98:0/2664794887 conn(0x7f39e40ffbc0 msgr2=0x7f39e4101320 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:23:07.293 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:23:06.842+0000 7f39ebd4c700 1 -- 172.21.15.98:0/2664794887 shutdown_connections 2023-07-14T19:23:07.293 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:23:06.842+0000 7f39ebd4c700 1 -- 172.21.15.98:0/2664794887 wait complete. 2023-07-14T19:23:07.293 INFO:teuthology.orchestra.run.smithi098.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2023-07-14T19:23:07.294 INFO:teuthology.orchestra.run.smithi098.stdout:systemctl: stdout disabled 2023-07-14T19:23:07.294 INFO:teuthology.orchestra.run.smithi098.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2023-07-14T19:23:07.294 INFO:teuthology.orchestra.run.smithi098.stdout:systemctl: stdout inactive 2023-07-14T19:23:07.294 INFO:teuthology.orchestra.run.smithi098.stdout:firewalld.service is not enabled 2023-07-14T19:23:07.294 INFO:teuthology.orchestra.run.smithi098.stdout:Not possible to open ports <[8443]>. firewalld.service is not available 2023-07-14T19:23:07.295 INFO:teuthology.orchestra.run.smithi098.stdout:Ceph Dashboard is now available at: 2023-07-14T19:23:07.295 INFO:teuthology.orchestra.run.smithi098.stdout: 2023-07-14T19:23:07.295 INFO:teuthology.orchestra.run.smithi098.stdout: URL: https://smithi098:8443/ 2023-07-14T19:23:07.295 INFO:teuthology.orchestra.run.smithi098.stdout: User: admin 2023-07-14T19:23:07.295 INFO:teuthology.orchestra.run.smithi098.stdout: Password: fum7937c4h 2023-07-14T19:23:07.296 INFO:teuthology.orchestra.run.smithi098.stdout: 2023-07-14T19:23:07.296 INFO:teuthology.orchestra.run.smithi098.stdout:Saving cluster configuration to /var/lib/ceph/9adea648-227b-11ee-9b33-001a4aab830c/config directory 2023-07-14T19:23:07.296 INFO:teuthology.orchestra.run.smithi098.stdout:Enabling autotune for osd_memory_target 2023-07-14T19:23:07.482 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:23:07 smithi098 ceph-mon[127486]: from='client.? 172.21.15.98:0/2664794887' entity='client.admin' cmd=[{"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"}]: dispatch 2023-07-14T19:23:08.239 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:23:07.744+0000 7fe3d7ee5700 1 Processor -- start 2023-07-14T19:23:08.240 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:23:07.745+0000 7fe3d7ee5700 1 -- start start 2023-07-14T19:23:08.240 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:23:07.745+0000 7fe3d7ee5700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe3d0104570 0x7fe3d0104980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:23:08.240 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:23:07.745+0000 7fe3d7ee5700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe3d0104ec0 con 0x7fe3d0104570 2023-07-14T19:23:08.240 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:23:07.745+0000 7fe3d5c81700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe3d0104570 0x7fe3d0104980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:23:08.240 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:23:07.745+0000 7fe3d5c81700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe3d0104570 0x7fe3d0104980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:44646/0 (socket says 172.21.15.98:44646) 2023-07-14T19:23:08.240 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:23:07.745+0000 7fe3d5c81700 1 -- 172.21.15.98:0/717124423 learned_addr learned my addr 172.21.15.98:0/717124423 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-07-14T19:23:08.240 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:23:07.745+0000 7fe3d5c81700 1 -- 172.21.15.98:0/717124423 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe3d0105000 con 0x7fe3d0104570 2023-07-14T19:23:08.241 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:23:07.745+0000 7fe3d5c81700 1 --2- 172.21.15.98:0/717124423 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe3d0104570 0x7fe3d0104980 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7fe3c00098a0 tx=0x7fe3c0009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f126c09189c1e80e server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:23:08.241 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:23:07.746+0000 7fe3d4c7f700 1 -- 172.21.15.98:0/717124423 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fe3c0004160 con 0x7fe3d0104570 2023-07-14T19:23:08.241 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:23:07.746+0000 7fe3d4c7f700 1 -- 172.21.15.98:0/717124423 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fe3c00042c0 con 0x7fe3d0104570 2023-07-14T19:23:08.241 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:23:07.746+0000 7fe3d4c7f700 1 -- 172.21.15.98:0/717124423 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fe3c00119b0 con 0x7fe3d0104570 2023-07-14T19:23:08.241 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:23:07.746+0000 7fe3d7ee5700 1 -- 172.21.15.98:0/717124423 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe3d0104570 msgr2=0x7fe3d0104980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:23:08.241 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:23:07.746+0000 7fe3d7ee5700 1 --2- 172.21.15.98:0/717124423 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe3d0104570 0x7fe3d0104980 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7fe3c00098a0 tx=0x7fe3c0009bb0 comp rx=0 tx=0).stop 2023-07-14T19:23:08.241 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:23:07.746+0000 7fe3d7ee5700 1 -- 172.21.15.98:0/717124423 shutdown_connections 2023-07-14T19:23:08.242 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:23:07.746+0000 7fe3d7ee5700 1 --2- 172.21.15.98:0/717124423 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe3d0104570 0x7fe3d0104980 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:23:08.242 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:23:07.746+0000 7fe3d7ee5700 1 -- 172.21.15.98:0/717124423 >> 172.21.15.98:0/717124423 conn(0x7fe3d00ffbc0 msgr2=0x7fe3d0101ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:23:08.242 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:23:07.746+0000 7fe3d7ee5700 1 -- 172.21.15.98:0/717124423 shutdown_connections 2023-07-14T19:23:08.242 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:23:07.746+0000 7fe3d7ee5700 1 -- 172.21.15.98:0/717124423 wait complete. 2023-07-14T19:23:08.242 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:23:07.747+0000 7fe3d7ee5700 1 Processor -- start 2023-07-14T19:23:08.242 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:23:07.747+0000 7fe3d7ee5700 1 -- start start 2023-07-14T19:23:08.242 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:23:07.747+0000 7fe3d7ee5700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe3d0104570 0x7fe3d0199470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:23:08.243 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:23:07.747+0000 7fe3d7ee5700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe3d01999b0 con 0x7fe3d0104570 2023-07-14T19:23:08.243 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:23:07.747+0000 7fe3d5c81700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe3d0104570 0x7fe3d0199470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:23:08.243 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:23:07.747+0000 7fe3d5c81700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe3d0104570 0x7fe3d0199470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:44648/0 (socket says 172.21.15.98:44648) 2023-07-14T19:23:08.243 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:23:07.747+0000 7fe3d5c81700 1 -- 172.21.15.98:0/2273475836 learned_addr learned my addr 172.21.15.98:0/2273475836 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-07-14T19:23:08.243 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:23:07.747+0000 7fe3d5c81700 1 -- 172.21.15.98:0/2273475836 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe3c0009550 con 0x7fe3d0104570 2023-07-14T19:23:08.243 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:23:07.748+0000 7fe3d5c81700 1 --2- 172.21.15.98:0/2273475836 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe3d0104570 0x7fe3d0199470 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7fe3c0003f00 tx=0x7fe3c0011a30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:23:08.243 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:23:07.748+0000 7fe3c6ffd700 1 -- 172.21.15.98:0/2273475836 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fe3c0003b20 con 0x7fe3d0104570 2023-07-14T19:23:08.244 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:23:07.748+0000 7fe3c6ffd700 1 -- 172.21.15.98:0/2273475836 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fe3c0003c80 con 0x7fe3d0104570 2023-07-14T19:23:08.244 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:23:07.748+0000 7fe3c6ffd700 1 -- 172.21.15.98:0/2273475836 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fe3c0019770 con 0x7fe3d0104570 2023-07-14T19:23:08.244 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:23:07.748+0000 7fe3d7ee5700 1 -- 172.21.15.98:0/2273475836 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe3d0199bb0 con 0x7fe3d0104570 2023-07-14T19:23:08.244 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:23:07.748+0000 7fe3d7ee5700 1 -- 172.21.15.98:0/2273475836 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe3d0199fd0 con 0x7fe3d0104570 2023-07-14T19:23:08.244 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:23:07.749+0000 7fe3d7ee5700 1 -- 172.21.15.98:0/2273475836 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe3d0193560 con 0x7fe3d0104570 2023-07-14T19:23:08.244 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:23:07.749+0000 7fe3c6ffd700 1 -- 172.21.15.98:0/2273475836 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 12) v1 ==== 45496+0+0 (secure 0 0 0) 0x7fe3c001ab60 con 0x7fe3d0104570 2023-07-14T19:23:08.244 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:23:07.749+0000 7fe3c6ffd700 1 --2- 172.21.15.98:0/2273475836 >> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] conn(0x7fe3bc0387b0 0x7fe3bc03ac60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:23:08.245 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:23:07.749+0000 7fe3c6ffd700 1 -- 172.21.15.98:0/2273475836 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7fe3c004cab0 con 0x7fe3d0104570 2023-07-14T19:23:08.245 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:23:07.749+0000 7fe3d5480700 1 --2- 172.21.15.98:0/2273475836 >> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] conn(0x7fe3bc0387b0 0x7fe3bc03ac60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:23:08.245 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:23:07.749+0000 7fe3d5480700 1 --2- 172.21.15.98:0/2273475836 >> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] conn(0x7fe3bc0387b0 0x7fe3bc03ac60 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7fe3cc006fd0 tx=0x7fe3cc006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:23:08.245 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:23:07.754+0000 7fe3c6ffd700 1 -- 172.21.15.98:0/2273475836 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7fe3c001a430 con 0x7fe3d0104570 2023-07-14T19:23:08.245 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:23:07.939+0000 7fe3d7ee5700 1 -- 172.21.15.98:0/2273475836 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command([{prefix=config set, name=osd_memory_target_autotune}] v 0) v1 -- 0x7fe3d0108a10 con 0x7fe3d0104570 2023-07-14T19:23:08.245 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:23:07.939+0000 7fe3c6ffd700 1 -- 172.21.15.98:0/2273475836 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{prefix=config set, name=osd_memory_target_autotune}]=0 v8)=0 v8) v1 ==== 127+0+0 (secure 0 0 0) 0x7fe3c002db40 con 0x7fe3d0104570 2023-07-14T19:23:08.245 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:23:07.941+0000 7fe3d7ee5700 1 -- 172.21.15.98:0/2273475836 >> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] conn(0x7fe3bc0387b0 msgr2=0x7fe3bc03ac60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:23:08.246 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:23:07.941+0000 7fe3d7ee5700 1 --2- 172.21.15.98:0/2273475836 >> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] conn(0x7fe3bc0387b0 0x7fe3bc03ac60 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7fe3cc006fd0 tx=0x7fe3cc006e40 comp rx=0 tx=0).stop 2023-07-14T19:23:08.246 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:23:07.941+0000 7fe3d7ee5700 1 -- 172.21.15.98:0/2273475836 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe3d0104570 msgr2=0x7fe3d0199470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:23:08.246 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:23:07.941+0000 7fe3d7ee5700 1 --2- 172.21.15.98:0/2273475836 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe3d0104570 0x7fe3d0199470 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7fe3c0003f00 tx=0x7fe3c0011a30 comp rx=0 tx=0).stop 2023-07-14T19:23:08.246 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:23:07.941+0000 7fe3d7ee5700 1 -- 172.21.15.98:0/2273475836 shutdown_connections 2023-07-14T19:23:08.246 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:23:07.941+0000 7fe3d7ee5700 1 --2- 172.21.15.98:0/2273475836 >> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] conn(0x7fe3bc0387b0 0x7fe3bc03ac60 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:23:08.246 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:23:07.941+0000 7fe3d7ee5700 1 --2- 172.21.15.98:0/2273475836 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe3d0104570 0x7fe3d0199470 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:23:08.246 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:23:07.941+0000 7fe3d7ee5700 1 -- 172.21.15.98:0/2273475836 >> 172.21.15.98:0/2273475836 conn(0x7fe3d00ffbc0 msgr2=0x7fe3d0101340 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:23:08.247 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:23:07.942+0000 7fe3d7ee5700 1 -- 172.21.15.98:0/2273475836 shutdown_connections 2023-07-14T19:23:08.247 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:23:07.942+0000 7fe3d7ee5700 1 -- 172.21.15.98:0/2273475836 wait complete. 2023-07-14T19:23:09.383 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:23:08.676+0000 7f6a064f6700 1 Processor -- start 2023-07-14T19:23:09.383 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:23:08.676+0000 7f6a064f6700 1 -- start start 2023-07-14T19:23:09.383 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:23:08.676+0000 7f6a064f6700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6a00106d20 0x7f6a00107130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:23:09.384 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:23:08.676+0000 7f6a064f6700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6a00107670 con 0x7f6a00106d20 2023-07-14T19:23:09.384 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:23:08.676+0000 7f69fffff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6a00106d20 0x7f6a00107130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:23:09.384 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:23:08.676+0000 7f69fffff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6a00106d20 0x7f6a00107130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:44652/0 (socket says 172.21.15.98:44652) 2023-07-14T19:23:09.384 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:23:08.676+0000 7f69fffff700 1 -- 172.21.15.98:0/1229370051 learned_addr learned my addr 172.21.15.98:0/1229370051 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-07-14T19:23:09.384 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:23:08.677+0000 7f69fffff700 1 -- 172.21.15.98:0/1229370051 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6a001077b0 con 0x7f6a00106d20 2023-07-14T19:23:09.384 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:23:08.677+0000 7f69fffff700 1 --2- 172.21.15.98:0/1229370051 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6a00106d20 0x7f6a00107130 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f69f00098a0 tx=0x7f69f0009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3144bdc1eaf9d5ea server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:23:09.385 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:23:08.677+0000 7f69feffd700 1 -- 172.21.15.98:0/1229370051 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f69f0004160 con 0x7f6a00106d20 2023-07-14T19:23:09.385 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:23:08.677+0000 7f69feffd700 1 -- 172.21.15.98:0/1229370051 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f69f00042c0 con 0x7f6a00106d20 2023-07-14T19:23:09.385 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:23:08.677+0000 7f69feffd700 1 -- 172.21.15.98:0/1229370051 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f69f00119b0 con 0x7f6a00106d20 2023-07-14T19:23:09.385 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:23:08.677+0000 7f6a064f6700 1 -- 172.21.15.98:0/1229370051 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6a00106d20 msgr2=0x7f6a00107130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:23:09.385 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:23:08.677+0000 7f6a064f6700 1 --2- 172.21.15.98:0/1229370051 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6a00106d20 0x7f6a00107130 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f69f00098a0 tx=0x7f69f0009bb0 comp rx=0 tx=0).stop 2023-07-14T19:23:09.386 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:23:08.677+0000 7f6a064f6700 1 -- 172.21.15.98:0/1229370051 shutdown_connections 2023-07-14T19:23:09.386 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:23:08.677+0000 7f6a064f6700 1 --2- 172.21.15.98:0/1229370051 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6a00106d20 0x7f6a00107130 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:23:09.386 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:23:08.677+0000 7f6a064f6700 1 -- 172.21.15.98:0/1229370051 >> 172.21.15.98:0/1229370051 conn(0x7f6a000ff330 msgr2=0x7f6a00101760 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:23:09.386 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:23:08.677+0000 7f6a064f6700 1 -- 172.21.15.98:0/1229370051 shutdown_connections 2023-07-14T19:23:09.386 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:23:08.678+0000 7f6a064f6700 1 -- 172.21.15.98:0/1229370051 wait complete. 2023-07-14T19:23:09.386 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:23:08.678+0000 7f6a064f6700 1 Processor -- start 2023-07-14T19:23:09.386 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:23:08.678+0000 7f6a064f6700 1 -- start start 2023-07-14T19:23:09.387 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:23:08.678+0000 7f6a064f6700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6a00106d20 0x7f6a00106420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:23:09.387 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:23:08.678+0000 7f6a064f6700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6a00106960 con 0x7f6a00106d20 2023-07-14T19:23:09.387 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:23:08.678+0000 7f69fffff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6a00106d20 0x7f6a00106420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:23:09.387 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:23:08.678+0000 7f69fffff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6a00106d20 0x7f6a00106420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:44654/0 (socket says 172.21.15.98:44654) 2023-07-14T19:23:09.387 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:23:08.678+0000 7f69fffff700 1 -- 172.21.15.98:0/2038278739 learned_addr learned my addr 172.21.15.98:0/2038278739 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-07-14T19:23:09.387 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:23:08.679+0000 7f69fffff700 1 -- 172.21.15.98:0/2038278739 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f69f0009550 con 0x7f6a00106d20 2023-07-14T19:23:09.387 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:23:08.679+0000 7f69fffff700 1 --2- 172.21.15.98:0/2038278739 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6a00106d20 0x7f6a00106420 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f69f0006af0 tx=0x7f69f0011a10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:23:09.387 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:23:08.679+0000 7f69fd7fa700 1 -- 172.21.15.98:0/2038278739 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f69f0003a40 con 0x7f6a00106d20 2023-07-14T19:23:09.388 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:23:08.679+0000 7f69fd7fa700 1 -- 172.21.15.98:0/2038278739 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f69f0003ba0 con 0x7f6a00106d20 2023-07-14T19:23:09.388 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:23:08.679+0000 7f6a064f6700 1 -- 172.21.15.98:0/2038278739 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6a00104a70 con 0x7f6a00106d20 2023-07-14T19:23:09.388 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:23:08.679+0000 7f69fd7fa700 1 -- 172.21.15.98:0/2038278739 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f69f00196c0 con 0x7f6a00106d20 2023-07-14T19:23:09.388 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:23:08.679+0000 7f6a064f6700 1 -- 172.21.15.98:0/2038278739 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6a00104e90 con 0x7f6a00106d20 2023-07-14T19:23:09.388 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:23:08.680+0000 7f69fd7fa700 1 -- 172.21.15.98:0/2038278739 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 13) v1 ==== 45539+0+0 (secure 0 0 0) 0x7f69f0019980 con 0x7f6a00106d20 2023-07-14T19:23:09.388 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:23:08.680+0000 7f69fd7fa700 1 --2- 172.21.15.98:0/2038278739 >> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] conn(0x7f69e80387b0 0x7f69e803ac60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:23:09.388 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:23:08.680+0000 7f6a064f6700 1 -- 172.21.15.98:0/2038278739 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6a00104250 con 0x7f6a00106d20 2023-07-14T19:23:09.389 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:23:08.680+0000 7f69fd7fa700 1 -- 172.21.15.98:0/2038278739 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f69f004ce00 con 0x7f6a00106d20 2023-07-14T19:23:09.389 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:23:08.680+0000 7f69ff7fe700 1 --2- 172.21.15.98:0/2038278739 >> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] conn(0x7f69e80387b0 0x7f69e803ac60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:23:09.389 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:23:08.680+0000 7f69ff7fe700 1 --2- 172.21.15.98:0/2038278739 >> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] conn(0x7f69e80387b0 0x7f69e803ac60 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f69f4006fd0 tx=0x7f69f4006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:23:09.389 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:23:08.685+0000 7f69fd7fa700 1 -- 172.21.15.98:0/2038278739 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f69f00243f0 con 0x7f6a00106d20 2023-07-14T19:23:09.389 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:23:08.972+0000 7f6a064f6700 1 -- 172.21.15.98:0/2038278739 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command([{prefix=config-key set, key=mgr/dashboard/cluster/status}] v 0) v1 -- 0x7f6a001043e0 con 0x7f6a00106d20 2023-07-14T19:23:09.389 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:23:08.975+0000 7f69fd7fa700 1 -- 172.21.15.98:0/2038278739 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{prefix=config-key set, key=mgr/dashboard/cluster/status}]=0 set mgr/dashboard/cluster/status v28)=0 set mgr/dashboard/cluster/status v28) v1 ==== 153+0+0 (secure 0 0 0) 0x7f69f0018350 con 0x7f6a00106d20 2023-07-14T19:23:09.389 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:23:08.977+0000 7f6a064f6700 1 -- 172.21.15.98:0/2038278739 >> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] conn(0x7f69e80387b0 msgr2=0x7f69e803ac60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:23:09.389 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:23:08.977+0000 7f6a064f6700 1 --2- 172.21.15.98:0/2038278739 >> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] conn(0x7f69e80387b0 0x7f69e803ac60 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f69f4006fd0 tx=0x7f69f4006e40 comp rx=0 tx=0).stop 2023-07-14T19:23:09.390 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:23:08.977+0000 7f6a064f6700 1 -- 172.21.15.98:0/2038278739 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6a00106d20 msgr2=0x7f6a00106420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:23:09.390 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:23:08.977+0000 7f6a064f6700 1 --2- 172.21.15.98:0/2038278739 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6a00106d20 0x7f6a00106420 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f69f0006af0 tx=0x7f69f0011a10 comp rx=0 tx=0).stop 2023-07-14T19:23:09.390 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:23:08.977+0000 7f6a064f6700 1 -- 172.21.15.98:0/2038278739 shutdown_connections 2023-07-14T19:23:09.390 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:23:08.977+0000 7f6a064f6700 1 --2- 172.21.15.98:0/2038278739 >> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] conn(0x7f69e80387b0 0x7f69e803ac60 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:23:09.390 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:23:08.977+0000 7f6a064f6700 1 --2- 172.21.15.98:0/2038278739 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6a00106d20 0x7f6a00106420 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:23:09.390 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:23:08.977+0000 7f6a064f6700 1 -- 172.21.15.98:0/2038278739 >> 172.21.15.98:0/2038278739 conn(0x7f6a000ff330 msgr2=0x7f6a00100a30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:23:09.390 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:23:08.977+0000 7f6a064f6700 1 -- 172.21.15.98:0/2038278739 shutdown_connections 2023-07-14T19:23:09.391 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2023-07-14T19:23:08.977+0000 7f6a064f6700 1 -- 172.21.15.98:0/2038278739 wait complete. 2023-07-14T19:23:09.391 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr set mgr/dashboard/cluster/status 2023-07-14T19:23:09.391 INFO:teuthology.orchestra.run.smithi098.stdout:You can access the Ceph CLI as following in case of multi-cluster or non-default config: 2023-07-14T19:23:09.391 INFO:teuthology.orchestra.run.smithi098.stdout: 2023-07-14T19:23:09.391 INFO:teuthology.orchestra.run.smithi098.stdout: sudo /home/ubuntu/cephtest/cephadm shell --fsid 9adea648-227b-11ee-9b33-001a4aab830c -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring 2023-07-14T19:23:09.391 INFO:teuthology.orchestra.run.smithi098.stdout: 2023-07-14T19:23:09.391 INFO:teuthology.orchestra.run.smithi098.stdout:Or, if you are only running a single cluster on this host: 2023-07-14T19:23:09.391 INFO:teuthology.orchestra.run.smithi098.stdout: 2023-07-14T19:23:09.392 INFO:teuthology.orchestra.run.smithi098.stdout: sudo /home/ubuntu/cephtest/cephadm shell 2023-07-14T19:23:09.392 INFO:teuthology.orchestra.run.smithi098.stdout: 2023-07-14T19:23:09.392 INFO:teuthology.orchestra.run.smithi098.stdout:Please consider enabling telemetry to help improve Ceph: 2023-07-14T19:23:09.392 INFO:teuthology.orchestra.run.smithi098.stdout: 2023-07-14T19:23:09.392 INFO:teuthology.orchestra.run.smithi098.stdout: ceph telemetry on 2023-07-14T19:23:09.392 INFO:teuthology.orchestra.run.smithi098.stdout: 2023-07-14T19:23:09.392 INFO:teuthology.orchestra.run.smithi098.stdout:For more information see: 2023-07-14T19:23:09.393 INFO:teuthology.orchestra.run.smithi098.stdout: 2023-07-14T19:23:09.393 INFO:teuthology.orchestra.run.smithi098.stdout: https://docs.ceph.com/en/latest/mgr/telemetry/ 2023-07-14T19:23:09.393 INFO:teuthology.orchestra.run.smithi098.stdout: 2023-07-14T19:23:09.393 INFO:teuthology.orchestra.run.smithi098.stdout:Bootstrap complete. 2023-07-14T19:23:09.417 INFO:tasks.cephadm:Fetching config... 2023-07-14T19:23:09.417 DEBUG:teuthology.orchestra.run.smithi098:> set -ex 2023-07-14T19:23:09.417 DEBUG:teuthology.orchestra.run.smithi098:> dd if=/etc/ceph/ceph.conf of=/dev/stdout 2023-07-14T19:23:09.434 INFO:tasks.cephadm:Fetching client.admin keyring... 2023-07-14T19:23:09.435 DEBUG:teuthology.orchestra.run.smithi098:> set -ex 2023-07-14T19:23:09.436 DEBUG:teuthology.orchestra.run.smithi098:> dd if=/etc/ceph/ceph.client.admin.keyring of=/dev/stdout 2023-07-14T19:23:09.493 INFO:tasks.cephadm:Fetching mon keyring... 2023-07-14T19:23:09.494 DEBUG:teuthology.orchestra.run.smithi098:> set -ex 2023-07-14T19:23:09.495 DEBUG:teuthology.orchestra.run.smithi098:> sudo dd if=/var/lib/ceph/9adea648-227b-11ee-9b33-001a4aab830c/mon.a/keyring of=/dev/stdout 2023-07-14T19:23:09.558 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:23:09 smithi098 ceph-mon[127486]: mgrmap e13: x(active, since 6s) 2023-07-14T19:23:09.558 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:23:09 smithi098 ceph-mon[127486]: from='client.? 172.21.15.98:0/2038278739' entity='client.admin' 2023-07-14T19:23:09.561 INFO:tasks.cephadm:Fetching pub ssh key... 2023-07-14T19:23:09.562 DEBUG:teuthology.orchestra.run.smithi098:> set -ex 2023-07-14T19:23:09.562 DEBUG:teuthology.orchestra.run.smithi098:> dd if=/home/ubuntu/cephtest/ceph.pub of=/dev/stdout 2023-07-14T19:23:09.619 INFO:tasks.cephadm:Installing pub ssh key for root users... 2023-07-14T19:23:09.620 DEBUG:teuthology.orchestra.run.smithi098:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDp+ClaF7LpEMFYWJOvDu+/3NsZN3Q2vTOLAtxVP/VXDLuI1Nkc+DgD93kbQKb+hjgnmg40GqrfWAPHhO1QF5DTbMKVdcRAocN3LcWnew6S0nCRsD7XBx9foCvlCUBnE7pPXeRZlbEwXdFFaHakHf39wsC/zYUK+1lC+VBEGDPAO7KiSPhS7raP/YlswnDG/be0GMpQEpSjJp/iwDaDa67DHNOtUbkGUBsjZZO+3q5nzadZRHZ8FCX6DQBNxQdWwCtmgCC/F0nv1DRozV9bNubFE9PV9jS7CJ0YrVhPGmhECaO+5xGOpxEUJ5nUVw5jF1/5s2rZwpeWye1Pwp9q9TwGOqevZpDjQICGaZHpa/bF08J6lV+hM7UceEDMeiX386vCmL1rCQDO0+vBpCBKAW+vLfrUd2fB++vy/YvjrRj+cix5y+HkHL1ciscATK9lyGl2/MxfQCxftu2d8wn7l3QKSMr7emoM5bMvMALMB/UMLXUzVtNem6TNliiAH2ZkA58= ceph-9adea648-227b-11ee-9b33-001a4aab830c' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2023-07-14T19:23:09.696 INFO:teuthology.orchestra.run.smithi098.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDp+ClaF7LpEMFYWJOvDu+/3NsZN3Q2vTOLAtxVP/VXDLuI1Nkc+DgD93kbQKb+hjgnmg40GqrfWAPHhO1QF5DTbMKVdcRAocN3LcWnew6S0nCRsD7XBx9foCvlCUBnE7pPXeRZlbEwXdFFaHakHf39wsC/zYUK+1lC+VBEGDPAO7KiSPhS7raP/YlswnDG/be0GMpQEpSjJp/iwDaDa67DHNOtUbkGUBsjZZO+3q5nzadZRHZ8FCX6DQBNxQdWwCtmgCC/F0nv1DRozV9bNubFE9PV9jS7CJ0YrVhPGmhECaO+5xGOpxEUJ5nUVw5jF1/5s2rZwpeWye1Pwp9q9TwGOqevZpDjQICGaZHpa/bF08J6lV+hM7UceEDMeiX386vCmL1rCQDO0+vBpCBKAW+vLfrUd2fB++vy/YvjrRj+cix5y+HkHL1ciscATK9lyGl2/MxfQCxftu2d8wn7l3QKSMr7emoM5bMvMALMB/UMLXUzVtNem6TNliiAH2ZkA58= ceph-9adea648-227b-11ee-9b33-001a4aab830c 2023-07-14T19:23:09.709 DEBUG:teuthology.orchestra.run.smithi101:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDp+ClaF7LpEMFYWJOvDu+/3NsZN3Q2vTOLAtxVP/VXDLuI1Nkc+DgD93kbQKb+hjgnmg40GqrfWAPHhO1QF5DTbMKVdcRAocN3LcWnew6S0nCRsD7XBx9foCvlCUBnE7pPXeRZlbEwXdFFaHakHf39wsC/zYUK+1lC+VBEGDPAO7KiSPhS7raP/YlswnDG/be0GMpQEpSjJp/iwDaDa67DHNOtUbkGUBsjZZO+3q5nzadZRHZ8FCX6DQBNxQdWwCtmgCC/F0nv1DRozV9bNubFE9PV9jS7CJ0YrVhPGmhECaO+5xGOpxEUJ5nUVw5jF1/5s2rZwpeWye1Pwp9q9TwGOqevZpDjQICGaZHpa/bF08J6lV+hM7UceEDMeiX386vCmL1rCQDO0+vBpCBKAW+vLfrUd2fB++vy/YvjrRj+cix5y+HkHL1ciscATK9lyGl2/MxfQCxftu2d8wn7l3QKSMr7emoM5bMvMALMB/UMLXUzVtNem6TNliiAH2ZkA58= ceph-9adea648-227b-11ee-9b33-001a4aab830c' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2023-07-14T19:23:09.755 INFO:teuthology.orchestra.run.smithi101.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDp+ClaF7LpEMFYWJOvDu+/3NsZN3Q2vTOLAtxVP/VXDLuI1Nkc+DgD93kbQKb+hjgnmg40GqrfWAPHhO1QF5DTbMKVdcRAocN3LcWnew6S0nCRsD7XBx9foCvlCUBnE7pPXeRZlbEwXdFFaHakHf39wsC/zYUK+1lC+VBEGDPAO7KiSPhS7raP/YlswnDG/be0GMpQEpSjJp/iwDaDa67DHNOtUbkGUBsjZZO+3q5nzadZRHZ8FCX6DQBNxQdWwCtmgCC/F0nv1DRozV9bNubFE9PV9jS7CJ0YrVhPGmhECaO+5xGOpxEUJ5nUVw5jF1/5s2rZwpeWye1Pwp9q9TwGOqevZpDjQICGaZHpa/bF08J6lV+hM7UceEDMeiX386vCmL1rCQDO0+vBpCBKAW+vLfrUd2fB++vy/YvjrRj+cix5y+HkHL1ciscATK9lyGl2/MxfQCxftu2d8wn7l3QKSMr7emoM5bMvMALMB/UMLXUzVtNem6TNliiAH2ZkA58= ceph-9adea648-227b-11ee-9b33-001a4aab830c 2023-07-14T19:23:09.773 DEBUG:teuthology.orchestra.run.smithi139:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDp+ClaF7LpEMFYWJOvDu+/3NsZN3Q2vTOLAtxVP/VXDLuI1Nkc+DgD93kbQKb+hjgnmg40GqrfWAPHhO1QF5DTbMKVdcRAocN3LcWnew6S0nCRsD7XBx9foCvlCUBnE7pPXeRZlbEwXdFFaHakHf39wsC/zYUK+1lC+VBEGDPAO7KiSPhS7raP/YlswnDG/be0GMpQEpSjJp/iwDaDa67DHNOtUbkGUBsjZZO+3q5nzadZRHZ8FCX6DQBNxQdWwCtmgCC/F0nv1DRozV9bNubFE9PV9jS7CJ0YrVhPGmhECaO+5xGOpxEUJ5nUVw5jF1/5s2rZwpeWye1Pwp9q9TwGOqevZpDjQICGaZHpa/bF08J6lV+hM7UceEDMeiX386vCmL1rCQDO0+vBpCBKAW+vLfrUd2fB++vy/YvjrRj+cix5y+HkHL1ciscATK9lyGl2/MxfQCxftu2d8wn7l3QKSMr7emoM5bMvMALMB/UMLXUzVtNem6TNliiAH2ZkA58= ceph-9adea648-227b-11ee-9b33-001a4aab830c' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2023-07-14T19:23:09.818 INFO:teuthology.orchestra.run.smithi139.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDp+ClaF7LpEMFYWJOvDu+/3NsZN3Q2vTOLAtxVP/VXDLuI1Nkc+DgD93kbQKb+hjgnmg40GqrfWAPHhO1QF5DTbMKVdcRAocN3LcWnew6S0nCRsD7XBx9foCvlCUBnE7pPXeRZlbEwXdFFaHakHf39wsC/zYUK+1lC+VBEGDPAO7KiSPhS7raP/YlswnDG/be0GMpQEpSjJp/iwDaDa67DHNOtUbkGUBsjZZO+3q5nzadZRHZ8FCX6DQBNxQdWwCtmgCC/F0nv1DRozV9bNubFE9PV9jS7CJ0YrVhPGmhECaO+5xGOpxEUJ5nUVw5jF1/5s2rZwpeWye1Pwp9q9TwGOqevZpDjQICGaZHpa/bF08J6lV+hM7UceEDMeiX386vCmL1rCQDO0+vBpCBKAW+vLfrUd2fB++vy/YvjrRj+cix5y+HkHL1ciscATK9lyGl2/MxfQCxftu2d8wn7l3QKSMr7emoM5bMvMALMB/UMLXUzVtNem6TNliiAH2ZkA58= ceph-9adea648-227b-11ee-9b33-001a4aab830c 2023-07-14T19:23:09.831 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81c9387d2e7576da065747ec9c16d1c7d69016ec shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 9adea648-227b-11ee-9b33-001a4aab830c -- ceph config set mgr mgr/cephadm/allow_ptrace true 2023-07-14T19:23:10.300 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/9adea648-227b-11ee-9b33-001a4aab830c/mon.a/config 2023-07-14T19:23:11.764 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:23:11.763+0000 7f960cc40700 1 -- 172.21.15.98:0/550362706 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f96080fe610 msgr2=0x7f96080fea20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:23:11.765 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:23:11.763+0000 7f960cc40700 1 --2- 172.21.15.98:0/550362706 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f96080fe610 0x7f96080fea20 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f95f8009910 tx=0x7f95f8009c20 comp rx=0 tx=0).stop 2023-07-14T19:23:11.765 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:23:11.763+0000 7f960cc40700 1 -- 172.21.15.98:0/550362706 shutdown_connections 2023-07-14T19:23:11.765 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:23:11.763+0000 7f960cc40700 1 --2- 172.21.15.98:0/550362706 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f96080fe610 0x7f96080fea20 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:23:11.765 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:23:11.763+0000 7f960cc40700 1 -- 172.21.15.98:0/550362706 >> 172.21.15.98:0/550362706 conn(0x7f96080f9ba0 msgr2=0x7f96080fbff0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:23:11.765 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:23:11.763+0000 7f960cc40700 1 -- 172.21.15.98:0/550362706 shutdown_connections 2023-07-14T19:23:11.765 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:23:11.763+0000 7f960cc40700 1 -- 172.21.15.98:0/550362706 wait complete. 2023-07-14T19:23:11.766 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:23:11.764+0000 7f960cc40700 1 Processor -- start 2023-07-14T19:23:11.766 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:23:11.764+0000 7f960cc40700 1 -- start start 2023-07-14T19:23:11.767 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:23:11.764+0000 7f960cc40700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f96080fe610 0x7f96080683c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:23:11.767 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:23:11.764+0000 7f960cc40700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f96081975b0 con 0x7f96080fe610 2023-07-14T19:23:11.767 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:23:11.765+0000 7f960659c700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f96080fe610 0x7f96080683c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:23:11.767 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:23:11.765+0000 7f960659c700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f96080fe610 0x7f96080683c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:44660/0 (socket says 172.21.15.98:44660) 2023-07-14T19:23:11.767 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:23:11.765+0000 7f960659c700 1 -- 172.21.15.98:0/1339839611 learned_addr learned my addr 172.21.15.98:0/1339839611 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-07-14T19:23:11.767 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:23:11.765+0000 7f960659c700 1 -- 172.21.15.98:0/1339839611 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f95f80095f0 con 0x7f96080fe610 2023-07-14T19:23:11.767 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:23:11.765+0000 7f960659c700 1 --2- 172.21.15.98:0/1339839611 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f96080fe610 0x7f96080683c0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f95f8006010 tx=0x7f95f80057c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:23:11.768 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:23:11.765+0000 7f95f77fe700 1 -- 172.21.15.98:0/1339839611 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f95f801d070 con 0x7f96080fe610 2023-07-14T19:23:11.768 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:23:11.765+0000 7f95f77fe700 1 -- 172.21.15.98:0/1339839611 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f95f80044e0 con 0x7f96080fe610 2023-07-14T19:23:11.768 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:23:11.765+0000 7f960cc40700 1 -- 172.21.15.98:0/1339839611 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f96081976f0 con 0x7f96080fe610 2023-07-14T19:23:11.768 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:23:11.765+0000 7f95f77fe700 1 -- 172.21.15.98:0/1339839611 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f95f80039b0 con 0x7f96080fe610 2023-07-14T19:23:11.768 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:23:11.765+0000 7f960cc40700 1 -- 172.21.15.98:0/1339839611 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9608197b30 con 0x7f96080fe610 2023-07-14T19:23:11.769 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:23:11.766+0000 7f95f77fe700 1 -- 172.21.15.98:0/1339839611 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 13) v1 ==== 45539+0+0 (secure 0 0 0) 0x7f95f8003c70 con 0x7f96080fe610 2023-07-14T19:23:11.769 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:23:11.766+0000 7f960cc40700 1 -- 172.21.15.98:0/1339839611 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9608061ce0 con 0x7f96080fe610 2023-07-14T19:23:11.769 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:23:11.766+0000 7f95f77fe700 1 --2- 172.21.15.98:0/1339839611 >> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] conn(0x7f95f00387b0 0x7f95f003ac60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:23:11.769 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:23:11.766+0000 7f95f77fe700 1 -- 172.21.15.98:0/1339839611 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f95f804c920 con 0x7f96080fe610 2023-07-14T19:23:11.769 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:23:11.766+0000 7f9605d9b700 1 --2- 172.21.15.98:0/1339839611 >> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] conn(0x7f95f00387b0 0x7f95f003ac60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:23:11.770 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:23:11.767+0000 7f9605d9b700 1 --2- 172.21.15.98:0/1339839611 >> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] conn(0x7f95f00387b0 0x7f95f003ac60 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f95fc006fd0 tx=0x7f95fc006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:23:11.774 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:23:11.773+0000 7f95f77fe700 1 -- 172.21.15.98:0/1339839611 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f9608061ce0 con 0x7f96080fe610 2023-07-14T19:23:11.989 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:23:11.988+0000 7f960cc40700 1 -- 172.21.15.98:0/1339839611 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command([{prefix=config set, name=mgr/cephadm/allow_ptrace}] v 0) v1 -- 0x7f960802cc30 con 0x7f96080fe610 2023-07-14T19:23:11.992 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:23:11.991+0000 7f95f77fe700 1 -- 172.21.15.98:0/1339839611 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/cephadm/allow_ptrace}]=0 v9)=0 v9) v1 ==== 125+0+0 (secure 0 0 0) 0x7f95f802a950 con 0x7f96080fe610 2023-07-14T19:23:11.994 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:23:11.994+0000 7f960cc40700 1 -- 172.21.15.98:0/1339839611 >> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] conn(0x7f95f00387b0 msgr2=0x7f95f003ac60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:23:11.995 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:23:11.994+0000 7f960cc40700 1 --2- 172.21.15.98:0/1339839611 >> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] conn(0x7f95f00387b0 0x7f95f003ac60 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f95fc006fd0 tx=0x7f95fc006e40 comp rx=0 tx=0).stop 2023-07-14T19:23:11.995 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:23:11.994+0000 7f960cc40700 1 -- 172.21.15.98:0/1339839611 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f96080fe610 msgr2=0x7f96080683c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:23:11.995 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:23:11.994+0000 7f960cc40700 1 --2- 172.21.15.98:0/1339839611 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f96080fe610 0x7f96080683c0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f95f8006010 tx=0x7f95f80057c0 comp rx=0 tx=0).stop 2023-07-14T19:23:11.995 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:23:11.994+0000 7f960cc40700 1 -- 172.21.15.98:0/1339839611 shutdown_connections 2023-07-14T19:23:11.996 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:23:11.994+0000 7f960cc40700 1 --2- 172.21.15.98:0/1339839611 >> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] conn(0x7f95f00387b0 0x7f95f003ac60 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:23:11.996 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:23:11.994+0000 7f960cc40700 1 --2- 172.21.15.98:0/1339839611 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f96080fe610 0x7f96080683c0 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:23:11.996 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:23:11.994+0000 7f960cc40700 1 -- 172.21.15.98:0/1339839611 >> 172.21.15.98:0/1339839611 conn(0x7f96080f9ba0 msgr2=0x7f96080fa8f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:23:11.996 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:23:11.994+0000 7f960cc40700 1 -- 172.21.15.98:0/1339839611 shutdown_connections 2023-07-14T19:23:11.996 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:23:11.994+0000 7f960cc40700 1 -- 172.21.15.98:0/1339839611 wait complete. 2023-07-14T19:23:12.599 INFO:tasks.cephadm:Distributing conf and client.admin keyring to all hosts + 0755 2023-07-14T19:23:12.599 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81c9387d2e7576da065747ec9c16d1c7d69016ec shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 9adea648-227b-11ee-9b33-001a4aab830c -- ceph orch client-keyring set client.admin '*' --mode 0755 2023-07-14T19:23:13.147 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/9adea648-227b-11ee-9b33-001a4aab830c/mon.a/config 2023-07-14T19:23:13.281 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:23:12 smithi098 ceph-mon[127486]: from='client.? 172.21.15.98:0/1339839611' entity='client.admin' 2023-07-14T19:23:13.282 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:23:12 smithi098 ceph-mon[127486]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' 2023-07-14T19:23:13.282 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:23:12 smithi098 ceph-mon[127486]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' 2023-07-14T19:23:13.282 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:23:12 smithi098 ceph-mon[127486]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi098", "name": "osd_memory_target"}]: dispatch 2023-07-14T19:23:13.282 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:23:12 smithi098 ceph-mon[127486]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' 2023-07-14T19:23:13.282 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:23:12 smithi098 ceph-mon[127486]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-14T19:23:13.282 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:23:12 smithi098 ceph-mon[127486]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' 2023-07-14T19:23:14.386 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:23:14.382+0000 7f292e43a700 1 -- 172.21.15.98:0/641230412 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f29280fe610 msgr2=0x7f29280fea20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:23:14.386 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:23:14.382+0000 7f292e43a700 1 --2- 172.21.15.98:0/641230412 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f29280fe610 0x7f29280fea20 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f2918009910 tx=0x7f2918009c20 comp rx=0 tx=0).stop 2023-07-14T19:23:14.386 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:23:14.382+0000 7f292e43a700 1 -- 172.21.15.98:0/641230412 shutdown_connections 2023-07-14T19:23:14.386 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:23:14.382+0000 7f292e43a700 1 --2- 172.21.15.98:0/641230412 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f29280fe610 0x7f29280fea20 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:23:14.387 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:23:14.382+0000 7f292e43a700 1 -- 172.21.15.98:0/641230412 >> 172.21.15.98:0/641230412 conn(0x7f29280f9ba0 msgr2=0x7f29280fbff0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:23:14.387 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:23:14.382+0000 7f292e43a700 1 -- 172.21.15.98:0/641230412 shutdown_connections 2023-07-14T19:23:14.387 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:23:14.382+0000 7f292e43a700 1 -- 172.21.15.98:0/641230412 wait complete. 2023-07-14T19:23:14.387 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:23:14.382+0000 7f292e43a700 1 Processor -- start 2023-07-14T19:23:14.388 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:23:14.382+0000 7f292e43a700 1 -- start start 2023-07-14T19:23:14.388 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:23:14.383+0000 7f292e43a700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f29280fe610 0x7f292818b5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:23:14.388 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:23:14.383+0000 7f292e43a700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f292818bae0 con 0x7f29280fe610 2023-07-14T19:23:14.389 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:23:14.383+0000 7f2927fff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f29280fe610 0x7f292818b5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:23:14.389 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:23:14.383+0000 7f2927fff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f29280fe610 0x7f292818b5a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:44666/0 (socket says 172.21.15.98:44666) 2023-07-14T19:23:14.389 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:23:14.383+0000 7f2927fff700 1 -- 172.21.15.98:0/2705971137 learned_addr learned my addr 172.21.15.98:0/2705971137 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-07-14T19:23:14.389 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:23:14.383+0000 7f2927fff700 1 -- 172.21.15.98:0/2705971137 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f29180095f0 con 0x7f29280fe610 2023-07-14T19:23:14.390 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:23:14.383+0000 7f2927fff700 1 --2- 172.21.15.98:0/2705971137 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f29280fe610 0x7f292818b5a0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f2918006010 tx=0x7f29180057c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:23:14.390 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:23:14.383+0000 7f29257fa700 1 -- 172.21.15.98:0/2705971137 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f291801d070 con 0x7f29280fe610 2023-07-14T19:23:14.390 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:23:14.383+0000 7f292e43a700 1 -- 172.21.15.98:0/2705971137 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2928189bf0 con 0x7f29280fe610 2023-07-14T19:23:14.391 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:23:14.383+0000 7f29257fa700 1 -- 172.21.15.98:0/2705971137 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f29180044e0 con 0x7f29280fe610 2023-07-14T19:23:14.391 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:23:14.383+0000 7f29257fa700 1 -- 172.21.15.98:0/2705971137 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f29180039b0 con 0x7f29280fe610 2023-07-14T19:23:14.391 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:23:14.383+0000 7f292e43a700 1 -- 172.21.15.98:0/2705971137 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f292818a030 con 0x7f29280fe610 2023-07-14T19:23:14.392 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:23:14.384+0000 7f29257fa700 1 -- 172.21.15.98:0/2705971137 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 13) v1 ==== 45539+0+0 (secure 0 0 0) 0x7f2918003c70 con 0x7f29280fe610 2023-07-14T19:23:14.392 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:23:14.384+0000 7f292e43a700 1 -- 172.21.15.98:0/2705971137 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2928104350 con 0x7f29280fe610 2023-07-14T19:23:14.392 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:23:14.385+0000 7f29257fa700 1 --2- 172.21.15.98:0/2705971137 >> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] conn(0x7f29140387b0 0x7f291403ac60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:23:14.392 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:23:14.385+0000 7f29257fa700 1 -- 172.21.15.98:0/2705971137 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f291804cd90 con 0x7f29280fe610 2023-07-14T19:23:14.393 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:23:14.385+0000 7f29277fe700 1 --2- 172.21.15.98:0/2705971137 >> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] conn(0x7f29140387b0 0x7f291403ac60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:23:14.393 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:23:14.385+0000 7f29277fe700 1 --2- 172.21.15.98:0/2705971137 >> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] conn(0x7f29140387b0 0x7f291403ac60 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f2910006fd0 tx=0x7f2910006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:23:14.394 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:23:14.389+0000 7f29257fa700 1 -- 172.21.15.98:0/2705971137 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f291802a950 con 0x7f29280fe610 2023-07-14T19:23:14.577 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:23:14.576+0000 7f292e43a700 1 -- 172.21.15.98:0/2705971137 --> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] -- mgr_command(tid 0: {"prefix": "orch client-keyring set", "entity": "client.admin", "placement": "*", "mode": "0755", "target": ["mon-mgr", ""]}) v1 -- 0x7f29280fb550 con 0x7f29140387b0 2023-07-14T19:23:14.580 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:23:14.579+0000 7f29257fa700 1 -- 172.21.15.98:0/2705971137 <== mgr.14152 v2:172.21.15.98:6800/452042164 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f29280fb550 con 0x7f29140387b0 2023-07-14T19:23:14.582 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:23:14.581+0000 7f292e43a700 1 -- 172.21.15.98:0/2705971137 >> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] conn(0x7f29140387b0 msgr2=0x7f291403ac60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:23:14.582 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:23:14.581+0000 7f292e43a700 1 --2- 172.21.15.98:0/2705971137 >> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] conn(0x7f29140387b0 0x7f291403ac60 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f2910006fd0 tx=0x7f2910006e40 comp rx=0 tx=0).stop 2023-07-14T19:23:14.582 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:23:14.581+0000 7f292e43a700 1 -- 172.21.15.98:0/2705971137 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f29280fe610 msgr2=0x7f292818b5a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:23:14.583 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:23:14.581+0000 7f292e43a700 1 --2- 172.21.15.98:0/2705971137 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f29280fe610 0x7f292818b5a0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f2918006010 tx=0x7f29180057c0 comp rx=0 tx=0).stop 2023-07-14T19:23:14.583 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:23:14.581+0000 7f292e43a700 1 -- 172.21.15.98:0/2705971137 shutdown_connections 2023-07-14T19:23:14.583 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:23:14.581+0000 7f292e43a700 1 --2- 172.21.15.98:0/2705971137 >> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] conn(0x7f29140387b0 0x7f291403ac60 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:23:14.583 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:23:14.581+0000 7f292e43a700 1 --2- 172.21.15.98:0/2705971137 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f29280fe610 0x7f292818b5a0 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:23:14.583 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:23:14.581+0000 7f292e43a700 1 -- 172.21.15.98:0/2705971137 >> 172.21.15.98:0/2705971137 conn(0x7f29280f9ba0 msgr2=0x7f29280fa8f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:23:14.583 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:23:14.581+0000 7f292e43a700 1 -- 172.21.15.98:0/2705971137 shutdown_connections 2023-07-14T19:23:14.583 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:23:14.581+0000 7f292e43a700 1 -- 172.21.15.98:0/2705971137 wait complete. 2023-07-14T19:23:15.010 INFO:tasks.cephadm:Writing (initial) conf and keyring to smithi101 2023-07-14T19:23:15.011 DEBUG:teuthology.orchestra.run.smithi101:> set -ex 2023-07-14T19:23:15.011 DEBUG:teuthology.orchestra.run.smithi101:> dd of=/etc/ceph/ceph.conf 2023-07-14T19:23:15.032 DEBUG:teuthology.orchestra.run.smithi101:> set -ex 2023-07-14T19:23:15.032 DEBUG:teuthology.orchestra.run.smithi101:> dd of=/etc/ceph/ceph.client.admin.keyring 2023-07-14T19:23:15.093 INFO:tasks.cephadm:Adding host smithi101 to orchestrator... 2023-07-14T19:23:15.093 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81c9387d2e7576da065747ec9c16d1c7d69016ec shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 9adea648-227b-11ee-9b33-001a4aab830c -- ceph orch host add smithi101 2023-07-14T19:23:15.659 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/9adea648-227b-11ee-9b33-001a4aab830c/mon.a/config 2023-07-14T19:23:15.857 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:23:15 smithi098 ceph-mon[127486]: from='client.14176 -' entity='client.admin' cmd=[{"prefix": "orch client-keyring set", "entity": "client.admin", "placement": "*", "mode": "0755", "target": ["mon-mgr", ""]}]: dispatch 2023-07-14T19:23:15.857 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:23:15 smithi098 ceph-mon[127486]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' 2023-07-14T19:23:15.857 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:23:15 smithi098 ceph-mon[127486]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-14T19:23:15.857 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:23:15 smithi098 ceph-mon[127486]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T19:23:15.858 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:23:15 smithi098 ceph-mon[127486]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-14T19:23:15.858 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:23:15 smithi098 ceph-mon[127486]: Updating smithi098:/etc/ceph/ceph.conf 2023-07-14T19:23:15.858 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:23:15 smithi098 ceph-mon[127486]: Updating smithi098:/var/lib/ceph/9adea648-227b-11ee-9b33-001a4aab830c/config/ceph.conf 2023-07-14T19:23:15.858 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:23:15 smithi098 ceph-mon[127486]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' 2023-07-14T19:23:15.858 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:23:15 smithi098 ceph-mon[127486]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' 2023-07-14T19:23:15.858 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:23:15 smithi098 ceph-mon[127486]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' 2023-07-14T19:23:16.897 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:23:16.894+0000 7fa176538700 1 -- 172.21.15.98:0/2242696462 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa170104660 msgr2=0x7fa170104a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:23:16.897 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:23:16.894+0000 7fa176538700 1 --2- 172.21.15.98:0/2242696462 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa170104660 0x7fa170104a30 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7fa160009910 tx=0x7fa160009c20 comp rx=0 tx=0).stop 2023-07-14T19:23:16.898 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:23:16.895+0000 7fa176538700 1 -- 172.21.15.98:0/2242696462 shutdown_connections 2023-07-14T19:23:16.898 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:23:16.895+0000 7fa176538700 1 --2- 172.21.15.98:0/2242696462 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa170104660 0x7fa170104a30 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:23:16.898 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:23:16.895+0000 7fa176538700 1 -- 172.21.15.98:0/2242696462 >> 172.21.15.98:0/2242696462 conn(0x7fa1700fa140 msgr2=0x7fa1700fc550 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:23:16.898 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:23:16.895+0000 7fa176538700 1 -- 172.21.15.98:0/2242696462 shutdown_connections 2023-07-14T19:23:16.898 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:23:16.895+0000 7fa176538700 1 -- 172.21.15.98:0/2242696462 wait complete. 2023-07-14T19:23:16.898 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:23:16.895+0000 7fa176538700 1 Processor -- start 2023-07-14T19:23:16.898 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:23:16.895+0000 7fa176538700 1 -- start start 2023-07-14T19:23:16.899 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:23:16.896+0000 7fa176538700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa170104660 0x7fa17019ca60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:23:16.899 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:23:16.896+0000 7fa176538700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa17019cfa0 con 0x7fa170104660 2023-07-14T19:23:16.899 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:23:16.896+0000 7fa16ffff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa170104660 0x7fa17019ca60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:23:16.899 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:23:16.896+0000 7fa16ffff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa170104660 0x7fa17019ca60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:44672/0 (socket says 172.21.15.98:44672) 2023-07-14T19:23:16.899 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:23:16.896+0000 7fa16ffff700 1 -- 172.21.15.98:0/2671384732 learned_addr learned my addr 172.21.15.98:0/2671384732 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-07-14T19:23:16.899 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:23:16.896+0000 7fa16ffff700 1 -- 172.21.15.98:0/2671384732 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa1600095f0 con 0x7fa170104660 2023-07-14T19:23:16.899 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:23:16.896+0000 7fa16ffff700 1 --2- 172.21.15.98:0/2671384732 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa170104660 0x7fa17019ca60 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7fa160004f40 tx=0x7fa160005020 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:23:16.900 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:23:16.896+0000 7fa16d7fa700 1 -- 172.21.15.98:0/2671384732 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fa16001d070 con 0x7fa170104660 2023-07-14T19:23:16.900 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:23:16.896+0000 7fa16d7fa700 1 -- 172.21.15.98:0/2671384732 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fa160004370 con 0x7fa170104660 2023-07-14T19:23:16.900 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:23:16.896+0000 7fa176538700 1 -- 172.21.15.98:0/2671384732 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa17019d200 con 0x7fa170104660 2023-07-14T19:23:16.900 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:23:16.896+0000 7fa16d7fa700 1 -- 172.21.15.98:0/2671384732 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fa1600039b0 con 0x7fa170104660 2023-07-14T19:23:16.900 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:23:16.896+0000 7fa176538700 1 -- 172.21.15.98:0/2671384732 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa170196ae0 con 0x7fa170104660 2023-07-14T19:23:16.901 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:23:16.897+0000 7fa16d7fa700 1 -- 172.21.15.98:0/2671384732 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 13) v1 ==== 45539+0+0 (secure 0 0 0) 0x7fa160003c70 con 0x7fa170104660 2023-07-14T19:23:16.901 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:23:16.897+0000 7fa16d7fa700 1 --2- 172.21.15.98:0/2671384732 >> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] conn(0x7fa1580387b0 0x7fa15803ac60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:23:16.901 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:23:16.897+0000 7fa176538700 1 -- 172.21.15.98:0/2671384732 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa170061c70 con 0x7fa170104660 2023-07-14T19:23:16.901 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:23:16.897+0000 7fa16d7fa700 1 -- 172.21.15.98:0/2671384732 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7fa16004c590 con 0x7fa170104660 2023-07-14T19:23:16.902 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:23:16.897+0000 7fa16f7fe700 1 --2- 172.21.15.98:0/2671384732 >> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] conn(0x7fa1580387b0 0x7fa15803ac60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:23:16.902 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:23:16.898+0000 7fa16f7fe700 1 --2- 172.21.15.98:0/2671384732 >> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] conn(0x7fa1580387b0 0x7fa15803ac60 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7fa164006fd0 tx=0x7fa164006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:23:16.903 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:23:16.902+0000 7fa16d7fa700 1 -- 172.21.15.98:0/2671384732 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7fa16002b440 con 0x7fa170104660 2023-07-14T19:23:17.031 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:23:16 smithi098 ceph-mon[127486]: Updating smithi098:/etc/ceph/ceph.client.admin.keyring 2023-07-14T19:23:17.032 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:23:16 smithi098 ceph-mon[127486]: Updating smithi098:/var/lib/ceph/9adea648-227b-11ee-9b33-001a4aab830c/config/ceph.client.admin.keyring 2023-07-14T19:23:17.091 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:23:17.090+0000 7fa176538700 1 -- 172.21.15.98:0/2671384732 --> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] -- mgr_command(tid 0: {"prefix": "orch host add", "hostname": "smithi101", "target": ["mon-mgr", ""]}) v1 -- 0x7fa1701975a0 con 0x7fa1580387b0 2023-07-14T19:23:18.281 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:23:17 smithi098 ceph-mon[127486]: from='client.14178 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi101", "target": ["mon-mgr", ""]}]: dispatch 2023-07-14T19:23:18.281 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:23:17 smithi098 ceph-mon[127486]: Deploying cephadm binary to smithi101 2023-07-14T19:23:19.409 INFO:teuthology.orchestra.run.smithi098.stdout:Added host 'smithi101' with addr '172.21.15.101' 2023-07-14T19:23:19.409 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:23:19.407+0000 7fa16d7fa700 1 -- 172.21.15.98:0/2671384732 <== mgr.14152 v2:172.21.15.98:6800/452042164 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+49 (secure 0 0 0) 0x7fa1701975a0 con 0x7fa1580387b0 2023-07-14T19:23:19.411 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:23:19.410+0000 7fa176538700 1 -- 172.21.15.98:0/2671384732 >> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] conn(0x7fa1580387b0 msgr2=0x7fa15803ac60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:23:19.411 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:23:19.410+0000 7fa176538700 1 --2- 172.21.15.98:0/2671384732 >> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] conn(0x7fa1580387b0 0x7fa15803ac60 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7fa164006fd0 tx=0x7fa164006e40 comp rx=0 tx=0).stop 2023-07-14T19:23:19.411 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:23:19.410+0000 7fa176538700 1 -- 172.21.15.98:0/2671384732 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa170104660 msgr2=0x7fa17019ca60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:23:19.411 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:23:19.410+0000 7fa176538700 1 --2- 172.21.15.98:0/2671384732 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa170104660 0x7fa17019ca60 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7fa160004f40 tx=0x7fa160005020 comp rx=0 tx=0).stop 2023-07-14T19:23:19.411 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:23:19.410+0000 7fa176538700 1 -- 172.21.15.98:0/2671384732 shutdown_connections 2023-07-14T19:23:19.412 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:23:19.410+0000 7fa176538700 1 --2- 172.21.15.98:0/2671384732 >> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] conn(0x7fa1580387b0 0x7fa15803ac60 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:23:19.412 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:23:19.410+0000 7fa176538700 1 --2- 172.21.15.98:0/2671384732 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa170104660 0x7fa17019ca60 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:23:19.412 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:23:19.410+0000 7fa176538700 1 -- 172.21.15.98:0/2671384732 >> 172.21.15.98:0/2671384732 conn(0x7fa1700fa140 msgr2=0x7fa170068940 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:23:19.412 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:23:19.410+0000 7fa176538700 1 -- 172.21.15.98:0/2671384732 shutdown_connections 2023-07-14T19:23:19.412 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:23:19.410+0000 7fa176538700 1 -- 172.21.15.98:0/2671384732 wait complete. 2023-07-14T19:23:19.821 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81c9387d2e7576da065747ec9c16d1c7d69016ec shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 9adea648-227b-11ee-9b33-001a4aab830c -- ceph orch host ls --format=json 2023-07-14T19:23:20.373 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/9adea648-227b-11ee-9b33-001a4aab830c/mon.a/config 2023-07-14T19:23:20.671 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:23:20 smithi098 ceph-mon[127486]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' 2023-07-14T19:23:20.671 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:23:20 smithi098 ceph-mon[127486]: Added host smithi101 2023-07-14T19:23:20.671 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:23:20 smithi098 ceph-mon[127486]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-14T19:23:20.671 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:23:20 smithi098 ceph-mon[127486]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' 2023-07-14T19:23:21.578 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:23:21.577+0000 7f2b622d9700 1 -- 172.21.15.98:0/191588038 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2b5c0ff690 msgr2=0x7f2b5c0ffa60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:23:21.578 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:23:21.577+0000 7f2b622d9700 1 --2- 172.21.15.98:0/191588038 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2b5c0ff690 0x7f2b5c0ffa60 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f2b50009910 tx=0x7f2b50009c20 comp rx=0 tx=0).stop 2023-07-14T19:23:21.578 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:23:21.577+0000 7f2b622d9700 1 -- 172.21.15.98:0/191588038 shutdown_connections 2023-07-14T19:23:21.578 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:23:21.577+0000 7f2b622d9700 1 --2- 172.21.15.98:0/191588038 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2b5c0ff690 0x7f2b5c0ffa60 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:23:21.578 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:23:21.577+0000 7f2b622d9700 1 -- 172.21.15.98:0/191588038 >> 172.21.15.98:0/191588038 conn(0x7f2b5c075040 msgr2=0x7f2b5c075440 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:23:21.579 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:23:21.577+0000 7f2b622d9700 1 -- 172.21.15.98:0/191588038 shutdown_connections 2023-07-14T19:23:21.579 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:23:21.577+0000 7f2b622d9700 1 -- 172.21.15.98:0/191588038 wait complete. 2023-07-14T19:23:21.580 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:23:21.577+0000 7f2b622d9700 1 Processor -- start 2023-07-14T19:23:21.580 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:23:21.578+0000 7f2b622d9700 1 -- start start 2023-07-14T19:23:21.581 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:23:21.578+0000 7f2b622d9700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2b5c0ff690 0x7f2b5c1998f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:23:21.581 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:23:21.578+0000 7f2b622d9700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2b5c199e30 con 0x7f2b5c0ff690 2023-07-14T19:23:21.581 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:23:21.578+0000 7f2b5bfff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2b5c0ff690 0x7f2b5c1998f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:23:21.581 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:23:21.578+0000 7f2b5bfff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2b5c0ff690 0x7f2b5c1998f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:44698/0 (socket says 172.21.15.98:44698) 2023-07-14T19:23:21.581 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:23:21.578+0000 7f2b5bfff700 1 -- 172.21.15.98:0/3062524171 learned_addr learned my addr 172.21.15.98:0/3062524171 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-07-14T19:23:21.581 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:23:21.578+0000 7f2b5bfff700 1 -- 172.21.15.98:0/3062524171 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2b500095f0 con 0x7f2b5c0ff690 2023-07-14T19:23:21.582 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:23:21.578+0000 7f2b5bfff700 1 --2- 172.21.15.98:0/3062524171 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2b5c0ff690 0x7f2b5c1998f0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f2b50004f40 tx=0x7f2b50005020 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:23:21.582 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:23:21.579+0000 7f2b597fa700 1 -- 172.21.15.98:0/3062524171 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f2b5001d070 con 0x7f2b5c0ff690 2023-07-14T19:23:21.582 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:23:21.579+0000 7f2b622d9700 1 -- 172.21.15.98:0/3062524171 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2b5c19a030 con 0x7f2b5c0ff690 2023-07-14T19:23:21.582 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:23:21.579+0000 7f2b597fa700 1 -- 172.21.15.98:0/3062524171 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f2b5000be20 con 0x7f2b5c0ff690 2023-07-14T19:23:21.582 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:23:21.579+0000 7f2b597fa700 1 -- 172.21.15.98:0/3062524171 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f2b50022d00 con 0x7f2b5c0ff690 2023-07-14T19:23:21.582 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:23:21.579+0000 7f2b622d9700 1 -- 172.21.15.98:0/3062524171 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2b5c19a450 con 0x7f2b5c0ff690 2023-07-14T19:23:21.582 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:23:21.579+0000 7f2b597fa700 1 -- 172.21.15.98:0/3062524171 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 13) v1 ==== 45539+0+0 (secure 0 0 0) 0x7f2b5000f460 con 0x7f2b5c0ff690 2023-07-14T19:23:21.583 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:23:21.580+0000 7f2b622d9700 1 -- 172.21.15.98:0/3062524171 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2b5c061c70 con 0x7f2b5c0ff690 2023-07-14T19:23:21.583 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:23:21.580+0000 7f2b597fa700 1 --2- 172.21.15.98:0/3062524171 >> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] conn(0x7f2b440387b0 0x7f2b4403ac60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:23:21.583 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:23:21.580+0000 7f2b597fa700 1 -- 172.21.15.98:0/3062524171 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f2b5004c970 con 0x7f2b5c0ff690 2023-07-14T19:23:21.583 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:23:21.580+0000 7f2b5b7fe700 1 --2- 172.21.15.98:0/3062524171 >> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] conn(0x7f2b440387b0 0x7f2b4403ac60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:23:21.583 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:23:21.580+0000 7f2b5b7fe700 1 --2- 172.21.15.98:0/3062524171 >> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] conn(0x7f2b440387b0 0x7f2b4403ac60 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f2b4c006fd0 tx=0x7f2b4c006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:23:21.585 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:23:21.585+0000 7f2b597fa700 1 -- 172.21.15.98:0/3062524171 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f2b5000b960 con 0x7f2b5c0ff690 2023-07-14T19:23:21.779 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:23:21.778+0000 7f2b622d9700 1 -- 172.21.15.98:0/3062524171 --> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] -- mgr_command(tid 0: {"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f2b5c1037b0 con 0x7f2b440387b0 2023-07-14T19:23:21.780 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:23:21.779+0000 7f2b597fa700 1 -- 172.21.15.98:0/3062524171 <== mgr.14152 v2:172.21.15.98:6800/452042164 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+160 (secure 0 0 0) 0x7f2b5c1037b0 con 0x7f2b440387b0 2023-07-14T19:23:21.780 INFO:teuthology.orchestra.run.smithi098.stdout: 2023-07-14T19:23:21.780 INFO:teuthology.orchestra.run.smithi098.stdout:[{"addr": "172.21.15.98", "hostname": "smithi098", "labels": [], "status": ""}, {"addr": "172.21.15.101", "hostname": "smithi101", "labels": [], "status": ""}] 2023-07-14T19:23:21.781 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:23:21 smithi098 ceph-mon[127486]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' 2023-07-14T19:23:21.782 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:23:21.781+0000 7f2b622d9700 1 -- 172.21.15.98:0/3062524171 >> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] conn(0x7f2b440387b0 msgr2=0x7f2b4403ac60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:23:21.782 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:23:21.781+0000 7f2b622d9700 1 --2- 172.21.15.98:0/3062524171 >> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] conn(0x7f2b440387b0 0x7f2b4403ac60 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f2b4c006fd0 tx=0x7f2b4c006e40 comp rx=0 tx=0).stop 2023-07-14T19:23:21.782 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:23:21.781+0000 7f2b622d9700 1 -- 172.21.15.98:0/3062524171 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2b5c0ff690 msgr2=0x7f2b5c1998f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:23:21.783 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:23:21.781+0000 7f2b622d9700 1 --2- 172.21.15.98:0/3062524171 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2b5c0ff690 0x7f2b5c1998f0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f2b50004f40 tx=0x7f2b50005020 comp rx=0 tx=0).stop 2023-07-14T19:23:21.783 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:23:21.781+0000 7f2b622d9700 1 -- 172.21.15.98:0/3062524171 shutdown_connections 2023-07-14T19:23:21.783 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:23:21.781+0000 7f2b622d9700 1 --2- 172.21.15.98:0/3062524171 >> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] conn(0x7f2b440387b0 0x7f2b4403ac60 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:23:21.783 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:23:21.781+0000 7f2b622d9700 1 --2- 172.21.15.98:0/3062524171 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2b5c0ff690 0x7f2b5c1998f0 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:23:21.783 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:23:21.781+0000 7f2b622d9700 1 -- 172.21.15.98:0/3062524171 >> 172.21.15.98:0/3062524171 conn(0x7f2b5c075040 msgr2=0x7f2b5c0fc190 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:23:21.783 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:23:21.781+0000 7f2b622d9700 1 -- 172.21.15.98:0/3062524171 shutdown_connections 2023-07-14T19:23:21.784 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:23:21.781+0000 7f2b622d9700 1 -- 172.21.15.98:0/3062524171 wait complete. 2023-07-14T19:23:22.228 INFO:tasks.cephadm:Writing (initial) conf and keyring to smithi139 2023-07-14T19:23:22.229 DEBUG:teuthology.orchestra.run.smithi139:> set -ex 2023-07-14T19:23:22.229 DEBUG:teuthology.orchestra.run.smithi139:> dd of=/etc/ceph/ceph.conf 2023-07-14T19:23:22.249 DEBUG:teuthology.orchestra.run.smithi139:> set -ex 2023-07-14T19:23:22.250 DEBUG:teuthology.orchestra.run.smithi139:> dd of=/etc/ceph/ceph.client.admin.keyring 2023-07-14T19:23:22.308 INFO:tasks.cephadm:Adding host smithi139 to orchestrator... 2023-07-14T19:23:22.309 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81c9387d2e7576da065747ec9c16d1c7d69016ec shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 9adea648-227b-11ee-9b33-001a4aab830c -- ceph orch host add smithi139 2023-07-14T19:23:22.858 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/9adea648-227b-11ee-9b33-001a4aab830c/mon.a/config 2023-07-14T19:23:23.031 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:23:22 smithi098 ceph-mon[127486]: pgmap v3: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-14T19:23:23.032 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:23:22 smithi098 ceph-mon[127486]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' 2023-07-14T19:23:23.032 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:23:22 smithi098 ceph-mon[127486]: from='client.14180 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-07-14T19:23:24.091 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:23:24.087+0000 7faf9df30700 1 -- 172.21.15.98:0/1636876427 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7faf981057c0 msgr2=0x7faf98105b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:23:24.092 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:23:24.087+0000 7faf9df30700 1 --2- 172.21.15.98:0/1636876427 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7faf981057c0 0x7faf98105b90 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7faf88009910 tx=0x7faf88009c20 comp rx=0 tx=0).stop 2023-07-14T19:23:24.092 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:23:24.087+0000 7faf9df30700 1 -- 172.21.15.98:0/1636876427 shutdown_connections 2023-07-14T19:23:24.092 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:23:24.087+0000 7faf9df30700 1 --2- 172.21.15.98:0/1636876427 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7faf981057c0 0x7faf98105b90 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:23:24.092 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:23:24.087+0000 7faf9df30700 1 -- 172.21.15.98:0/1636876427 >> 172.21.15.98:0/1636876427 conn(0x7faf980743e0 msgr2=0x7faf980747e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:23:24.092 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:23:24.087+0000 7faf9df30700 1 -- 172.21.15.98:0/1636876427 shutdown_connections 2023-07-14T19:23:24.093 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:23:24.087+0000 7faf9df30700 1 -- 172.21.15.98:0/1636876427 wait complete. 2023-07-14T19:23:24.093 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:23:24.088+0000 7faf9df30700 1 Processor -- start 2023-07-14T19:23:24.093 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:23:24.088+0000 7faf9df30700 1 -- start start 2023-07-14T19:23:24.093 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:23:24.088+0000 7faf9df30700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7faf981057c0 0x7faf9819a970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:23:24.093 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:23:24.088+0000 7faf9df30700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faf9819aeb0 con 0x7faf981057c0 2023-07-14T19:23:24.093 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:23:24.089+0000 7faf977fe700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7faf981057c0 0x7faf9819a970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:23:24.093 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:23:24.089+0000 7faf977fe700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7faf981057c0 0x7faf9819a970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:44704/0 (socket says 172.21.15.98:44704) 2023-07-14T19:23:24.094 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:23:24.089+0000 7faf977fe700 1 -- 172.21.15.98:0/989424239 learned_addr learned my addr 172.21.15.98:0/989424239 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-07-14T19:23:24.094 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:23:24.089+0000 7faf977fe700 1 -- 172.21.15.98:0/989424239 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faf880095f0 con 0x7faf981057c0 2023-07-14T19:23:24.094 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:23:24.089+0000 7faf977fe700 1 --2- 172.21.15.98:0/989424239 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7faf981057c0 0x7faf9819a970 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7faf88004f40 tx=0x7faf88005020 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:23:24.094 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:23:24.089+0000 7faf94ff9700 1 -- 172.21.15.98:0/989424239 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7faf8801d070 con 0x7faf981057c0 2023-07-14T19:23:24.094 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:23:24.089+0000 7faf94ff9700 1 -- 172.21.15.98:0/989424239 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7faf88004370 con 0x7faf981057c0 2023-07-14T19:23:24.094 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:23:24.089+0000 7faf9df30700 1 -- 172.21.15.98:0/989424239 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faf9819b0b0 con 0x7faf981057c0 2023-07-14T19:23:24.094 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:23:24.089+0000 7faf94ff9700 1 -- 172.21.15.98:0/989424239 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7faf880039b0 con 0x7faf981057c0 2023-07-14T19:23:24.094 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:23:24.089+0000 7faf9df30700 1 -- 172.21.15.98:0/989424239 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faf981949f0 con 0x7faf981057c0 2023-07-14T19:23:24.095 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:23:24.090+0000 7faf94ff9700 1 -- 172.21.15.98:0/989424239 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 13) v1 ==== 45539+0+0 (secure 0 0 0) 0x7faf88003c70 con 0x7faf981057c0 2023-07-14T19:23:24.095 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:23:24.090+0000 7faf9df30700 1 -- 172.21.15.98:0/989424239 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faf98061c70 con 0x7faf981057c0 2023-07-14T19:23:24.095 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:23:24.090+0000 7faf94ff9700 1 --2- 172.21.15.98:0/989424239 >> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] conn(0x7faf840387b0 0x7faf8403ac60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:23:24.095 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:23:24.090+0000 7faf94ff9700 1 -- 172.21.15.98:0/989424239 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7faf8804c530 con 0x7faf981057c0 2023-07-14T19:23:24.095 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:23:24.090+0000 7faf96ffd700 1 --2- 172.21.15.98:0/989424239 >> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] conn(0x7faf840387b0 0x7faf8403ac60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:23:24.095 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:23:24.091+0000 7faf96ffd700 1 --2- 172.21.15.98:0/989424239 >> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] conn(0x7faf840387b0 0x7faf8403ac60 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7faf80006fd0 tx=0x7faf80006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:23:24.096 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:23:24.095+0000 7faf94ff9700 1 -- 172.21.15.98:0/989424239 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7faf88029770 con 0x7faf981057c0 2023-07-14T19:23:24.285 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:23:24.284+0000 7faf9df30700 1 -- 172.21.15.98:0/989424239 --> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] -- mgr_command(tid 0: {"prefix": "orch host add", "hostname": "smithi139", "target": ["mon-mgr", ""]}) v1 -- 0x7faf98195420 con 0x7faf840387b0 2023-07-14T19:23:25.031 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:23:24 smithi098 ceph-mon[127486]: pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-14T19:23:25.031 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:23:24 smithi098 ceph-mon[127486]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' 2023-07-14T19:23:25.032 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:23:24 smithi098 ceph-mon[127486]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' 2023-07-14T19:23:25.032 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:23:24 smithi098 ceph-mon[127486]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' 2023-07-14T19:23:25.032 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:23:24 smithi098 ceph-mon[127486]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' 2023-07-14T19:23:25.032 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:23:24 smithi098 ceph-mon[127486]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi101", "name": "osd_memory_target"}]: dispatch 2023-07-14T19:23:25.032 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:23:24 smithi098 ceph-mon[127486]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T19:23:25.032 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:23:24 smithi098 ceph-mon[127486]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-14T19:23:26.031 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:23:25 smithi098 ceph-mon[127486]: from='client.14182 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi139", "target": ["mon-mgr", ""]}]: dispatch 2023-07-14T19:23:26.031 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:23:25 smithi098 ceph-mon[127486]: Updating smithi101:/etc/ceph/ceph.conf 2023-07-14T19:23:26.032 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:23:25 smithi098 ceph-mon[127486]: Updating smithi101:/var/lib/ceph/9adea648-227b-11ee-9b33-001a4aab830c/config/ceph.conf 2023-07-14T19:23:26.032 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:23:25 smithi098 ceph-mon[127486]: Deploying cephadm binary to smithi139 2023-07-14T19:23:26.562 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:23:26.561+0000 7faf94ff9700 1 -- 172.21.15.98:0/989424239 <== mgr.14152 v2:172.21.15.98:6800/452042164 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+49 (secure 0 0 0) 0x7faf98195420 con 0x7faf840387b0 2023-07-14T19:23:26.563 INFO:teuthology.orchestra.run.smithi098.stdout:Added host 'smithi139' with addr '172.21.15.139' 2023-07-14T19:23:26.564 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:23:26.564+0000 7faf9df30700 1 -- 172.21.15.98:0/989424239 >> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] conn(0x7faf840387b0 msgr2=0x7faf8403ac60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:23:26.565 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:23:26.564+0000 7faf9df30700 1 --2- 172.21.15.98:0/989424239 >> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] conn(0x7faf840387b0 0x7faf8403ac60 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7faf80006fd0 tx=0x7faf80006e40 comp rx=0 tx=0).stop 2023-07-14T19:23:26.565 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:23:26.564+0000 7faf9df30700 1 -- 172.21.15.98:0/989424239 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7faf981057c0 msgr2=0x7faf9819a970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:23:26.565 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:23:26.564+0000 7faf9df30700 1 --2- 172.21.15.98:0/989424239 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7faf981057c0 0x7faf9819a970 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7faf88004f40 tx=0x7faf88005020 comp rx=0 tx=0).stop 2023-07-14T19:23:26.565 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:23:26.564+0000 7faf9df30700 1 -- 172.21.15.98:0/989424239 shutdown_connections 2023-07-14T19:23:26.565 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:23:26.564+0000 7faf9df30700 1 --2- 172.21.15.98:0/989424239 >> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] conn(0x7faf840387b0 0x7faf8403ac60 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:23:26.565 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:23:26.564+0000 7faf9df30700 1 --2- 172.21.15.98:0/989424239 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7faf981057c0 0x7faf9819a970 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:23:26.565 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:23:26.564+0000 7faf9df30700 1 -- 172.21.15.98:0/989424239 >> 172.21.15.98:0/989424239 conn(0x7faf980743e0 msgr2=0x7faf98106d40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:23:26.566 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:23:26.564+0000 7faf9df30700 1 -- 172.21.15.98:0/989424239 shutdown_connections 2023-07-14T19:23:26.566 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:23:26.564+0000 7faf9df30700 1 -- 172.21.15.98:0/989424239 wait complete. 2023-07-14T19:23:26.935 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:23:26 smithi098 ceph-mon[127486]: Updating smithi101:/etc/ceph/ceph.client.admin.keyring 2023-07-14T19:23:26.935 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:23:26 smithi098 ceph-mon[127486]: Updating smithi101:/var/lib/ceph/9adea648-227b-11ee-9b33-001a4aab830c/config/ceph.client.admin.keyring 2023-07-14T19:23:26.936 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:23:26 smithi098 ceph-mon[127486]: pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-14T19:23:26.936 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:23:26 smithi098 ceph-mon[127486]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' 2023-07-14T19:23:26.936 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:23:26 smithi098 ceph-mon[127486]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' 2023-07-14T19:23:26.937 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:23:26 smithi098 ceph-mon[127486]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' 2023-07-14T19:23:26.937 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:23:26 smithi098 ceph-mon[127486]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' 2023-07-14T19:23:26.937 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:23:26 smithi098 ceph-mon[127486]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-14T19:23:27.004 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81c9387d2e7576da065747ec9c16d1c7d69016ec shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 9adea648-227b-11ee-9b33-001a4aab830c -- ceph orch host ls --format=json 2023-07-14T19:23:27.554 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/9adea648-227b-11ee-9b33-001a4aab830c/mon.a/config 2023-07-14T19:23:28.031 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:23:27 smithi098 ceph-mon[127486]: Added host smithi139 2023-07-14T19:23:28.032 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:23:27 smithi098 ceph-mon[127486]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' 2023-07-14T19:23:28.032 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:23:27 smithi098 ceph-mon[127486]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' 2023-07-14T19:23:28.764 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:23:28.763+0000 7f55d99d7700 1 -- 172.21.15.98:0/560551449 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f55d4105720 msgr2=0x7f55d4105af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:23:28.765 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:23:28.763+0000 7f55d99d7700 1 --2- 172.21.15.98:0/560551449 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f55d4105720 0x7f55d4105af0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f55c8009910 tx=0x7f55c8009c20 comp rx=0 tx=0).stop 2023-07-14T19:23:28.765 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:23:28.763+0000 7f55d99d7700 1 -- 172.21.15.98:0/560551449 shutdown_connections 2023-07-14T19:23:28.765 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:23:28.763+0000 7f55d99d7700 1 --2- 172.21.15.98:0/560551449 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f55d4105720 0x7f55d4105af0 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:23:28.765 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:23:28.763+0000 7f55d99d7700 1 -- 172.21.15.98:0/560551449 >> 172.21.15.98:0/560551449 conn(0x7f55d40fb260 msgr2=0x7f55d40fd670 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:23:28.765 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:23:28.763+0000 7f55d99d7700 1 -- 172.21.15.98:0/560551449 shutdown_connections 2023-07-14T19:23:28.766 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:23:28.763+0000 7f55d99d7700 1 -- 172.21.15.98:0/560551449 wait complete. 2023-07-14T19:23:28.766 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:23:28.764+0000 7f55d99d7700 1 Processor -- start 2023-07-14T19:23:28.767 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:23:28.764+0000 7f55d99d7700 1 -- start start 2023-07-14T19:23:28.768 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:23:28.764+0000 7f55d99d7700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f55d4105720 0x7f55d419a950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:23:28.768 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:23:28.764+0000 7f55d99d7700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f55d419ae90 con 0x7f55d4105720 2023-07-14T19:23:28.768 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:23:28.764+0000 7f55d2ffd700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f55d4105720 0x7f55d419a950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:23:28.768 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:23:28.764+0000 7f55d2ffd700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f55d4105720 0x7f55d419a950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:44730/0 (socket says 172.21.15.98:44730) 2023-07-14T19:23:28.768 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:23:28.764+0000 7f55d2ffd700 1 -- 172.21.15.98:0/4068317379 learned_addr learned my addr 172.21.15.98:0/4068317379 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-07-14T19:23:28.769 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:23:28.765+0000 7f55d2ffd700 1 -- 172.21.15.98:0/4068317379 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f55c80095f0 con 0x7f55d4105720 2023-07-14T19:23:28.769 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:23:28.765+0000 7f55d2ffd700 1 --2- 172.21.15.98:0/4068317379 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f55d4105720 0x7f55d419a950 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f55c80092e0 tx=0x7f55c8004ff0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:23:28.769 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:23:28.765+0000 7f55d89d5700 1 -- 172.21.15.98:0/4068317379 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f55c801d070 con 0x7f55d4105720 2023-07-14T19:23:28.769 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:23:28.765+0000 7f55d89d5700 1 -- 172.21.15.98:0/4068317379 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f55c800bc10 con 0x7f55d4105720 2023-07-14T19:23:28.769 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:23:28.765+0000 7f55d99d7700 1 -- 172.21.15.98:0/4068317379 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f55d419b090 con 0x7f55d4105720 2023-07-14T19:23:28.769 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:23:28.765+0000 7f55d89d5700 1 -- 172.21.15.98:0/4068317379 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f55c8003930 con 0x7f55d4105720 2023-07-14T19:23:28.769 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:23:28.765+0000 7f55d99d7700 1 -- 172.21.15.98:0/4068317379 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f55d41949d0 con 0x7f55d4105720 2023-07-14T19:23:28.770 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:23:28.766+0000 7f55d89d5700 1 -- 172.21.15.98:0/4068317379 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 13) v1 ==== 45539+0+0 (secure 0 0 0) 0x7f55c8003b60 con 0x7f55d4105720 2023-07-14T19:23:28.770 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:23:28.766+0000 7f55d99d7700 1 -- 172.21.15.98:0/4068317379 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f55d4061c70 con 0x7f55d4105720 2023-07-14T19:23:28.770 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:23:28.766+0000 7f55d89d5700 1 --2- 172.21.15.98:0/4068317379 >> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] conn(0x7f55bc038800 0x7f55bc03acb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:23:28.770 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:23:28.766+0000 7f55d89d5700 1 -- 172.21.15.98:0/4068317379 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f55c804c510 con 0x7f55d4105720 2023-07-14T19:23:28.770 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:23:28.766+0000 7f55d27fc700 1 --2- 172.21.15.98:0/4068317379 >> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] conn(0x7f55bc038800 0x7f55bc03acb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:23:28.770 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:23:28.767+0000 7f55d27fc700 1 --2- 172.21.15.98:0/4068317379 >> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] conn(0x7f55bc038800 0x7f55bc03acb0 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f55c4006fd0 tx=0x7f55c4006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:23:28.771 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:23:28.771+0000 7f55d89d5700 1 -- 172.21.15.98:0/4068317379 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f55c8029950 con 0x7f55d4105720 2023-07-14T19:23:28.965 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:23:28.964+0000 7f55d99d7700 1 -- 172.21.15.98:0/4068317379 --> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] -- mgr_command(tid 0: {"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f55d4195400 con 0x7f55bc038800 2023-07-14T19:23:28.966 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:23:28.965+0000 7f55d89d5700 1 -- 172.21.15.98:0/4068317379 <== mgr.14152 v2:172.21.15.98:6800/452042164 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+240 (secure 0 0 0) 0x7f55d4195400 con 0x7f55bc038800 2023-07-14T19:23:28.966 INFO:teuthology.orchestra.run.smithi098.stdout: 2023-07-14T19:23:28.966 INFO:teuthology.orchestra.run.smithi098.stdout:[{"addr": "172.21.15.98", "hostname": "smithi098", "labels": [], "status": ""}, {"addr": "172.21.15.101", "hostname": "smithi101", "labels": [], "status": ""}, {"addr": "172.21.15.139", "hostname": "smithi139", "labels": [], "status": ""}] 2023-07-14T19:23:28.968 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:23:28.967+0000 7f55d99d7700 1 -- 172.21.15.98:0/4068317379 >> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] conn(0x7f55bc038800 msgr2=0x7f55bc03acb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:23:28.968 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:23:28.967+0000 7f55d99d7700 1 --2- 172.21.15.98:0/4068317379 >> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] conn(0x7f55bc038800 0x7f55bc03acb0 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f55c4006fd0 tx=0x7f55c4006e40 comp rx=0 tx=0).stop 2023-07-14T19:23:28.968 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:23:28.967+0000 7f55d99d7700 1 -- 172.21.15.98:0/4068317379 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f55d4105720 msgr2=0x7f55d419a950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:23:28.968 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:23:28.967+0000 7f55d99d7700 1 --2- 172.21.15.98:0/4068317379 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f55d4105720 0x7f55d419a950 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f55c80092e0 tx=0x7f55c8004ff0 comp rx=0 tx=0).stop 2023-07-14T19:23:28.969 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:23:28.967+0000 7f55d99d7700 1 -- 172.21.15.98:0/4068317379 shutdown_connections 2023-07-14T19:23:28.969 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:23:28.967+0000 7f55d99d7700 1 --2- 172.21.15.98:0/4068317379 >> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] conn(0x7f55bc038800 0x7f55bc03acb0 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:23:28.969 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:23:28.967+0000 7f55d99d7700 1 --2- 172.21.15.98:0/4068317379 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f55d4105720 0x7f55d419a950 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:23:28.969 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:23:28.967+0000 7f55d99d7700 1 -- 172.21.15.98:0/4068317379 >> 172.21.15.98:0/4068317379 conn(0x7f55d40fb260 msgr2=0x7f55d40fbec0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:23:28.969 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:23:28.967+0000 7f55d99d7700 1 -- 172.21.15.98:0/4068317379 shutdown_connections 2023-07-14T19:23:28.969 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:23:28.967+0000 7f55d99d7700 1 -- 172.21.15.98:0/4068317379 wait complete. 2023-07-14T19:23:28.993 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:23:28 smithi098 ceph-mon[127486]: pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-14T19:23:29.298 INFO:tasks.cephadm:Setting crush tunables to default 2023-07-14T19:23:29.299 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81c9387d2e7576da065747ec9c16d1c7d69016ec shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 9adea648-227b-11ee-9b33-001a4aab830c -- ceph osd crush tunables default 2023-07-14T19:23:29.852 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/9adea648-227b-11ee-9b33-001a4aab830c/mon.a/config 2023-07-14T19:23:30.236 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:23:29 smithi098 ceph-mon[127486]: from='client.14184 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-07-14T19:23:30.236 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:23:29 smithi098 ceph-mon[127486]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' 2023-07-14T19:23:30.237 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:23:29 smithi098 ceph-mon[127486]: pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-14T19:23:31.081 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:23:31.079+0000 7f3dc5245700 1 -- 172.21.15.98:0/3123430619 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3dc0104660 msgr2=0x7f3dc0104a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:23:31.081 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:23:31.079+0000 7f3dc5245700 1 --2- 172.21.15.98:0/3123430619 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3dc0104660 0x7f3dc0104a30 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f3db0009910 tx=0x7f3db0009c20 comp rx=0 tx=0).stop 2023-07-14T19:23:31.081 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:23:31.079+0000 7f3dc5245700 1 -- 172.21.15.98:0/3123430619 shutdown_connections 2023-07-14T19:23:31.081 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:23:31.079+0000 7f3dc5245700 1 --2- 172.21.15.98:0/3123430619 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3dc0104660 0x7f3dc0104a30 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:23:31.081 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:23:31.079+0000 7f3dc5245700 1 -- 172.21.15.98:0/3123430619 >> 172.21.15.98:0/3123430619 conn(0x7f3dc00743e0 msgr2=0x7f3dc00747e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:23:31.081 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:23:31.079+0000 7f3dc5245700 1 -- 172.21.15.98:0/3123430619 shutdown_connections 2023-07-14T19:23:31.082 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:23:31.079+0000 7f3dc5245700 1 -- 172.21.15.98:0/3123430619 wait complete. 2023-07-14T19:23:31.083 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:23:31.080+0000 7f3dc5245700 1 Processor -- start 2023-07-14T19:23:31.083 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:23:31.080+0000 7f3dc5245700 1 -- start start 2023-07-14T19:23:31.083 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:23:31.081+0000 7f3dc5245700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3dc0104660 0x7f3dc01959f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:23:31.083 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:23:31.081+0000 7f3dc5245700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3dc0195f30 con 0x7f3dc0104660 2023-07-14T19:23:31.083 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:23:31.081+0000 7f3dbed9d700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3dc0104660 0x7f3dc01959f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:23:31.084 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:23:31.081+0000 7f3dbed9d700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3dc0104660 0x7f3dc01959f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:44736/0 (socket says 172.21.15.98:44736) 2023-07-14T19:23:31.084 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:23:31.081+0000 7f3dbed9d700 1 -- 172.21.15.98:0/1384810344 learned_addr learned my addr 172.21.15.98:0/1384810344 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-07-14T19:23:31.084 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:23:31.081+0000 7f3dbed9d700 1 -- 172.21.15.98:0/1384810344 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3db00095f0 con 0x7f3dc0104660 2023-07-14T19:23:31.084 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:23:31.081+0000 7f3dbed9d700 1 --2- 172.21.15.98:0/1384810344 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3dc0104660 0x7f3dc01959f0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f3db0006010 tx=0x7f3db0004ff0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:23:31.084 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:23:31.081+0000 7f3da7fff700 1 -- 172.21.15.98:0/1384810344 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f3db001d070 con 0x7f3dc0104660 2023-07-14T19:23:31.084 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:23:31.081+0000 7f3dc5245700 1 -- 172.21.15.98:0/1384810344 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3dc0196130 con 0x7f3dc0104660 2023-07-14T19:23:31.085 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:23:31.081+0000 7f3da7fff700 1 -- 172.21.15.98:0/1384810344 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f3db000bc10 con 0x7f3dc0104660 2023-07-14T19:23:31.085 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:23:31.081+0000 7f3dc5245700 1 -- 172.21.15.98:0/1384810344 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3dc01965b0 con 0x7f3dc0104660 2023-07-14T19:23:31.085 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:23:31.081+0000 7f3da7fff700 1 -- 172.21.15.98:0/1384810344 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f3db0003910 con 0x7f3dc0104660 2023-07-14T19:23:31.085 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:23:31.082+0000 7f3da7fff700 1 -- 172.21.15.98:0/1384810344 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 13) v1 ==== 45539+0+0 (secure 0 0 0) 0x7f3db0003bd0 con 0x7f3dc0104660 2023-07-14T19:23:31.085 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:23:31.082+0000 7f3dc5245700 1 -- 172.21.15.98:0/1384810344 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3dc0196860 con 0x7f3dc0104660 2023-07-14T19:23:31.085 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:23:31.082+0000 7f3da7fff700 1 --2- 172.21.15.98:0/1384810344 >> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] conn(0x7f3da80387b0 0x7f3da803ac60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:23:31.086 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:23:31.082+0000 7f3da7fff700 1 -- 172.21.15.98:0/1384810344 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f3db004c4d0 con 0x7f3dc0104660 2023-07-14T19:23:31.086 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:23:31.083+0000 7f3dbe59c700 1 --2- 172.21.15.98:0/1384810344 >> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] conn(0x7f3da80387b0 0x7f3da803ac60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:23:31.086 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:23:31.083+0000 7f3dbe59c700 1 --2- 172.21.15.98:0/1384810344 >> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] conn(0x7f3da80387b0 0x7f3da803ac60 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f3db4006fd0 tx=0x7f3db4006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:23:31.088 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:23:31.087+0000 7f3da7fff700 1 -- 172.21.15.98:0/1384810344 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f3dc0196860 con 0x7f3dc0104660 2023-07-14T19:23:31.278 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:23:31.277+0000 7f3dc5245700 1 -- 172.21.15.98:0/1384810344 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd crush tunables", "profile": "default"} v 0) v1 -- 0x7f3dc0103900 con 0x7f3dc0104660 2023-07-14T19:23:32.220 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:23:32.219+0000 7f3da7fff700 1 -- 172.21.15.98:0/1384810344 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush tunables", "profile": "default"}]=0 adjusted tunables profile to default v4) v1 ==== 124+0+0 (secure 0 0 0) 0x7f3db0026020 con 0x7f3dc0104660 2023-07-14T19:23:32.222 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:23:32.222+0000 7f3dc5245700 1 -- 172.21.15.98:0/1384810344 >> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] conn(0x7f3da80387b0 msgr2=0x7f3da803ac60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:23:32.223 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:23:32.222+0000 7f3dc5245700 1 --2- 172.21.15.98:0/1384810344 >> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] conn(0x7f3da80387b0 0x7f3da803ac60 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f3db4006fd0 tx=0x7f3db4006e40 comp rx=0 tx=0).stop 2023-07-14T19:23:32.223 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:23:32.222+0000 7f3dc5245700 1 -- 172.21.15.98:0/1384810344 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3dc0104660 msgr2=0x7f3dc01959f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:23:32.223 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:23:32.222+0000 7f3dc5245700 1 --2- 172.21.15.98:0/1384810344 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3dc0104660 0x7f3dc01959f0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f3db0006010 tx=0x7f3db0004ff0 comp rx=0 tx=0).stop 2023-07-14T19:23:32.223 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:23:32.222+0000 7f3dc5245700 1 -- 172.21.15.98:0/1384810344 shutdown_connections 2023-07-14T19:23:32.223 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:23:32.222+0000 7f3dc5245700 1 --2- 172.21.15.98:0/1384810344 >> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] conn(0x7f3da80387b0 0x7f3da803ac60 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:23:32.223 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:23:32.222+0000 7f3dc5245700 1 --2- 172.21.15.98:0/1384810344 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3dc0104660 0x7f3dc01959f0 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:23:32.224 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:23:32.222+0000 7f3dc5245700 1 -- 172.21.15.98:0/1384810344 >> 172.21.15.98:0/1384810344 conn(0x7f3dc00743e0 msgr2=0x7f3dc0105c00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:23:32.224 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:23:32.222+0000 7f3dc5245700 1 -- 172.21.15.98:0/1384810344 shutdown_connections 2023-07-14T19:23:32.224 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:23:32.222+0000 7f3dc5245700 1 -- 172.21.15.98:0/1384810344 wait complete. 2023-07-14T19:23:32.224 INFO:teuthology.orchestra.run.smithi098.stderr:adjusted tunables profile to default 2023-07-14T19:23:32.531 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:23:32 smithi098 ceph-mon[127486]: from='client.? 172.21.15.98:0/1384810344' entity='client.admin' cmd=[{"prefix": "osd crush tunables", "profile": "default"}]: dispatch 2023-07-14T19:23:32.532 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:23:32 smithi098 ceph-mon[127486]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-14T19:23:32.532 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:23:32 smithi098 ceph-mon[127486]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' 2023-07-14T19:23:32.532 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:23:32 smithi098 ceph-mon[127486]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' 2023-07-14T19:23:32.532 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:23:32 smithi098 ceph-mon[127486]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' 2023-07-14T19:23:32.532 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:23:32 smithi098 ceph-mon[127486]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' 2023-07-14T19:23:32.532 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:23:32 smithi098 ceph-mon[127486]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi139", "name": "osd_memory_target"}]: dispatch 2023-07-14T19:23:32.533 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:23:32 smithi098 ceph-mon[127486]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T19:23:32.533 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:23:32 smithi098 ceph-mon[127486]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-14T19:23:32.533 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:23:32 smithi098 ceph-mon[127486]: Updating smithi139:/etc/ceph/ceph.conf 2023-07-14T19:23:32.600 INFO:tasks.cephadm:Adding mon.a on smithi098 2023-07-14T19:23:32.600 INFO:tasks.cephadm:Adding mon.b on smithi101 2023-07-14T19:23:32.600 INFO:tasks.cephadm:Adding mon.c on smithi139 2023-07-14T19:23:32.601 DEBUG:teuthology.orchestra.run.smithi139:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81c9387d2e7576da065747ec9c16d1c7d69016ec shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 9adea648-227b-11ee-9b33-001a4aab830c -- ceph orch apply mon '3;smithi098:172.21.15.98=a;smithi101:172.21.15.101=b;smithi139:172.21.15.139=c' 2023-07-14T19:23:33.194 INFO:teuthology.orchestra.run.smithi139.stderr:Inferring config /var/lib/ceph/9adea648-227b-11ee-9b33-001a4aab830c/config/ceph.conf 2023-07-14T19:23:33.531 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:23:33 smithi098 ceph-mon[127486]: Updating smithi139:/var/lib/ceph/9adea648-227b-11ee-9b33-001a4aab830c/config/ceph.conf 2023-07-14T19:23:33.532 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:23:33 smithi098 ceph-mon[127486]: from='client.? 172.21.15.98:0/1384810344' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2023-07-14T19:23:33.532 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:23:33 smithi098 ceph-mon[127486]: osdmap e4: 0 total, 0 up, 0 in 2023-07-14T19:23:33.532 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:23:33 smithi098 ceph-mon[127486]: Updating smithi139:/etc/ceph/ceph.client.admin.keyring 2023-07-14T19:23:33.532 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:23:33 smithi098 ceph-mon[127486]: Updating smithi139:/var/lib/ceph/9adea648-227b-11ee-9b33-001a4aab830c/config/ceph.client.admin.keyring 2023-07-14T19:23:33.532 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:23:33 smithi098 ceph-mon[127486]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' 2023-07-14T19:23:33.532 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:23:33 smithi098 ceph-mon[127486]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' 2023-07-14T19:23:33.532 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:23:33 smithi098 ceph-mon[127486]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' 2023-07-14T19:23:34.430 INFO:teuthology.orchestra.run.smithi139.stderr:2023-07-14T19:23:34.428+0000 7faee0277700 1 -- 172.21.15.139:0/205834528 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7faed80ff690 msgr2=0x7faed80ffa60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:23:34.431 INFO:teuthology.orchestra.run.smithi139.stderr:2023-07-14T19:23:34.428+0000 7faee0277700 1 --2- 172.21.15.139:0/205834528 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7faed80ff690 0x7faed80ffa60 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7faed4009910 tx=0x7faed4009c20 comp rx=0 tx=0).stop 2023-07-14T19:23:34.431 INFO:teuthology.orchestra.run.smithi139.stderr:2023-07-14T19:23:34.428+0000 7faee0277700 1 -- 172.21.15.139:0/205834528 shutdown_connections 2023-07-14T19:23:34.431 INFO:teuthology.orchestra.run.smithi139.stderr:2023-07-14T19:23:34.428+0000 7faee0277700 1 --2- 172.21.15.139:0/205834528 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7faed80ff690 0x7faed80ffa60 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:23:34.431 INFO:teuthology.orchestra.run.smithi139.stderr:2023-07-14T19:23:34.428+0000 7faee0277700 1 -- 172.21.15.139:0/205834528 >> 172.21.15.139:0/205834528 conn(0x7faed8075040 msgr2=0x7faed8075440 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:23:34.431 INFO:teuthology.orchestra.run.smithi139.stderr:2023-07-14T19:23:34.428+0000 7faee0277700 1 -- 172.21.15.139:0/205834528 shutdown_connections 2023-07-14T19:23:34.431 INFO:teuthology.orchestra.run.smithi139.stderr:2023-07-14T19:23:34.428+0000 7faee0277700 1 -- 172.21.15.139:0/205834528 wait complete. 2023-07-14T19:23:34.431 INFO:teuthology.orchestra.run.smithi139.stderr:2023-07-14T19:23:34.429+0000 7faee0277700 1 Processor -- start 2023-07-14T19:23:34.432 INFO:teuthology.orchestra.run.smithi139.stderr:2023-07-14T19:23:34.429+0000 7faee0277700 1 -- start start 2023-07-14T19:23:34.432 INFO:teuthology.orchestra.run.smithi139.stderr:2023-07-14T19:23:34.429+0000 7faee0277700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7faed80ff690 0x7faed81998f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:23:34.432 INFO:teuthology.orchestra.run.smithi139.stderr:2023-07-14T19:23:34.429+0000 7faee0277700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faed8199e30 con 0x7faed80ff690 2023-07-14T19:23:34.433 INFO:teuthology.orchestra.run.smithi139.stderr:2023-07-14T19:23:34.429+0000 7faede013700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7faed80ff690 0x7faed81998f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:23:34.433 INFO:teuthology.orchestra.run.smithi139.stderr:2023-07-14T19:23:34.429+0000 7faede013700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7faed80ff690 0x7faed81998f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.139:60610/0 (socket says 172.21.15.139:60610) 2023-07-14T19:23:34.434 INFO:teuthology.orchestra.run.smithi139.stderr:2023-07-14T19:23:34.429+0000 7faede013700 1 -- 172.21.15.139:0/2298400169 learned_addr learned my addr 172.21.15.139:0/2298400169 (peer_addr_for_me v2:172.21.15.139:0/0) 2023-07-14T19:23:34.434 INFO:teuthology.orchestra.run.smithi139.stderr:2023-07-14T19:23:34.429+0000 7faede013700 1 -- 172.21.15.139:0/2298400169 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faed40095f0 con 0x7faed80ff690 2023-07-14T19:23:34.434 INFO:teuthology.orchestra.run.smithi139.stderr:2023-07-14T19:23:34.430+0000 7faede013700 1 --2- 172.21.15.139:0/2298400169 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7faed80ff690 0x7faed81998f0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7faed4004f40 tx=0x7faed4005020 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:23:34.434 INFO:teuthology.orchestra.run.smithi139.stderr:2023-07-14T19:23:34.430+0000 7faecf7fe700 1 -- 172.21.15.139:0/2298400169 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7faed401f070 con 0x7faed80ff690 2023-07-14T19:23:34.434 INFO:teuthology.orchestra.run.smithi139.stderr:2023-07-14T19:23:34.430+0000 7faee0277700 1 -- 172.21.15.139:0/2298400169 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faed819a030 con 0x7faed80ff690 2023-07-14T19:23:34.434 INFO:teuthology.orchestra.run.smithi139.stderr:2023-07-14T19:23:34.430+0000 7faecf7fe700 1 -- 172.21.15.139:0/2298400169 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7faed400be20 con 0x7faed80ff690 2023-07-14T19:23:34.434 INFO:teuthology.orchestra.run.smithi139.stderr:2023-07-14T19:23:34.430+0000 7faecf7fe700 1 -- 172.21.15.139:0/2298400169 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7faed4024470 con 0x7faed80ff690 2023-07-14T19:23:34.435 INFO:teuthology.orchestra.run.smithi139.stderr:2023-07-14T19:23:34.430+0000 7faee0277700 1 -- 172.21.15.139:0/2298400169 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faed819a450 con 0x7faed80ff690 2023-07-14T19:23:34.435 INFO:teuthology.orchestra.run.smithi139.stderr:2023-07-14T19:23:34.431+0000 7faee0277700 1 -- 172.21.15.139:0/2298400169 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faed8061c70 con 0x7faed80ff690 2023-07-14T19:23:34.435 INFO:teuthology.orchestra.run.smithi139.stderr:2023-07-14T19:23:34.431+0000 7faecf7fe700 1 -- 172.21.15.139:0/2298400169 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 13) v1 ==== 45539+0+0 (secure 0 0 0) 0x7faed400b960 con 0x7faed80ff690 2023-07-14T19:23:34.435 INFO:teuthology.orchestra.run.smithi139.stderr:2023-07-14T19:23:34.431+0000 7faecf7fe700 1 --2- 172.21.15.139:0/2298400169 >> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] conn(0x7faec40387b0 0x7faec403ac60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:23:34.435 INFO:teuthology.orchestra.run.smithi139.stderr:2023-07-14T19:23:34.431+0000 7faecf7fe700 1 -- 172.21.15.139:0/2298400169 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7faed404ec20 con 0x7faed80ff690 2023-07-14T19:23:34.435 INFO:teuthology.orchestra.run.smithi139.stderr:2023-07-14T19:23:34.431+0000 7faedd812700 1 --2- 172.21.15.139:0/2298400169 >> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] conn(0x7faec40387b0 0x7faec403ac60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:23:34.435 INFO:teuthology.orchestra.run.smithi139.stderr:2023-07-14T19:23:34.432+0000 7faedd812700 1 --2- 172.21.15.139:0/2298400169 >> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] conn(0x7faec40387b0 0x7faec403ac60 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7faec8006fd0 tx=0x7faec8006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:23:34.437 INFO:teuthology.orchestra.run.smithi139.stderr:2023-07-14T19:23:34.436+0000 7faecf7fe700 1 -- 172.21.15.139:0/2298400169 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7faed402cd30 con 0x7faed80ff690 2023-07-14T19:23:34.531 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:23:34 smithi098 ceph-mon[127486]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-14T19:23:34.631 INFO:teuthology.orchestra.run.smithi139.stderr:2023-07-14T19:23:34.629+0000 7faee0277700 1 -- 172.21.15.139:0/2298400169 --> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi098:172.21.15.98=a;smithi101:172.21.15.101=b;smithi139:172.21.15.139=c", "target": ["mon-mgr", ""]}) v1 -- 0x7faed8103870 con 0x7faec40387b0 2023-07-14T19:23:34.637 INFO:teuthology.orchestra.run.smithi139.stderr:2023-07-14T19:23:34.636+0000 7faecf7fe700 1 -- 172.21.15.139:0/2298400169 <== mgr.14152 v2:172.21.15.98:6800/452042164 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+24 (secure 0 0 0) 0x7faed8103870 con 0x7faec40387b0 2023-07-14T19:23:34.637 INFO:teuthology.orchestra.run.smithi139.stdout:Scheduled mon update... 2023-07-14T19:23:34.639 INFO:teuthology.orchestra.run.smithi139.stderr:2023-07-14T19:23:34.638+0000 7faee0277700 1 -- 172.21.15.139:0/2298400169 >> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] conn(0x7faec40387b0 msgr2=0x7faec403ac60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:23:34.640 INFO:teuthology.orchestra.run.smithi139.stderr:2023-07-14T19:23:34.638+0000 7faee0277700 1 --2- 172.21.15.139:0/2298400169 >> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] conn(0x7faec40387b0 0x7faec403ac60 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7faec8006fd0 tx=0x7faec8006e40 comp rx=0 tx=0).stop 2023-07-14T19:23:34.640 INFO:teuthology.orchestra.run.smithi139.stderr:2023-07-14T19:23:34.638+0000 7faee0277700 1 -- 172.21.15.139:0/2298400169 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7faed80ff690 msgr2=0x7faed81998f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:23:34.640 INFO:teuthology.orchestra.run.smithi139.stderr:2023-07-14T19:23:34.638+0000 7faee0277700 1 --2- 172.21.15.139:0/2298400169 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7faed80ff690 0x7faed81998f0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7faed4004f40 tx=0x7faed4005020 comp rx=0 tx=0).stop 2023-07-14T19:23:34.640 INFO:teuthology.orchestra.run.smithi139.stderr:2023-07-14T19:23:34.638+0000 7faee0277700 1 -- 172.21.15.139:0/2298400169 shutdown_connections 2023-07-14T19:23:34.640 INFO:teuthology.orchestra.run.smithi139.stderr:2023-07-14T19:23:34.638+0000 7faee0277700 1 --2- 172.21.15.139:0/2298400169 >> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] conn(0x7faec40387b0 0x7faec403ac60 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:23:34.640 INFO:teuthology.orchestra.run.smithi139.stderr:2023-07-14T19:23:34.638+0000 7faee0277700 1 --2- 172.21.15.139:0/2298400169 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7faed80ff690 0x7faed81998f0 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:23:34.640 INFO:teuthology.orchestra.run.smithi139.stderr:2023-07-14T19:23:34.638+0000 7faee0277700 1 -- 172.21.15.139:0/2298400169 >> 172.21.15.139:0/2298400169 conn(0x7faed8075040 msgr2=0x7faed80fc190 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:23:34.641 INFO:teuthology.orchestra.run.smithi139.stderr:2023-07-14T19:23:34.638+0000 7faee0277700 1 -- 172.21.15.139:0/2298400169 shutdown_connections 2023-07-14T19:23:34.641 INFO:teuthology.orchestra.run.smithi139.stderr:2023-07-14T19:23:34.638+0000 7faee0277700 1 -- 172.21.15.139:0/2298400169 wait complete. 2023-07-14T19:23:35.073 DEBUG:teuthology.orchestra.run.smithi101:mon.b> sudo journalctl -f -n 0 -u ceph-9adea648-227b-11ee-9b33-001a4aab830c@mon.b.service 2023-07-14T19:23:35.076 DEBUG:teuthology.orchestra.run.smithi139:mon.c> sudo journalctl -f -n 0 -u ceph-9adea648-227b-11ee-9b33-001a4aab830c@mon.c.service 2023-07-14T19:23:35.079 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2023-07-14T19:23:35.079 DEBUG:teuthology.orchestra.run.smithi139:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81c9387d2e7576da065747ec9c16d1c7d69016ec shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 9adea648-227b-11ee-9b33-001a4aab830c -- ceph mon dump -f json 2023-07-14T19:23:35.108 INFO:journalctl@ceph.mon.c.smithi139.stdout:-- Logs begin at Fri 2023-07-14 18:56:10 UTC. -- 2023-07-14T19:23:35.111 INFO:journalctl@ceph.mon.b.smithi101.stdout:-- Logs begin at Fri 2023-07-14 18:56:58 UTC. -- 2023-07-14T19:23:35.986 INFO:teuthology.orchestra.run.smithi139.stderr:Inferring config /var/lib/ceph/9adea648-227b-11ee-9b33-001a4aab830c/config/ceph.conf 2023-07-14T19:23:36.031 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:23:35 smithi098 ceph-mon[127486]: from='client.14188 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi098:172.21.15.98=a;smithi101:172.21.15.101=b;smithi139:172.21.15.139=c", "target": ["mon-mgr", ""]}]: dispatch 2023-07-14T19:23:36.032 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:23:35 smithi098 ceph-mon[127486]: Saving service mon spec with placement smithi098:172.21.15.98=a;smithi101:172.21.15.101=b;smithi139:172.21.15.139=c;count:3 2023-07-14T19:23:36.032 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:23:35 smithi098 ceph-mon[127486]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' 2023-07-14T19:23:36.032 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:23:35 smithi098 ceph-mon[127486]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-14T19:23:36.032 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:23:35 smithi098 ceph-mon[127486]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T19:23:36.032 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:23:35 smithi098 ceph-mon[127486]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-14T19:23:36.032 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:23:35 smithi098 ceph-mon[127486]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' 2023-07-14T19:23:36.033 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:23:35 smithi098 ceph-mon[127486]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2023-07-14T19:23:36.033 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:23:35 smithi098 ceph-mon[127486]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T19:23:36.033 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:23:35 smithi098 ceph-mon[127486]: Deploying daemon mon.c on smithi139 2023-07-14T19:23:37.031 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:23:36 smithi098 ceph-mon[127486]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-14T19:23:37.821 INFO:teuthology.orchestra.run.smithi139.stderr:2023-07-14T19:23:37.818+0000 7f18d0890700 1 -- 172.21.15.139:0/3439604113 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f18cc104660 msgr2=0x7f18cc104a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:23:37.821 INFO:teuthology.orchestra.run.smithi139.stderr:2023-07-14T19:23:37.818+0000 7f18d0890700 1 --2- 172.21.15.139:0/3439604113 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f18cc104660 0x7f18cc104a30 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f18b4009910 tx=0x7f18b4009c20 comp rx=0 tx=0).stop 2023-07-14T19:23:37.821 INFO:teuthology.orchestra.run.smithi139.stderr:2023-07-14T19:23:37.818+0000 7f18d0890700 1 -- 172.21.15.139:0/3439604113 shutdown_connections 2023-07-14T19:23:37.821 INFO:teuthology.orchestra.run.smithi139.stderr:2023-07-14T19:23:37.818+0000 7f18d0890700 1 --2- 172.21.15.139:0/3439604113 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f18cc104660 0x7f18cc104a30 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:23:37.822 INFO:teuthology.orchestra.run.smithi139.stderr:2023-07-14T19:23:37.818+0000 7f18d0890700 1 -- 172.21.15.139:0/3439604113 >> 172.21.15.139:0/3439604113 conn(0x7f18cc0743e0 msgr2=0x7f18cc0747e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:23:37.822 INFO:teuthology.orchestra.run.smithi139.stderr:2023-07-14T19:23:37.818+0000 7f18d0890700 1 -- 172.21.15.139:0/3439604113 shutdown_connections 2023-07-14T19:23:37.822 INFO:teuthology.orchestra.run.smithi139.stderr:2023-07-14T19:23:37.818+0000 7f18d0890700 1 -- 172.21.15.139:0/3439604113 wait complete. 2023-07-14T19:23:37.822 INFO:teuthology.orchestra.run.smithi139.stderr:2023-07-14T19:23:37.819+0000 7f18d0890700 1 Processor -- start 2023-07-14T19:23:37.822 INFO:teuthology.orchestra.run.smithi139.stderr:2023-07-14T19:23:37.819+0000 7f18d0890700 1 -- start start 2023-07-14T19:23:37.823 INFO:teuthology.orchestra.run.smithi139.stderr:2023-07-14T19:23:37.819+0000 7f18d0890700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f18cc104660 0x7f18cc1998e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:23:37.823 INFO:teuthology.orchestra.run.smithi139.stderr:2023-07-14T19:23:37.819+0000 7f18d0890700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f18cc199e20 con 0x7f18cc104660 2023-07-14T19:23:37.824 INFO:teuthology.orchestra.run.smithi139.stderr:2023-07-14T19:23:37.820+0000 7f18ca59c700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f18cc104660 0x7f18cc1998e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:23:37.825 INFO:teuthology.orchestra.run.smithi139.stderr:2023-07-14T19:23:37.820+0000 7f18ca59c700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f18cc104660 0x7f18cc1998e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.139:60616/0 (socket says 172.21.15.139:60616) 2023-07-14T19:23:37.825 INFO:teuthology.orchestra.run.smithi139.stderr:2023-07-14T19:23:37.820+0000 7f18ca59c700 1 -- 172.21.15.139:0/4207913916 learned_addr learned my addr 172.21.15.139:0/4207913916 (peer_addr_for_me v2:172.21.15.139:0/0) 2023-07-14T19:23:37.825 INFO:teuthology.orchestra.run.smithi139.stderr:2023-07-14T19:23:37.820+0000 7f18ca59c700 1 -- 172.21.15.139:0/4207913916 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f18b40095f0 con 0x7f18cc104660 2023-07-14T19:23:37.825 INFO:teuthology.orchestra.run.smithi139.stderr:2023-07-14T19:23:37.820+0000 7f18ca59c700 1 --2- 172.21.15.139:0/4207913916 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f18cc104660 0x7f18cc1998e0 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f18b4006010 tx=0x7f18b4004ff0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:23:37.825 INFO:teuthology.orchestra.run.smithi139.stderr:2023-07-14T19:23:37.820+0000 7f18c37fe700 1 -- 172.21.15.139:0/4207913916 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f18b401d070 con 0x7f18cc104660 2023-07-14T19:23:37.825 INFO:teuthology.orchestra.run.smithi139.stderr:2023-07-14T19:23:37.820+0000 7f18c37fe700 1 -- 172.21.15.139:0/4207913916 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f18b400bc10 con 0x7f18cc104660 2023-07-14T19:23:37.825 INFO:teuthology.orchestra.run.smithi139.stderr:2023-07-14T19:23:37.820+0000 7f18c37fe700 1 -- 172.21.15.139:0/4207913916 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f18b4003910 con 0x7f18cc104660 2023-07-14T19:23:37.826 INFO:teuthology.orchestra.run.smithi139.stderr:2023-07-14T19:23:37.820+0000 7f18d0890700 1 -- 172.21.15.139:0/4207913916 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f18cc19a080 con 0x7f18cc104660 2023-07-14T19:23:37.826 INFO:teuthology.orchestra.run.smithi139.stderr:2023-07-14T19:23:37.820+0000 7f18d0890700 1 -- 172.21.15.139:0/4207913916 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f18cc1938d0 con 0x7f18cc104660 2023-07-14T19:23:37.826 INFO:teuthology.orchestra.run.smithi139.stderr:2023-07-14T19:23:37.821+0000 7f18c37fe700 1 -- 172.21.15.139:0/4207913916 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 13) v1 ==== 45539+0+0 (secure 0 0 0) 0x7f18b400bd80 con 0x7f18cc104660 2023-07-14T19:23:37.826 INFO:teuthology.orchestra.run.smithi139.stderr:2023-07-14T19:23:37.821+0000 7f18d0890700 1 -- 172.21.15.139:0/4207913916 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f18cc061c70 con 0x7f18cc104660 2023-07-14T19:23:37.826 INFO:teuthology.orchestra.run.smithi139.stderr:2023-07-14T19:23:37.821+0000 7f18c37fe700 1 --2- 172.21.15.139:0/4207913916 >> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] conn(0x7f18b80387b0 0x7f18b803ac60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:23:37.826 INFO:teuthology.orchestra.run.smithi139.stderr:2023-07-14T19:23:37.821+0000 7f18c37fe700 1 -- 172.21.15.139:0/4207913916 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f18b404c540 con 0x7f18cc104660 2023-07-14T19:23:37.826 INFO:teuthology.orchestra.run.smithi139.stderr:2023-07-14T19:23:37.822+0000 7f18c9d9b700 1 --2- 172.21.15.139:0/4207913916 >> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] conn(0x7f18b80387b0 0x7f18b803ac60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:23:37.827 INFO:teuthology.orchestra.run.smithi139.stderr:2023-07-14T19:23:37.822+0000 7f18c9d9b700 1 --2- 172.21.15.139:0/4207913916 >> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] conn(0x7f18b80387b0 0x7f18b803ac60 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f18bc006fd0 tx=0x7f18bc006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:23:37.828 INFO:teuthology.orchestra.run.smithi139.stderr:2023-07-14T19:23:37.827+0000 7f18c37fe700 1 -- 172.21.15.139:0/4207913916 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f18b4029360 con 0x7f18cc104660 2023-07-14T19:23:38.125 INFO:teuthology.orchestra.run.smithi139.stderr:2023-07-14T19:23:38.121+0000 7f18d0890700 1 -- 172.21.15.139:0/4207913916 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f18cc1942f0 con 0x7f18cc104660 2023-07-14T19:23:38.125 INFO:teuthology.orchestra.run.smithi139.stderr:2023-07-14T19:23:38.122+0000 7f18c37fe700 1 -- 172.21.15.139:0/4207913916 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+736 (secure 0 0 0) 0x7f18b4026070 con 0x7f18cc104660 2023-07-14T19:23:38.125 INFO:teuthology.orchestra.run.smithi139.stdout: 2023-07-14T19:23:38.126 INFO:teuthology.orchestra.run.smithi139.stdout:{"epoch":1,"fsid":"9adea648-227b-11ee-9b33-001a4aab830c","modified":"2023-07-14T19:22:04.850090Z","created":"2023-07-14T19:22:04.850090Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:3300","nonce":0},{"type":"v1","addr":"172.21.15.98:6789","nonce":0}]},"addr":"172.21.15.98:6789/0","public_addr":"172.21.15.98:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-07-14T19:23:38.126 INFO:teuthology.orchestra.run.smithi139.stderr:2023-07-14T19:23:38.124+0000 7f18d0890700 1 -- 172.21.15.139:0/4207913916 >> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] conn(0x7f18b80387b0 msgr2=0x7f18b803ac60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:23:38.126 INFO:teuthology.orchestra.run.smithi139.stderr:2023-07-14T19:23:38.124+0000 7f18d0890700 1 --2- 172.21.15.139:0/4207913916 >> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] conn(0x7f18b80387b0 0x7f18b803ac60 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f18bc006fd0 tx=0x7f18bc006e40 comp rx=0 tx=0).stop 2023-07-14T19:23:38.127 INFO:teuthology.orchestra.run.smithi139.stderr:2023-07-14T19:23:38.124+0000 7f18d0890700 1 -- 172.21.15.139:0/4207913916 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f18cc104660 msgr2=0x7f18cc1998e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:23:38.127 INFO:teuthology.orchestra.run.smithi139.stderr:2023-07-14T19:23:38.124+0000 7f18d0890700 1 --2- 172.21.15.139:0/4207913916 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f18cc104660 0x7f18cc1998e0 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f18b4006010 tx=0x7f18b4004ff0 comp rx=0 tx=0).stop 2023-07-14T19:23:38.127 INFO:teuthology.orchestra.run.smithi139.stderr:2023-07-14T19:23:38.124+0000 7f18d0890700 1 -- 172.21.15.139:0/4207913916 shutdown_connections 2023-07-14T19:23:38.127 INFO:teuthology.orchestra.run.smithi139.stderr:2023-07-14T19:23:38.124+0000 7f18d0890700 1 --2- 172.21.15.139:0/4207913916 >> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] conn(0x7f18b80387b0 0x7f18b803ac60 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:23:38.127 INFO:teuthology.orchestra.run.smithi139.stderr:2023-07-14T19:23:38.124+0000 7f18d0890700 1 --2- 172.21.15.139:0/4207913916 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f18cc104660 0x7f18cc1998e0 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:23:38.127 INFO:teuthology.orchestra.run.smithi139.stderr:2023-07-14T19:23:38.124+0000 7f18d0890700 1 -- 172.21.15.139:0/4207913916 >> 172.21.15.139:0/4207913916 conn(0x7f18cc0743e0 msgr2=0x7f18cc105c00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:23:38.127 INFO:teuthology.orchestra.run.smithi139.stderr:2023-07-14T19:23:38.124+0000 7f18d0890700 1 -- 172.21.15.139:0/4207913916 shutdown_connections 2023-07-14T19:23:38.128 INFO:teuthology.orchestra.run.smithi139.stderr:2023-07-14T19:23:38.125+0000 7f18d0890700 1 -- 172.21.15.139:0/4207913916 wait complete. 2023-07-14T19:23:38.128 INFO:teuthology.orchestra.run.smithi139.stderr:dumped monmap epoch 1 2023-07-14T19:23:38.619 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:23:38 smithi139 podman[129251]: 2023-07-14T19:23:38.971 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:23:38 smithi139 ceph-mon[129293]: set uid:gid to 167:167 (ceph:ceph) 2023-07-14T19:23:38.971 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:23:38 smithi139 ceph-mon[129293]: ceph version 18.0.0-4940-g81c9387d (81c9387d2e7576da065747ec9c16d1c7d69016ec) reef (dev), process ceph-mon, pid 7 2023-07-14T19:23:38.971 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:23:38 smithi139 ceph-mon[129293]: pidfile_write: ignore empty --pid-file 2023-07-14T19:23:38.971 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:23:38 smithi139 ceph-mon[129293]: load: jerasure load: lrc 2023-07-14T19:23:38.971 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:23:38 smithi139 ceph-mon[129293]: rocksdb: RocksDB version: 7.9.2 2023-07-14T19:23:38.972 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:23:38 smithi139 ceph-mon[129293]: rocksdb: Git sha 0 2023-07-14T19:23:38.972 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:23:38 smithi139 ceph-mon[129293]: rocksdb: Compile date 2023-07-14 14:45:42 2023-07-14T19:23:38.972 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:23:38 smithi139 ceph-mon[129293]: rocksdb: DB SUMMARY 2023-07-14T19:23:38.972 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:23:38 smithi139 ceph-mon[129293]: rocksdb: DB Session ID: PLMSCVIVS6IAT0IZT1Y8 2023-07-14T19:23:38.972 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:23:38 smithi139 ceph-mon[129293]: rocksdb: CURRENT file: CURRENT 2023-07-14T19:23:38.972 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:23:38 smithi139 ceph-mon[129293]: rocksdb: IDENTITY file: IDENTITY 2023-07-14T19:23:38.972 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:23:38 smithi139 ceph-mon[129293]: rocksdb: MANIFEST file: MANIFEST-000005 size: 59 Bytes 2023-07-14T19:23:38.973 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:23:38 smithi139 ceph-mon[129293]: rocksdb: SST files in /var/lib/ceph/mon/ceph-c/store.db dir, Total Num: 0, files: 2023-07-14T19:23:38.973 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:23:38 smithi139 ceph-mon[129293]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-c/store.db: 000004.log size: 511 ; 2023-07-14T19:23:38.973 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:23:38 smithi139 ceph-mon[129293]: rocksdb: Options.error_if_exists: 0 2023-07-14T19:23:38.973 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:23:38 smithi139 ceph-mon[129293]: rocksdb: Options.create_if_missing: 0 2023-07-14T19:23:38.973 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:23:38 smithi139 ceph-mon[129293]: rocksdb: Options.paranoid_checks: 1 2023-07-14T19:23:38.973 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:23:38 smithi139 ceph-mon[129293]: rocksdb: Options.flush_verify_memtable_count: 1 2023-07-14T19:23:38.973 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:23:38 smithi139 ceph-mon[129293]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2023-07-14T19:23:38.973 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:23:38 smithi139 ceph-mon[129293]: rocksdb: Options.verify_sst_unique_id_in_manifest: 1 2023-07-14T19:23:38.974 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:23:38 smithi139 ceph-mon[129293]: rocksdb: Options.env: 0x559a14c3cc60 2023-07-14T19:23:38.974 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:23:38 smithi139 ceph-mon[129293]: rocksdb: Options.fs: PosixFileSystem 2023-07-14T19:23:38.974 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:23:38 smithi139 ceph-mon[129293]: rocksdb: Options.info_log: 0x559a157932a0 2023-07-14T19:23:38.974 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:23:38 smithi139 ceph-mon[129293]: rocksdb: Options.max_file_opening_threads: 16 2023-07-14T19:23:38.974 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:23:38 smithi139 ceph-mon[129293]: rocksdb: Options.statistics: (nil) 2023-07-14T19:23:38.974 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:23:38 smithi139 ceph-mon[129293]: rocksdb: Options.use_fsync: 0 2023-07-14T19:23:38.974 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:23:38 smithi139 ceph-mon[129293]: rocksdb: Options.max_log_file_size: 0 2023-07-14T19:23:38.975 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:23:38 smithi139 ceph-mon[129293]: rocksdb: Options.max_manifest_file_size: 1073741824 2023-07-14T19:23:38.975 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:23:38 smithi139 ceph-mon[129293]: rocksdb: Options.log_file_time_to_roll: 0 2023-07-14T19:23:38.975 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:23:38 smithi139 ceph-mon[129293]: rocksdb: Options.keep_log_file_num: 1000 2023-07-14T19:23:38.975 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:23:38 smithi139 ceph-mon[129293]: rocksdb: Options.recycle_log_file_num: 0 2023-07-14T19:23:38.975 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:23:38 smithi139 ceph-mon[129293]: rocksdb: Options.allow_fallocate: 1 2023-07-14T19:23:38.975 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:23:38 smithi139 ceph-mon[129293]: rocksdb: Options.allow_mmap_reads: 0 2023-07-14T19:23:38.975 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:23:38 smithi139 ceph-mon[129293]: rocksdb: Options.allow_mmap_writes: 0 2023-07-14T19:23:38.975 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:23:38 smithi139 ceph-mon[129293]: rocksdb: Options.use_direct_reads: 0 2023-07-14T19:23:38.976 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:23:38 smithi139 ceph-mon[129293]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2023-07-14T19:23:38.976 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:23:38 smithi139 ceph-mon[129293]: rocksdb: Options.create_missing_column_families: 0 2023-07-14T19:23:38.976 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:23:38 smithi139 ceph-mon[129293]: rocksdb: Options.db_log_dir: 2023-07-14T19:23:38.976 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:23:38 smithi139 ceph-mon[129293]: rocksdb: Options.wal_dir: 2023-07-14T19:23:38.976 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:23:38 smithi139 ceph-mon[129293]: rocksdb: Options.table_cache_numshardbits: 6 2023-07-14T19:23:38.977 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:23:38 smithi139 ceph-mon[129293]: rocksdb: Options.WAL_ttl_seconds: 0 2023-07-14T19:23:38.978 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:23:38 smithi139 ceph-mon[129293]: rocksdb: Options.WAL_size_limit_MB: 0 2023-07-14T19:23:38.978 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:23:38 smithi139 ceph-mon[129293]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2023-07-14T19:23:38.978 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:23:38 smithi139 ceph-mon[129293]: rocksdb: Options.manifest_preallocation_size: 4194304 2023-07-14T19:23:38.978 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:23:38 smithi139 ceph-mon[129293]: rocksdb: Options.is_fd_close_on_exec: 1 2023-07-14T19:23:38.978 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:23:38 smithi139 ceph-mon[129293]: rocksdb: Options.advise_random_on_open: 1 2023-07-14T19:23:38.978 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:23:38 smithi139 ceph-mon[129293]: rocksdb: Options.db_write_buffer_size: 0 2023-07-14T19:23:38.978 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:23:38 smithi139 ceph-mon[129293]: rocksdb: Options.write_buffer_manager: 0x559a156b43c0 2023-07-14T19:23:38.979 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:23:38 smithi139 ceph-mon[129293]: rocksdb: Options.access_hint_on_compaction_start: 1 2023-07-14T19:23:38.979 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:23:38 smithi139 ceph-mon[129293]: rocksdb: Options.random_access_max_buffer_size: 1048576 2023-07-14T19:23:38.979 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:23:38 smithi139 ceph-mon[129293]: rocksdb: Options.use_adaptive_mutex: 0 2023-07-14T19:23:38.979 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:23:38 smithi139 ceph-mon[129293]: rocksdb: Options.rate_limiter: (nil) 2023-07-14T19:23:38.979 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:23:38 smithi139 ceph-mon[129293]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2023-07-14T19:23:38.979 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:23:38 smithi139 ceph-mon[129293]: rocksdb: Options.wal_recovery_mode: 2 2023-07-14T19:23:38.979 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:23:38 smithi139 ceph-mon[129293]: rocksdb: Options.enable_thread_tracking: 0 2023-07-14T19:23:38.979 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:23:38 smithi139 ceph-mon[129293]: rocksdb: Options.enable_pipelined_write: 0 2023-07-14T19:23:38.980 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:23:38 smithi139 ceph-mon[129293]: rocksdb: Options.unordered_write: 0 2023-07-14T19:23:38.980 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:23:38 smithi139 ceph-mon[129293]: rocksdb: Options.allow_concurrent_memtable_write: 1 2023-07-14T19:23:38.980 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:23:38 smithi139 ceph-mon[129293]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2023-07-14T19:23:38.980 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:23:38 smithi139 ceph-mon[129293]: rocksdb: Options.write_thread_max_yield_usec: 100 2023-07-14T19:23:38.980 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:23:38 smithi139 ceph-mon[129293]: rocksdb: Options.write_thread_slow_yield_usec: 3 2023-07-14T19:23:38.980 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:23:38 smithi139 ceph-mon[129293]: rocksdb: Options.row_cache: None 2023-07-14T19:23:38.980 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:23:38 smithi139 ceph-mon[129293]: rocksdb: Options.wal_filter: None 2023-07-14T19:23:38.981 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:23:38 smithi139 ceph-mon[129293]: rocksdb: Options.avoid_flush_during_recovery: 0 2023-07-14T19:23:38.981 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:23:38 smithi139 ceph-mon[129293]: rocksdb: Options.allow_ingest_behind: 0 2023-07-14T19:23:38.981 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:23:38 smithi139 ceph-mon[129293]: rocksdb: Options.two_write_queues: 0 2023-07-14T19:23:38.981 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:23:38 smithi139 ceph-mon[129293]: rocksdb: Options.manual_wal_flush: 0 2023-07-14T19:23:38.981 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:23:38 smithi139 ceph-mon[129293]: rocksdb: Options.wal_compression: 0 2023-07-14T19:23:38.981 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:23:38 smithi139 ceph-mon[129293]: rocksdb: Options.atomic_flush: 0 2023-07-14T19:23:38.981 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:23:38 smithi139 ceph-mon[129293]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2023-07-14T19:23:38.981 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:23:38 smithi139 ceph-mon[129293]: rocksdb: Options.persist_stats_to_disk: 0 2023-07-14T19:23:38.982 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:23:38 smithi139 ceph-mon[129293]: rocksdb: Options.write_dbid_to_manifest: 0 2023-07-14T19:23:38.982 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:23:38 smithi139 ceph-mon[129293]: rocksdb: Options.log_readahead_size: 0 2023-07-14T19:23:38.982 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:23:38 smithi139 ceph-mon[129293]: rocksdb: Options.file_checksum_gen_factory: Unknown 2023-07-14T19:23:38.982 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:23:38 smithi139 ceph-mon[129293]: rocksdb: Options.best_efforts_recovery: 0 2023-07-14T19:23:38.982 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:23:38 smithi139 ceph-mon[129293]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2023-07-14T19:23:38.982 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:23:38 smithi139 ceph-mon[129293]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2023-07-14T19:23:38.982 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:23:38 smithi139 ceph-mon[129293]: rocksdb: Options.allow_data_in_errors: 0 2023-07-14T19:23:38.982 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:23:38 smithi139 ceph-mon[129293]: rocksdb: Options.db_host_id: __hostname__ 2023-07-14T19:23:38.983 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:23:38 smithi139 ceph-mon[129293]: rocksdb: Options.enforce_single_del_contracts: true 2023-07-14T19:23:38.983 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:23:38 smithi139 ceph-mon[129293]: rocksdb: Options.max_background_jobs: 2 2023-07-14T19:23:38.983 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:23:38 smithi139 ceph-mon[129293]: rocksdb: Options.max_background_compactions: -1 2023-07-14T19:23:38.983 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:23:38 smithi139 ceph-mon[129293]: rocksdb: Options.max_subcompactions: 1 2023-07-14T19:23:38.983 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:23:38 smithi139 ceph-mon[129293]: rocksdb: Options.avoid_flush_during_shutdown: 0 2023-07-14T19:23:38.983 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:23:38 smithi139 ceph-mon[129293]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2023-07-14T19:23:38.983 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:23:38 smithi139 ceph-mon[129293]: rocksdb: Options.delayed_write_rate : 16777216 2023-07-14T19:23:38.984 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:23:38 smithi139 ceph-mon[129293]: rocksdb: Options.max_total_wal_size: 0 2023-07-14T19:23:38.984 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:23:38 smithi139 ceph-mon[129293]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2023-07-14T19:23:38.984 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:23:38 smithi139 ceph-mon[129293]: rocksdb: Options.stats_dump_period_sec: 600 2023-07-14T19:23:38.984 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:23:38 smithi139 ceph-mon[129293]: rocksdb: Options.stats_persist_period_sec: 600 2023-07-14T19:23:38.984 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:23:38 smithi139 ceph-mon[129293]: rocksdb: Options.stats_history_buffer_size: 1048576 2023-07-14T19:23:38.984 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:23:38 smithi139 ceph-mon[129293]: rocksdb: Options.max_open_files: -1 2023-07-14T19:23:38.984 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:23:38 smithi139 ceph-mon[129293]: rocksdb: Options.bytes_per_sync: 0 2023-07-14T19:23:38.984 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:23:38 smithi139 ceph-mon[129293]: rocksdb: Options.wal_bytes_per_sync: 0 2023-07-14T19:23:38.985 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:23:38 smithi139 ceph-mon[129293]: rocksdb: Options.strict_bytes_per_sync: 0 2023-07-14T19:23:38.985 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:23:38 smithi139 ceph-mon[129293]: rocksdb: Options.compaction_readahead_size: 0 2023-07-14T19:23:38.985 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:23:38 smithi139 ceph-mon[129293]: rocksdb: Options.max_background_flushes: -1 2023-07-14T19:23:38.985 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:23:38 smithi139 ceph-mon[129293]: rocksdb: Compression algorithms supported: 2023-07-14T19:23:38.985 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:23:38 smithi139 ceph-mon[129293]: rocksdb: kZSTDNotFinalCompression supported: 0 2023-07-14T19:23:38.985 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:23:38 smithi139 ceph-mon[129293]: rocksdb: kZSTD supported: 0 2023-07-14T19:23:38.985 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:23:38 smithi139 ceph-mon[129293]: rocksdb: kXpressCompression supported: 0 2023-07-14T19:23:38.986 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:23:38 smithi139 ceph-mon[129293]: rocksdb: kLZ4HCCompression supported: 1 2023-07-14T19:23:38.986 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:23:38 smithi139 ceph-mon[129293]: rocksdb: kZlibCompression supported: 1 2023-07-14T19:23:38.986 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:23:38 smithi139 bash[129251]: fa3a6a6f918f98e276e4d7a3734ace2b8f6bb67f424ed1cfba1e7566512da72c 2023-07-14T19:23:38.986 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:23:38 smithi139 ceph-mon[129293]: rocksdb: kSnappyCompression supported: 1 2023-07-14T19:23:38.986 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:23:38 smithi139 ceph-mon[129293]: rocksdb: kLZ4Compression supported: 1 2023-07-14T19:23:38.986 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:23:38 smithi139 ceph-mon[129293]: rocksdb: kBZip2Compression supported: 0 2023-07-14T19:23:38.986 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:23:38 smithi139 ceph-mon[129293]: rocksdb: Fast CRC32 supported: Supported on x86 2023-07-14T19:23:38.986 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:23:38 smithi139 ceph-mon[129293]: rocksdb: DMutex implementation: pthread_mutex_t 2023-07-14T19:23:38.987 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:23:38 smithi139 ceph-mon[129293]: rocksdb: [db/version_set.cc:5527] Recovering from manifest file: /var/lib/ceph/mon/ceph-c/store.db/MANIFEST-000005 2023-07-14T19:23:38.987 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:23:38 smithi139 ceph-mon[129293]: rocksdb: [db/column_family.cc:630] --------------- Options for column family [default]: 2023-07-14T19:23:38.987 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:23:38 smithi139 ceph-mon[129293]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2023-07-14T19:23:38.987 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:23:38 smithi139 ceph-mon[129293]: rocksdb: Options.merge_operator: 2023-07-14T19:23:38.987 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:23:38 smithi139 ceph-mon[129293]: rocksdb: Options.compaction_filter: None 2023-07-14T19:23:38.987 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:23:38 smithi139 ceph-mon[129293]: rocksdb: Options.compaction_filter_factory: None 2023-07-14T19:23:38.987 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:23:38 smithi139 ceph-mon[129293]: rocksdb: Options.sst_partitioner_factory: None 2023-07-14T19:23:38.987 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:23:38 smithi139 ceph-mon[129293]: rocksdb: Options.memtable_factory: SkipListFactory 2023-07-14T19:23:38.988 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:23:38 smithi139 ceph-mon[129293]: rocksdb: Options.table_factory: BlockBasedTable 2023-07-14T19:23:38.988 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:23:38 smithi139 ceph-mon[129293]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x559a157933a0) 2023-07-14T19:23:38.988 INFO:journalctl@ceph.mon.c.smithi139.stdout: cache_index_and_filter_blocks: 1 2023-07-14T19:23:38.988 INFO:journalctl@ceph.mon.c.smithi139.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2023-07-14T19:23:38.988 INFO:journalctl@ceph.mon.c.smithi139.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2023-07-14T19:23:38.988 INFO:journalctl@ceph.mon.c.smithi139.stdout: pin_top_level_index_and_filter: 1 2023-07-14T19:23:38.988 INFO:journalctl@ceph.mon.c.smithi139.stdout: index_type: 0 2023-07-14T19:23:38.989 INFO:journalctl@ceph.mon.c.smithi139.stdout: data_block_index_type: 0 2023-07-14T19:23:38.989 INFO:journalctl@ceph.mon.c.smithi139.stdout: index_shortening: 1 2023-07-14T19:23:38.989 INFO:journalctl@ceph.mon.c.smithi139.stdout: data_block_hash_table_util_ratio: 0.750000 2023-07-14T19:23:38.989 INFO:journalctl@ceph.mon.c.smithi139.stdout: checksum: 4 2023-07-14T19:23:38.989 INFO:journalctl@ceph.mon.c.smithi139.stdout: no_block_cache: 0 2023-07-14T19:23:38.989 INFO:journalctl@ceph.mon.c.smithi139.stdout: block_cache: 0x559a157531f0 2023-07-14T19:23:38.989 INFO:journalctl@ceph.mon.c.smithi139.stdout: block_cache_name: BinnedLRUCache 2023-07-14T19:23:38.989 INFO:journalctl@ceph.mon.c.smithi139.stdout: block_cache_options: 2023-07-14T19:23:38.990 INFO:journalctl@ceph.mon.c.smithi139.stdout: capacity : 536870912 2023-07-14T19:23:38.990 INFO:journalctl@ceph.mon.c.smithi139.stdout: num_shard_bits : 4 2023-07-14T19:23:38.990 INFO:journalctl@ceph.mon.c.smithi139.stdout: strict_capacity_limit : 0 2023-07-14T19:23:38.990 INFO:journalctl@ceph.mon.c.smithi139.stdout: high_pri_pool_ratio: 0.000 2023-07-14T19:23:38.990 INFO:journalctl@ceph.mon.c.smithi139.stdout: block_cache_compressed: (nil) 2023-07-14T19:23:38.990 INFO:journalctl@ceph.mon.c.smithi139.stdout: persistent_cache: (nil) 2023-07-14T19:23:38.990 INFO:journalctl@ceph.mon.c.smithi139.stdout: block_size: 4096 2023-07-14T19:23:38.991 INFO:journalctl@ceph.mon.c.smithi139.stdout: block_size_deviation: 10 2023-07-14T19:23:38.991 INFO:journalctl@ceph.mon.c.smithi139.stdout: block_restart_interval: 16 2023-07-14T19:23:38.991 INFO:journalctl@ceph.mon.c.smithi139.stdout: index_block_restart_interval: 1 2023-07-14T19:23:38.991 INFO:journalctl@ceph.mon.c.smithi139.stdout: metadata_block_size: 4096 2023-07-14T19:23:38.991 INFO:journalctl@ceph.mon.c.smithi139.stdout: partition_filters: 0 2023-07-14T19:23:38.991 INFO:journalctl@ceph.mon.c.smithi139.stdout: use_delta_encoding: 1 2023-07-14T19:23:38.991 INFO:journalctl@ceph.mon.c.smithi139.stdout: filter_policy: bloomfilter 2023-07-14T19:23:38.991 INFO:journalctl@ceph.mon.c.smithi139.stdout: whole_key_filtering: 1 2023-07-14T19:23:38.992 INFO:journalctl@ceph.mon.c.smithi139.stdout: verify_compression: 0 2023-07-14T19:23:38.992 INFO:journalctl@ceph.mon.c.smithi139.stdout: read_amp_bytes_per_bit: 0 2023-07-14T19:23:38.992 INFO:journalctl@ceph.mon.c.smithi139.stdout: format_version: 5 2023-07-14T19:23:38.992 INFO:journalctl@ceph.mon.c.smithi139.stdout: enable_index_compression: 1 2023-07-14T19:23:38.992 INFO:journalctl@ceph.mon.c.smithi139.stdout: block_align: 0 2023-07-14T19:23:38.992 INFO:journalctl@ceph.mon.c.smithi139.stdout: max_auto_readahead_size: 262144 2023-07-14T19:23:38.992 INFO:journalctl@ceph.mon.c.smithi139.stdout: prepopulate_block_cache: 0 2023-07-14T19:23:38.992 INFO:journalctl@ceph.mon.c.smithi139.stdout: initial_auto_readahead_size: 8192 2023-07-14T19:23:38.993 INFO:journalctl@ceph.mon.c.smithi139.stdout: num_file_reads_for_auto_readahead: 2 2023-07-14T19:23:38.993 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:23:38 smithi139 ceph-mon[129293]: rocksdb: Options.write_buffer_size: 33554432 2023-07-14T19:23:38.993 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:23:38 smithi139 ceph-mon[129293]: rocksdb: Options.max_write_buffer_number: 2 2023-07-14T19:23:38.993 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:23:38 smithi139 ceph-mon[129293]: rocksdb: Options.compression: NoCompression 2023-07-14T19:23:38.993 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:23:38 smithi139 ceph-mon[129293]: rocksdb: Options.bottommost_compression: Disabled 2023-07-14T19:23:38.993 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:23:38 smithi139 ceph-mon[129293]: rocksdb: Options.prefix_extractor: nullptr 2023-07-14T19:23:38.993 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:23:38 smithi139 ceph-mon[129293]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2023-07-14T19:23:38.994 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:23:38 smithi139 ceph-mon[129293]: rocksdb: Options.num_levels: 7 2023-07-14T19:23:38.994 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:23:38 smithi139 ceph-mon[129293]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2023-07-14T19:23:38.994 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:23:38 smithi139 ceph-mon[129293]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2023-07-14T19:23:38.994 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:23:38 smithi139 ceph-mon[129293]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2023-07-14T19:23:38.994 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:23:38 smithi139 ceph-mon[129293]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2023-07-14T19:23:38.994 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:23:38 smithi139 ceph-mon[129293]: rocksdb: Options.bottommost_compression_opts.level: 32767 2023-07-14T19:23:38.994 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:23:38 smithi139 ceph-mon[129293]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2023-07-14T19:23:38.994 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:23:38 smithi139 ceph-mon[129293]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2023-07-14T19:23:38.995 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:23:38 smithi139 ceph-mon[129293]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2023-07-14T19:23:38.995 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:23:38 smithi139 ceph-mon[129293]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2023-07-14T19:23:38.995 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:23:38 smithi139 ceph-mon[129293]: rocksdb: Options.bottommost_compression_opts.enabled: false 2023-07-14T19:23:38.995 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:23:38 smithi139 ceph-mon[129293]: rocksdb: Options.bottommost_compression_opts.max_dict_buffer_bytes: 0 2023-07-14T19:23:38.995 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:23:38 smithi139 ceph-mon[129293]: rocksdb: Options.bottommost_compression_opts.use_zstd_dict_trainer: true 2023-07-14T19:23:38.995 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:23:38 smithi139 ceph-mon[129293]: rocksdb: Options.compression_opts.window_bits: -14 2023-07-14T19:23:38.995 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:23:38 smithi139 ceph-mon[129293]: rocksdb: Options.compression_opts.level: 32767 2023-07-14T19:23:38.995 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:23:38 smithi139 ceph-mon[129293]: rocksdb: Options.compression_opts.strategy: 0 2023-07-14T19:23:38.996 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:23:38 smithi139 ceph-mon[129293]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2023-07-14T19:23:38.996 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:23:38 smithi139 ceph-mon[129293]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2023-07-14T19:23:38.996 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:23:38 smithi139 ceph-mon[129293]: rocksdb: Options.compression_opts.use_zstd_dict_trainer: true 2023-07-14T19:23:38.996 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:23:38 smithi139 ceph-mon[129293]: rocksdb: Options.compression_opts.parallel_threads: 1 2023-07-14T19:23:38.996 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:23:38 smithi139 ceph-mon[129293]: rocksdb: Options.compression_opts.enabled: false 2023-07-14T19:23:38.996 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:23:38 smithi139 ceph-mon[129293]: rocksdb: Options.compression_opts.max_dict_buffer_bytes: 0 2023-07-14T19:23:38.996 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:23:38 smithi139 ceph-mon[129293]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2023-07-14T19:23:38.996 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:23:38 smithi139 ceph-mon[129293]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2023-07-14T19:23:38.997 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:23:38 smithi139 ceph-mon[129293]: rocksdb: Options.level0_stop_writes_trigger: 36 2023-07-14T19:23:38.997 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:23:38 smithi139 ceph-mon[129293]: rocksdb: Options.target_file_size_base: 67108864 2023-07-14T19:23:38.997 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:23:38 smithi139 ceph-mon[129293]: rocksdb: Options.target_file_size_multiplier: 1 2023-07-14T19:23:38.997 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:23:38 smithi139 ceph-mon[129293]: rocksdb: Options.max_bytes_for_level_base: 268435456 2023-07-14T19:23:38.997 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:23:38 smithi139 ceph-mon[129293]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2023-07-14T19:23:38.997 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:23:38 smithi139 ceph-mon[129293]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2023-07-14T19:23:38.997 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:23:38 smithi139 ceph-mon[129293]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2023-07-14T19:23:38.998 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:23:38 smithi139 ceph-mon[129293]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2023-07-14T19:23:38.998 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:23:38 smithi139 ceph-mon[129293]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2023-07-14T19:23:38.998 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:23:38 smithi139 ceph-mon[129293]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2023-07-14T19:23:38.998 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:23:38 smithi139 ceph-mon[129293]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2023-07-14T19:23:38.998 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:23:38 smithi139 ceph-mon[129293]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2023-07-14T19:23:38.998 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:23:38 smithi139 ceph-mon[129293]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2023-07-14T19:23:38.998 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:23:38 smithi139 ceph-mon[129293]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2023-07-14T19:23:38.998 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:23:38 smithi139 ceph-mon[129293]: rocksdb: Options.max_compaction_bytes: 1677721600 2023-07-14T19:23:38.999 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:23:38 smithi139 ceph-mon[129293]: rocksdb: Options.ignore_max_compaction_bytes_for_input: true 2023-07-14T19:23:38.999 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:23:38 smithi139 ceph-mon[129293]: rocksdb: Options.arena_block_size: 1048576 2023-07-14T19:23:38.999 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:23:38 smithi139 ceph-mon[129293]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2023-07-14T19:23:38.999 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:23:38 smithi139 ceph-mon[129293]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2023-07-14T19:23:38.999 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:23:38 smithi139 ceph-mon[129293]: rocksdb: Options.disable_auto_compactions: 0 2023-07-14T19:23:38.999 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:23:38 smithi139 ceph-mon[129293]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2023-07-14T19:23:38.999 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:23:38 smithi139 ceph-mon[129293]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2023-07-14T19:23:38.999 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:23:38 smithi139 ceph-mon[129293]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2023-07-14T19:23:39.000 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:23:38 smithi139 ceph-mon[129293]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2023-07-14T19:23:39.000 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:23:38 smithi139 ceph-mon[129293]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2023-07-14T19:23:39.000 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:23:38 smithi139 ceph-mon[129293]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2023-07-14T19:23:39.000 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:23:38 smithi139 ceph-mon[129293]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2023-07-14T19:23:39.000 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:23:38 smithi139 ceph-mon[129293]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2023-07-14T19:23:39.000 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:23:38 smithi139 ceph-mon[129293]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2023-07-14T19:23:39.000 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:23:38 smithi139 ceph-mon[129293]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2023-07-14T19:23:39.001 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:23:38 smithi139 ceph-mon[129293]: rocksdb: Options.table_properties_collectors: 2023-07-14T19:23:39.001 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:23:38 smithi139 ceph-mon[129293]: rocksdb: Options.inplace_update_support: 0 2023-07-14T19:23:39.001 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:23:38 smithi139 ceph-mon[129293]: rocksdb: Options.inplace_update_num_locks: 10000 2023-07-14T19:23:39.001 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:23:38 smithi139 ceph-mon[129293]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2023-07-14T19:23:39.001 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:23:38 smithi139 ceph-mon[129293]: rocksdb: Options.memtable_whole_key_filtering: 0 2023-07-14T19:23:39.001 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:23:38 smithi139 ceph-mon[129293]: rocksdb: Options.memtable_huge_page_size: 0 2023-07-14T19:23:39.001 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:23:38 smithi139 ceph-mon[129293]: rocksdb: Options.bloom_locality: 0 2023-07-14T19:23:39.001 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:23:38 smithi139 ceph-mon[129293]: rocksdb: Options.max_successive_merges: 0 2023-07-14T19:23:39.002 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:23:38 smithi139 ceph-mon[129293]: rocksdb: Options.optimize_filters_for_hits: 0 2023-07-14T19:23:39.002 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:23:38 smithi139 ceph-mon[129293]: rocksdb: Options.paranoid_file_checks: 0 2023-07-14T19:23:39.002 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:23:38 smithi139 ceph-mon[129293]: rocksdb: Options.force_consistency_checks: 1 2023-07-14T19:23:39.002 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:23:38 smithi139 ceph-mon[129293]: rocksdb: Options.report_bg_io_stats: 0 2023-07-14T19:23:39.002 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:23:38 smithi139 ceph-mon[129293]: rocksdb: Options.ttl: 2592000 2023-07-14T19:23:39.002 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:23:38 smithi139 ceph-mon[129293]: rocksdb: Options.periodic_compaction_seconds: 0 2023-07-14T19:23:39.002 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:23:38 smithi139 ceph-mon[129293]: rocksdb: Options.preclude_last_level_data_seconds: 0 2023-07-14T19:23:39.002 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:23:38 smithi139 ceph-mon[129293]: rocksdb: Options.preserve_internal_time_seconds: 0 2023-07-14T19:23:39.003 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:23:38 smithi139 ceph-mon[129293]: rocksdb: Options.enable_blob_files: false 2023-07-14T19:23:39.003 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:23:38 smithi139 ceph-mon[129293]: rocksdb: Options.min_blob_size: 0 2023-07-14T19:23:39.003 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:23:38 smithi139 ceph-mon[129293]: rocksdb: Options.blob_file_size: 268435456 2023-07-14T19:23:39.003 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:23:38 smithi139 ceph-mon[129293]: rocksdb: Options.blob_compression_type: NoCompression 2023-07-14T19:23:39.003 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:23:38 smithi139 ceph-mon[129293]: rocksdb: Options.enable_blob_garbage_collection: false 2023-07-14T19:23:39.003 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:23:38 smithi139 ceph-mon[129293]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2023-07-14T19:23:39.003 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:23:38 smithi139 ceph-mon[129293]: rocksdb: Options.blob_garbage_collection_force_threshold: 1.000000 2023-07-14T19:23:39.004 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:23:38 smithi139 ceph-mon[129293]: rocksdb: Options.blob_compaction_readahead_size: 0 2023-07-14T19:23:39.004 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:23:38 smithi139 ceph-mon[129293]: rocksdb: Options.blob_file_starting_level: 0 2023-07-14T19:23:39.004 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:23:38 smithi139 ceph-mon[129293]: rocksdb: Options.experimental_mempurge_threshold: 0.000000 2023-07-14T19:23:39.004 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:23:38 smithi139 ceph-mon[129293]: rocksdb: [db/version_set.cc:5566] Recovered from manifest file:/var/lib/ceph/mon/ceph-c/store.db/MANIFEST-000005 succeeded,manifest_file_number is 5, next_file_number is 7, last_sequence is 0, log_number is 0,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2023-07-14T19:23:39.004 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:23:38 smithi139 ceph-mon[129293]: rocksdb: [db/version_set.cc:5581] Column family [default] (ID 0), log number is 0 2023-07-14T19:23:39.004 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:23:38 smithi139 ceph-mon[129293]: rocksdb: [db/db_impl/db_impl_open.cc:539] DB ID: 8d8ad61b-09e0-4c92-b40c-8787a6861e3f 2023-07-14T19:23:39.004 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:23:38 smithi139 ceph-mon[129293]: rocksdb: EVENT_LOG_v1 {"time_micros": 1689362618780654, "job": 1, "event": "recovery_started", "wal_files": [4]} 2023-07-14T19:23:39.004 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:23:38 smithi139 ceph-mon[129293]: rocksdb: [db/db_impl/db_impl_open.cc:1043] Recovering log #4 mode 2 2023-07-14T19:23:39.005 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:23:38 smithi139 ceph-mon[129293]: rocksdb: EVENT_LOG_v1 {"time_micros": 1689362618781051, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 8, "file_size": 1622, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 1, "largest_seqno": 5, "table_properties": {"data_size": 523, "index_size": 31, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 69, "raw_key_size": 115, "raw_average_key_size": 23, "raw_value_size": 401, "raw_average_value_size": 80, "num_data_blocks": 1, "num_entries": 5, "num_filter_entries": 5, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1689362618, "oldest_key_time": 0, "file_creation_time": 0, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "8d8ad61b-09e0-4c92-b40c-8787a6861e3f", "db_session_id": "PLMSCVIVS6IAT0IZT1Y8", "orig_file_number": 8, "seqno_to_time_mapping": "N/A"}} 2023-07-14T19:23:39.005 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:23:38 smithi139 ceph-mon[129293]: rocksdb: EVENT_LOG_v1 {"time_micros": 1689362618781116, "job": 1, "event": "recovery_finished"} 2023-07-14T19:23:39.005 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:23:38 smithi139 ceph-mon[129293]: rocksdb: [db/version_set.cc:5047] Creating manifest 10 2023-07-14T19:23:39.005 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:23:38 smithi139 ceph-mon[129293]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-c/store.db/000004.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2023-07-14T19:23:39.005 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:23:38 smithi139 ceph-mon[129293]: rocksdb: [db/db_impl/db_impl_open.cc:1987] SstFileManager instance 0x559a157ec000 2023-07-14T19:23:39.005 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:23:38 smithi139 ceph-mon[129293]: rocksdb: DB pointer 0x559a157d8000 2023-07-14T19:23:39.005 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:23:38 smithi139 ceph-mon[129293]: rocksdb: [db/db_impl/db_impl.cc:1109] ------- DUMPING STATS ------- 2023-07-14T19:23:39.006 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:23:38 smithi139 ceph-mon[129293]: rocksdb: [db/db_impl/db_impl.cc:1111] 2023-07-14T19:23:39.006 INFO:journalctl@ceph.mon.c.smithi139.stdout: ** DB Stats ** 2023-07-14T19:23:39.006 INFO:journalctl@ceph.mon.c.smithi139.stdout: Uptime(secs): 0.0 total, 0.0 interval 2023-07-14T19:23:39.006 INFO:journalctl@ceph.mon.c.smithi139.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2023-07-14T19:23:39.006 INFO:journalctl@ceph.mon.c.smithi139.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2023-07-14T19:23:39.006 INFO:journalctl@ceph.mon.c.smithi139.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2023-07-14T19:23:39.006 INFO:journalctl@ceph.mon.c.smithi139.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2023-07-14T19:23:39.006 INFO:journalctl@ceph.mon.c.smithi139.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2023-07-14T19:23:39.007 INFO:journalctl@ceph.mon.c.smithi139.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2023-07-14T19:23:39.007 INFO:journalctl@ceph.mon.c.smithi139.stdout: 2023-07-14T19:23:39.007 INFO:journalctl@ceph.mon.c.smithi139.stdout: ** Compaction Stats [default] ** 2023-07-14T19:23:39.007 INFO:journalctl@ceph.mon.c.smithi139.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2023-07-14T19:23:39.007 INFO:journalctl@ceph.mon.c.smithi139.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 2023-07-14T19:23:39.007 INFO:journalctl@ceph.mon.c.smithi139.stdout: L0 1/0 1.58 KB 0.2 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 4.0 0.00 0.00 1 0.000 0 0 0.0 0.0 2023-07-14T19:23:39.007 INFO:journalctl@ceph.mon.c.smithi139.stdout: Sum 1/0 1.58 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 4.0 0.00 0.00 1 0.000 0 0 0.0 0.0 2023-07-14T19:23:39.008 INFO:journalctl@ceph.mon.c.smithi139.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 4.0 0.00 0.00 1 0.000 0 0 0.0 0.0 2023-07-14T19:23:39.008 INFO:journalctl@ceph.mon.c.smithi139.stdout: 2023-07-14T19:23:39.008 INFO:journalctl@ceph.mon.c.smithi139.stdout: ** Compaction Stats [default] ** 2023-07-14T19:23:39.008 INFO:journalctl@ceph.mon.c.smithi139.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2023-07-14T19:23:39.008 INFO:journalctl@ceph.mon.c.smithi139.stdout: --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2023-07-14T19:23:39.008 INFO:journalctl@ceph.mon.c.smithi139.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 4.0 0.00 0.00 1 0.000 0 0 0.0 0.0 2023-07-14T19:23:39.008 INFO:journalctl@ceph.mon.c.smithi139.stdout: 2023-07-14T19:23:39.008 INFO:journalctl@ceph.mon.c.smithi139.stdout: Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 2023-07-14T19:23:39.009 INFO:journalctl@ceph.mon.c.smithi139.stdout: 2023-07-14T19:23:39.009 INFO:journalctl@ceph.mon.c.smithi139.stdout: Uptime(secs): 0.0 total, 0.0 interval 2023-07-14T19:23:39.009 INFO:journalctl@ceph.mon.c.smithi139.stdout: Flush(GB): cumulative 0.000, interval 0.000 2023-07-14T19:23:39.009 INFO:journalctl@ceph.mon.c.smithi139.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2023-07-14T19:23:39.009 INFO:journalctl@ceph.mon.c.smithi139.stdout: AddFile(Total Files): cumulative 0, interval 0 2023-07-14T19:23:39.009 INFO:journalctl@ceph.mon.c.smithi139.stdout: AddFile(L0 Files): cumulative 0, interval 0 2023-07-14T19:23:39.009 INFO:journalctl@ceph.mon.c.smithi139.stdout: AddFile(Keys): cumulative 0, interval 0 2023-07-14T19:23:39.009 INFO:journalctl@ceph.mon.c.smithi139.stdout: Cumulative compaction: 0.00 GB write, 0.65 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2023-07-14T19:23:39.010 INFO:journalctl@ceph.mon.c.smithi139.stdout: Interval compaction: 0.00 GB write, 0.65 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2023-07-14T19:23:39.010 INFO:journalctl@ceph.mon.c.smithi139.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2023-07-14T19:23:39.010 INFO:journalctl@ceph.mon.c.smithi139.stdout: Block cache BinnedLRUCache@0x559a157531f0#7 capacity: 512.00 MB usage: 0.22 KB table_size: 0 occupancy: 18446744073709551615 collections: 1 last_copies: 0 last_secs: 7e-06 secs_since: 0 2023-07-14T19:23:39.010 INFO:journalctl@ceph.mon.c.smithi139.stdout: Block cache entry stats(count,size,portion): FilterBlock(1,0.11 KB,2.08616e-05%) IndexBlock(1,0.11 KB,2.08616e-05%) Misc(1,0.00 KB,0%) 2023-07-14T19:23:39.010 INFO:journalctl@ceph.mon.c.smithi139.stdout: 2023-07-14T19:23:39.010 INFO:journalctl@ceph.mon.c.smithi139.stdout: ** File Read Latency Histogram By Level [default] ** 2023-07-14T19:23:39.010 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:23:38 smithi139 ceph-mon[129293]: mon.c does not exist in monmap, will attempt to join an existing cluster 2023-07-14T19:23:39.011 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:23:38 smithi139 ceph-mon[129293]: using public_addr v2:172.21.15.139:0/0 -> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] 2023-07-14T19:23:39.011 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:23:38 smithi139 ceph-mon[129293]: starting mon.c rank -1 at public addrs [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] at bind addrs [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] mon_data /var/lib/ceph/mon/ceph-c fsid 9adea648-227b-11ee-9b33-001a4aab830c 2023-07-14T19:23:39.011 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:23:38 smithi139 ceph-mon[129293]: mon.c@-1(???) e0 preinit fsid 9adea648-227b-11ee-9b33-001a4aab830c 2023-07-14T19:23:39.011 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:23:38 smithi139 ceph-mon[129293]: mon.c@-1(synchronizing).mds e1 new map 2023-07-14T19:23:39.011 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:23:38 smithi139 ceph-mon[129293]: mon.c@-1(synchronizing).mds e1 print_map 2023-07-14T19:23:39.011 INFO:journalctl@ceph.mon.c.smithi139.stdout: e1 2023-07-14T19:23:39.011 INFO:journalctl@ceph.mon.c.smithi139.stdout: enable_multiple, ever_enabled_multiple: 1,1 2023-07-14T19:23:39.011 INFO:journalctl@ceph.mon.c.smithi139.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2023-07-14T19:23:39.012 INFO:journalctl@ceph.mon.c.smithi139.stdout: legacy client fscid: -1 2023-07-14T19:23:39.012 INFO:journalctl@ceph.mon.c.smithi139.stdout: 2023-07-14T19:23:39.012 INFO:journalctl@ceph.mon.c.smithi139.stdout: No filesystems configured 2023-07-14T19:23:39.012 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:23:38 smithi139 ceph-mon[129293]: mon.c@-1(synchronizing).osd e0 _set_cache_ratios kv ratio 0.25 inc ratio 0.375 full ratio 0.375 2023-07-14T19:23:39.012 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:23:38 smithi139 ceph-mon[129293]: mon.c@-1(synchronizing).osd e0 register_cache_with_pcm pcm target: 2147483648 pcm max: 1020054732 pcm min: 134217728 inc_osd_cache size: 1 2023-07-14T19:23:39.012 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:23:38 smithi139 ceph-mon[129293]: mon.c@-1(synchronizing).osd e1 e1: 0 total, 0 up, 0 in 2023-07-14T19:23:39.012 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:23:38 smithi139 ceph-mon[129293]: mon.c@-1(synchronizing).osd e2 e2: 0 total, 0 up, 0 in 2023-07-14T19:23:39.013 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:23:38 smithi139 ceph-mon[129293]: mon.c@-1(synchronizing).osd e3 e3: 0 total, 0 up, 0 in 2023-07-14T19:23:39.013 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:23:38 smithi139 ceph-mon[129293]: mon.c@-1(synchronizing).osd e4 e4: 0 total, 0 up, 0 in 2023-07-14T19:23:39.013 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:23:38 smithi139 ceph-mon[129293]: mon.c@-1(synchronizing).osd e4 crush map has features 3314932999778484224, adjusting msgr requires 2023-07-14T19:23:39.013 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:23:38 smithi139 ceph-mon[129293]: mon.c@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2023-07-14T19:23:39.013 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:23:38 smithi139 ceph-mon[129293]: mon.c@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2023-07-14T19:23:39.013 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:23:38 smithi139 ceph-mon[129293]: mon.c@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2023-07-14T19:23:39.013 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:23:38 smithi139 ceph-mon[129293]: from='client.? 172.21.15.98:0/1384810344' entity='client.admin' cmd=[{"prefix": "osd crush tunables", "profile": "default"}]: dispatch 2023-07-14T19:23:39.013 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:23:38 smithi139 ceph-mon[129293]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-14T19:23:39.014 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:23:38 smithi139 ceph-mon[129293]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' 2023-07-14T19:23:39.014 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:23:38 smithi139 ceph-mon[129293]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' 2023-07-14T19:23:39.014 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:23:38 smithi139 ceph-mon[129293]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' 2023-07-14T19:23:39.014 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:23:38 smithi139 ceph-mon[129293]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' 2023-07-14T19:23:39.014 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:23:38 smithi139 ceph-mon[129293]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi139", "name": "osd_memory_target"}]: dispatch 2023-07-14T19:23:39.014 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:23:38 smithi139 ceph-mon[129293]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T19:23:39.014 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:23:38 smithi139 ceph-mon[129293]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-14T19:23:39.014 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:23:38 smithi139 ceph-mon[129293]: Updating smithi139:/etc/ceph/ceph.conf 2023-07-14T19:23:39.015 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:23:38 smithi139 ceph-mon[129293]: Updating smithi139:/var/lib/ceph/9adea648-227b-11ee-9b33-001a4aab830c/config/ceph.conf 2023-07-14T19:23:39.015 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:23:38 smithi139 ceph-mon[129293]: from='client.? 172.21.15.98:0/1384810344' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2023-07-14T19:23:39.015 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:23:38 smithi139 ceph-mon[129293]: osdmap e4: 0 total, 0 up, 0 in 2023-07-14T19:23:39.015 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:23:38 smithi139 ceph-mon[129293]: Updating smithi139:/etc/ceph/ceph.client.admin.keyring 2023-07-14T19:23:39.015 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:23:38 smithi139 ceph-mon[129293]: Updating smithi139:/var/lib/ceph/9adea648-227b-11ee-9b33-001a4aab830c/config/ceph.client.admin.keyring 2023-07-14T19:23:39.015 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:23:38 smithi139 ceph-mon[129293]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' 2023-07-14T19:23:39.015 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:23:38 smithi139 ceph-mon[129293]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' 2023-07-14T19:23:39.016 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:23:38 smithi139 ceph-mon[129293]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' 2023-07-14T19:23:39.016 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:23:38 smithi139 ceph-mon[129293]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-14T19:23:39.016 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:23:38 smithi139 ceph-mon[129293]: from='client.14188 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi098:172.21.15.98=a;smithi101:172.21.15.101=b;smithi139:172.21.15.139=c", "target": ["mon-mgr", ""]}]: dispatch 2023-07-14T19:23:39.016 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:23:38 smithi139 ceph-mon[129293]: Saving service mon spec with placement smithi098:172.21.15.98=a;smithi101:172.21.15.101=b;smithi139:172.21.15.139=c;count:3 2023-07-14T19:23:39.016 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:23:38 smithi139 ceph-mon[129293]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' 2023-07-14T19:23:39.016 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:23:38 smithi139 ceph-mon[129293]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-14T19:23:39.016 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:23:38 smithi139 ceph-mon[129293]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T19:23:39.016 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:23:38 smithi139 ceph-mon[129293]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-14T19:23:39.017 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:23:38 smithi139 ceph-mon[129293]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' 2023-07-14T19:23:39.017 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:23:38 smithi139 ceph-mon[129293]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2023-07-14T19:23:39.017 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:23:38 smithi139 ceph-mon[129293]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T19:23:39.017 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:23:38 smithi139 ceph-mon[129293]: Deploying daemon mon.c on smithi139 2023-07-14T19:23:39.017 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:23:38 smithi139 ceph-mon[129293]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-14T19:23:39.017 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:23:38 smithi139 ceph-mon[129293]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-14T19:23:39.017 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:23:38 smithi139 ceph-mon[129293]: from='client.? 172.21.15.139:0/4207913916' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-07-14T19:23:39.018 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:23:38 smithi139 ceph-mon[129293]: mon.c@-1(synchronizing).paxosservice(auth 1..3) refresh upgraded, format 0 -> 3 2023-07-14T19:23:39.018 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:23:38 smithi139 ceph-mon[129293]: expand_channel_meta expand map: {default=false} 2023-07-14T19:23:39.018 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:23:38 smithi139 ceph-mon[129293]: expand_channel_meta from 'false' to 'false' 2023-07-14T19:23:39.018 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:23:38 smithi139 ceph-mon[129293]: expand_channel_meta expanded map: {default=false} 2023-07-14T19:23:39.018 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:23:38 smithi139 ceph-mon[129293]: expand_channel_meta expand map: {default=info} 2023-07-14T19:23:39.018 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:23:38 smithi139 ceph-mon[129293]: expand_channel_meta from 'info' to 'info' 2023-07-14T19:23:39.018 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:23:38 smithi139 ceph-mon[129293]: expand_channel_meta expanded map: {default=info} 2023-07-14T19:23:39.018 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:23:38 smithi139 ceph-mon[129293]: expand_channel_meta expand map: {default=daemon} 2023-07-14T19:23:39.019 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:23:38 smithi139 ceph-mon[129293]: expand_channel_meta from 'daemon' to 'daemon' 2023-07-14T19:23:39.019 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:23:38 smithi139 ceph-mon[129293]: expand_channel_meta expanded map: {default=daemon} 2023-07-14T19:23:39.019 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:23:38 smithi139 ceph-mon[129293]: expand_channel_meta expand map: {default=debug} 2023-07-14T19:23:39.019 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:23:38 smithi139 ceph-mon[129293]: expand_channel_meta from 'debug' to 'debug' 2023-07-14T19:23:39.019 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:23:38 smithi139 ceph-mon[129293]: expand_channel_meta expanded map: {default=debug} 2023-07-14T19:23:39.019 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:23:38 smithi139 ceph-mon[129293]: mon.c@-1(synchronizing) e1 handle_conf_change mon_allow_pool_delete 2023-07-14T19:23:39.019 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:23:38 smithi139 systemd[1]: Started Ceph mon.c for 9adea648-227b-11ee-9b33-001a4aab830c. 2023-07-14T19:23:39.031 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:23:38 smithi098 ceph-mon[127486]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-14T19:23:39.031 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:23:38 smithi098 ceph-mon[127486]: from='client.? 172.21.15.139:0/4207913916' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-07-14T19:23:39.646 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2023-07-14T19:23:39.646 DEBUG:teuthology.orchestra.run.smithi139:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81c9387d2e7576da065747ec9c16d1c7d69016ec shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 9adea648-227b-11ee-9b33-001a4aab830c -- ceph mon dump -f json 2023-07-14T19:23:40.202 INFO:teuthology.orchestra.run.smithi139.stderr:Inferring config /var/lib/ceph/9adea648-227b-11ee-9b33-001a4aab830c/mon.c/config 2023-07-14T19:23:44.221 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:23:43 smithi139 ceph-mon[129293]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2023-07-14T19:23:44.221 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:23:43 smithi139 ceph-mon[129293]: mon.a calling monitor election 2023-07-14T19:23:44.222 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:23:43 smithi139 ceph-mon[129293]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-07-14T19:23:44.222 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:23:43 smithi139 ceph-mon[129293]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-14T19:23:44.222 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:23:43 smithi139 ceph-mon[129293]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-07-14T19:23:44.222 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:23:43 smithi139 ceph-mon[129293]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-07-14T19:23:44.222 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:23:43 smithi139 ceph-mon[129293]: mon.c calling monitor election 2023-07-14T19:23:44.222 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:23:43 smithi139 ceph-mon[129293]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-14T19:23:44.223 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:23:43 smithi139 ceph-mon[129293]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-07-14T19:23:44.223 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:23:43 smithi139 ceph-mon[129293]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-07-14T19:23:44.223 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:23:43 smithi139 ceph-mon[129293]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-07-14T19:23:44.223 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:23:43 smithi139 ceph-mon[129293]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2023-07-14T19:23:44.223 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:23:43 smithi139 ceph-mon[129293]: monmap e2: 2 mons at {a=[v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0],c=[v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0]} removed_ranks: {} 2023-07-14T19:23:44.223 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:23:43 smithi139 ceph-mon[129293]: fsmap 2023-07-14T19:23:44.223 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:23:43 smithi139 ceph-mon[129293]: osdmap e4: 0 total, 0 up, 0 in 2023-07-14T19:23:44.223 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:23:43 smithi139 ceph-mon[129293]: mgrmap e13: x(active, since 42s) 2023-07-14T19:23:44.224 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:23:43 smithi139 ceph-mon[129293]: overall HEALTH_OK 2023-07-14T19:23:44.224 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:23:43 smithi139 ceph-mon[129293]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' 2023-07-14T19:23:44.224 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:23:43 smithi139 ceph-mon[129293]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' 2023-07-14T19:23:44.281 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:23:43 smithi098 ceph-mon[127486]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2023-07-14T19:23:44.282 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:23:43 smithi098 ceph-mon[127486]: mon.a calling monitor election 2023-07-14T19:23:44.282 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:23:43 smithi098 ceph-mon[127486]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-07-14T19:23:44.282 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:23:43 smithi098 ceph-mon[127486]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-14T19:23:44.283 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:23:43 smithi098 ceph-mon[127486]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-07-14T19:23:44.283 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:23:43 smithi098 ceph-mon[127486]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-07-14T19:23:44.283 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:23:43 smithi098 ceph-mon[127486]: mon.c calling monitor election 2023-07-14T19:23:44.283 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:23:43 smithi098 ceph-mon[127486]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-14T19:23:44.283 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:23:43 smithi098 ceph-mon[127486]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-07-14T19:23:44.283 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:23:43 smithi098 ceph-mon[127486]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-07-14T19:23:44.283 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:23:43 smithi098 ceph-mon[127486]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-07-14T19:23:44.284 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:23:43 smithi098 ceph-mon[127486]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2023-07-14T19:23:44.284 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:23:43 smithi098 ceph-mon[127486]: monmap e2: 2 mons at {a=[v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0],c=[v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0]} removed_ranks: {} 2023-07-14T19:23:44.284 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:23:43 smithi098 ceph-mon[127486]: fsmap 2023-07-14T19:23:44.284 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:23:43 smithi098 ceph-mon[127486]: osdmap e4: 0 total, 0 up, 0 in 2023-07-14T19:23:44.284 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:23:43 smithi098 ceph-mon[127486]: mgrmap e13: x(active, since 42s) 2023-07-14T19:23:44.284 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:23:43 smithi098 ceph-mon[127486]: overall HEALTH_OK 2023-07-14T19:23:44.285 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:23:43 smithi098 ceph-mon[127486]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' 2023-07-14T19:23:44.285 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:23:43 smithi098 ceph-mon[127486]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' 2023-07-14T19:23:44.435 INFO:teuthology.orchestra.run.smithi139.stderr:2023-07-14T19:23:44.434+0000 7f6566912700 1 -- 172.21.15.139:0/3609620506 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f65480035c0 msgr2=0x7f6548005a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:23:44.435 INFO:teuthology.orchestra.run.smithi139.stderr:2023-07-14T19:23:44.434+0000 7f6566912700 1 --2- 172.21.15.139:0/3609620506 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f65480035c0 0x7f6548005a40 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f65600ffc70 tx=0x7f655400fe60 comp rx=0 tx=0).stop 2023-07-14T19:23:44.435 INFO:teuthology.orchestra.run.smithi139.stderr:2023-07-14T19:23:44.434+0000 7f6566912700 1 -- 172.21.15.139:0/3609620506 shutdown_connections 2023-07-14T19:23:44.435 INFO:teuthology.orchestra.run.smithi139.stderr:2023-07-14T19:23:44.434+0000 7f6566912700 1 --2- 172.21.15.139:0/3609620506 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f65480035c0 0x7f6548005a40 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:23:44.435 INFO:teuthology.orchestra.run.smithi139.stderr:2023-07-14T19:23:44.434+0000 7f6566912700 1 -- 172.21.15.139:0/3609620506 >> 172.21.15.139:0/3609620506 conn(0x7f6560074500 msgr2=0x7f6560074900 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:23:44.436 INFO:teuthology.orchestra.run.smithi139.stderr:2023-07-14T19:23:44.434+0000 7f6566912700 1 -- 172.21.15.139:0/3609620506 shutdown_connections 2023-07-14T19:23:44.436 INFO:teuthology.orchestra.run.smithi139.stderr:2023-07-14T19:23:44.434+0000 7f6566912700 1 -- 172.21.15.139:0/3609620506 wait complete. 2023-07-14T19:23:44.437 INFO:teuthology.orchestra.run.smithi139.stderr:2023-07-14T19:23:44.435+0000 7f6566912700 1 Processor -- start 2023-07-14T19:23:44.437 INFO:teuthology.orchestra.run.smithi139.stderr:2023-07-14T19:23:44.435+0000 7f6566912700 1 -- start start 2023-07-14T19:23:44.437 INFO:teuthology.orchestra.run.smithi139.stderr:2023-07-14T19:23:44.436+0000 7f6566912700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f65480035c0 0x7f6560194d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:23:44.437 INFO:teuthology.orchestra.run.smithi139.stderr:2023-07-14T19:23:44.436+0000 7f6566912700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f6560195250 0x7f65601996c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:23:44.437 INFO:teuthology.orchestra.run.smithi139.stderr:2023-07-14T19:23:44.436+0000 7f6566912700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f65601957d0 con 0x7f65480035c0 2023-07-14T19:23:44.438 INFO:teuthology.orchestra.run.smithi139.stderr:2023-07-14T19:23:44.436+0000 7f6566912700 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6560195940 con 0x7f6560195250 2023-07-14T19:23:44.438 INFO:teuthology.orchestra.run.smithi139.stderr:2023-07-14T19:23:44.436+0000 7f655f7fe700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f6560195250 0x7f65601996c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:23:44.438 INFO:teuthology.orchestra.run.smithi139.stderr:2023-07-14T19:23:44.436+0000 7f655f7fe700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f6560195250 0x7f65601996c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.139:3300/0 says I am v2:172.21.15.139:37020/0 (socket says 172.21.15.139:37020) 2023-07-14T19:23:44.438 INFO:teuthology.orchestra.run.smithi139.stderr:2023-07-14T19:23:44.436+0000 7f655f7fe700 1 -- 172.21.15.139:0/2991630039 learned_addr learned my addr 172.21.15.139:0/2991630039 (peer_addr_for_me v2:172.21.15.139:0/0) 2023-07-14T19:23:44.438 INFO:teuthology.orchestra.run.smithi139.stderr:2023-07-14T19:23:44.436+0000 7f655ffff700 1 --2- 172.21.15.139:0/2991630039 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f65480035c0 0x7f6560194d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:23:44.438 INFO:teuthology.orchestra.run.smithi139.stderr:2023-07-14T19:23:44.436+0000 7f655f7fe700 1 -- 172.21.15.139:0/2991630039 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f6560195250 msgr2=0x7f65601996c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2023-07-14T19:23:44.438 INFO:teuthology.orchestra.run.smithi139.stderr:2023-07-14T19:23:44.436+0000 7f655f7fe700 1 -- 172.21.15.139:0/2991630039 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f6560195250 msgr2=0x7f65601996c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2023-07-14T19:23:44.439 INFO:teuthology.orchestra.run.smithi139.stderr:2023-07-14T19:23:44.436+0000 7f655f7fe700 1 --2- 172.21.15.139:0/2991630039 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f6560195250 0x7f65601996c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-07-14T19:23:44.439 INFO:teuthology.orchestra.run.smithi139.stderr:2023-07-14T19:23:44.436+0000 7f655f7fe700 1 --2- 172.21.15.139:0/2991630039 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f6560195250 0x7f65601996c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-07-14T19:23:44.439 INFO:teuthology.orchestra.run.smithi139.stderr:2023-07-14T19:23:44.436+0000 7f655ffff700 1 -- 172.21.15.139:0/2991630039 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f6560195250 msgr2=0x7f65601996c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-07-14T19:23:44.439 INFO:teuthology.orchestra.run.smithi139.stderr:2023-07-14T19:23:44.436+0000 7f655ffff700 1 --2- 172.21.15.139:0/2991630039 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f6560195250 0x7f65601996c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:23:44.440 INFO:teuthology.orchestra.run.smithi139.stderr:2023-07-14T19:23:44.436+0000 7f655ffff700 1 -- 172.21.15.139:0/2991630039 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f655400f8b0 con 0x7f65480035c0 2023-07-14T19:23:44.440 INFO:teuthology.orchestra.run.smithi139.stderr:2023-07-14T19:23:44.437+0000 7f655ffff700 1 --2- 172.21.15.139:0/2991630039 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f65480035c0 0x7f6560194d10 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f65540046a0 tx=0x7f6554013c90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:23:44.441 INFO:teuthology.orchestra.run.smithi139.stderr:2023-07-14T19:23:44.437+0000 7f655d7fa700 1 -- 172.21.15.139:0/2991630039 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 309+0+0 (secure 0 0 0) 0x7f6554023070 con 0x7f65480035c0 2023-07-14T19:23:44.441 INFO:teuthology.orchestra.run.smithi139.stderr:2023-07-14T19:23:44.437+0000 7f6566912700 1 -- 172.21.15.139:0/2991630039 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6560199c60 con 0x7f65480035c0 2023-07-14T19:23:44.442 INFO:teuthology.orchestra.run.smithi139.stderr:2023-07-14T19:23:44.437+0000 7f655d7fa700 1 -- 172.21.15.139:0/2991630039 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f6554004040 con 0x7f65480035c0 2023-07-14T19:23:44.442 INFO:teuthology.orchestra.run.smithi139.stderr:2023-07-14T19:23:44.437+0000 7f655d7fa700 1 -- 172.21.15.139:0/2991630039 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 309+0+0 (secure 0 0 0) 0x7f655401cb50 con 0x7f65480035c0 2023-07-14T19:23:44.442 INFO:teuthology.orchestra.run.smithi139.stderr:2023-07-14T19:23:44.437+0000 7f6566912700 1 -- 172.21.15.139:0/2991630039 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f656019a1e0 con 0x7f65480035c0 2023-07-14T19:23:44.442 INFO:teuthology.orchestra.run.smithi139.stderr:2023-07-14T19:23:44.438+0000 7f655d7fa700 1 -- 172.21.15.139:0/2991630039 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 13) v1 ==== 45539+0+0 (secure 0 0 0) 0x7f6554031440 con 0x7f65480035c0 2023-07-14T19:23:44.442 INFO:teuthology.orchestra.run.smithi139.stderr:2023-07-14T19:23:44.438+0000 7f655d7fa700 1 --2- 172.21.15.139:0/2991630039 >> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] conn(0x7f654c0385f0 0x7f654c03aaa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:23:44.442 INFO:teuthology.orchestra.run.smithi139.stderr:2023-07-14T19:23:44.438+0000 7f6566912700 1 -- 172.21.15.139:0/2991630039 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6560061c70 con 0x7f65480035c0 2023-07-14T19:23:44.443 INFO:teuthology.orchestra.run.smithi139.stderr:2023-07-14T19:23:44.438+0000 7f655d7fa700 1 -- 172.21.15.139:0/2991630039 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f655404d550 con 0x7f65480035c0 2023-07-14T19:23:44.443 INFO:teuthology.orchestra.run.smithi139.stderr:2023-07-14T19:23:44.438+0000 7f655f7fe700 1 --2- 172.21.15.139:0/2991630039 >> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] conn(0x7f654c0385f0 0x7f654c03aaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:23:44.443 INFO:teuthology.orchestra.run.smithi139.stderr:2023-07-14T19:23:44.439+0000 7f655f7fe700 1 --2- 172.21.15.139:0/2991630039 >> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] conn(0x7f654c0385f0 0x7f654c03aaa0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f6550009180 tx=0x7f6550008040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:23:44.444 INFO:teuthology.orchestra.run.smithi139.stderr:2023-07-14T19:23:44.444+0000 7f655d7fa700 1 -- 172.21.15.139:0/2991630039 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f655402f950 con 0x7f65480035c0 2023-07-14T19:23:44.703 INFO:teuthology.orchestra.run.smithi139.stderr:2023-07-14T19:23:44.702+0000 7f6566912700 1 -- 172.21.15.139:0/2991630039 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f656019f960 con 0x7f65480035c0 2023-07-14T19:23:44.705 INFO:teuthology.orchestra.run.smithi139.stderr:2023-07-14T19:23:44.703+0000 7f655d7fa700 1 -- 172.21.15.139:0/2991630039 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 2 v2) v1 ==== 95+0+1006 (secure 0 0 0) 0x7f6554016150 con 0x7f65480035c0 2023-07-14T19:23:44.705 INFO:teuthology.orchestra.run.smithi139.stdout: 2023-07-14T19:23:44.706 INFO:teuthology.orchestra.run.smithi139.stdout:{"epoch":2,"fsid":"9adea648-227b-11ee-9b33-001a4aab830c","modified":"2023-07-14T19:23:38.807497Z","created":"2023-07-14T19:22:04.850090Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:3300","nonce":0},{"type":"v1","addr":"172.21.15.98:6789","nonce":0}]},"addr":"172.21.15.98:6789/0","public_addr":"172.21.15.98:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:3300","nonce":0},{"type":"v1","addr":"172.21.15.139:6789","nonce":0}]},"addr":"172.21.15.139:6789/0","public_addr":"172.21.15.139:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0,1]} 2023-07-14T19:23:44.707 INFO:teuthology.orchestra.run.smithi139.stderr:2023-07-14T19:23:44.705+0000 7f6566912700 1 -- 172.21.15.139:0/2991630039 >> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] conn(0x7f654c0385f0 msgr2=0x7f654c03aaa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:23:44.708 INFO:teuthology.orchestra.run.smithi139.stderr:2023-07-14T19:23:44.705+0000 7f6566912700 1 --2- 172.21.15.139:0/2991630039 >> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] conn(0x7f654c0385f0 0x7f654c03aaa0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f6550009180 tx=0x7f6550008040 comp rx=0 tx=0).stop 2023-07-14T19:23:44.708 INFO:teuthology.orchestra.run.smithi139.stderr:2023-07-14T19:23:44.705+0000 7f6566912700 1 -- 172.21.15.139:0/2991630039 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f65480035c0 msgr2=0x7f6560194d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:23:44.708 INFO:teuthology.orchestra.run.smithi139.stderr:2023-07-14T19:23:44.705+0000 7f6566912700 1 --2- 172.21.15.139:0/2991630039 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f65480035c0 0x7f6560194d10 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f65540046a0 tx=0x7f6554013c90 comp rx=0 tx=0).stop 2023-07-14T19:23:44.708 INFO:teuthology.orchestra.run.smithi139.stderr:2023-07-14T19:23:44.706+0000 7f6566912700 1 -- 172.21.15.139:0/2991630039 shutdown_connections 2023-07-14T19:23:44.709 INFO:teuthology.orchestra.run.smithi139.stderr:2023-07-14T19:23:44.706+0000 7f6566912700 1 --2- 172.21.15.139:0/2991630039 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f65480035c0 0x7f6560194d10 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:23:44.709 INFO:teuthology.orchestra.run.smithi139.stderr:2023-07-14T19:23:44.706+0000 7f6566912700 1 --2- 172.21.15.139:0/2991630039 >> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] conn(0x7f654c0385f0 0x7f654c03aaa0 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:23:44.709 INFO:teuthology.orchestra.run.smithi139.stderr:2023-07-14T19:23:44.706+0000 7f6566912700 1 --2- 172.21.15.139:0/2991630039 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f6560195250 0x7f65601996c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:23:44.710 INFO:teuthology.orchestra.run.smithi139.stderr:2023-07-14T19:23:44.706+0000 7f6566912700 1 -- 172.21.15.139:0/2991630039 >> 172.21.15.139:0/2991630039 conn(0x7f6560074500 msgr2=0x7f6560067da0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:23:44.710 INFO:teuthology.orchestra.run.smithi139.stderr:2023-07-14T19:23:44.706+0000 7f6566912700 1 -- 172.21.15.139:0/2991630039 shutdown_connections 2023-07-14T19:23:44.710 INFO:teuthology.orchestra.run.smithi139.stderr:2023-07-14T19:23:44.706+0000 7f6566912700 1 -- 172.21.15.139:0/2991630039 wait complete. 2023-07-14T19:23:44.711 INFO:teuthology.orchestra.run.smithi139.stderr:dumped monmap epoch 2 2023-07-14T19:23:45.049 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:23:44 smithi139 ceph-mon[129293]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-14T19:23:45.049 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:23:44 smithi139 ceph-mon[129293]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' 2023-07-14T19:23:45.049 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:23:44 smithi139 ceph-mon[129293]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2023-07-14T19:23:45.049 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:23:44 smithi139 ceph-mon[129293]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T19:23:45.050 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:23:44 smithi139 ceph-mon[129293]: Deploying daemon mon.b on smithi101 2023-07-14T19:23:45.050 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:23:44 smithi139 ceph-mon[129293]: from='client.? 172.21.15.139:0/2991630039' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-07-14T19:23:45.050 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:23:44 smithi139 ceph-mon[129293]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-07-14T19:23:45.281 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:23:44 smithi098 ceph-mon[127486]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-14T19:23:45.282 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:23:44 smithi098 ceph-mon[127486]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' 2023-07-14T19:23:45.282 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:23:44 smithi098 ceph-mon[127486]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2023-07-14T19:23:45.282 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:23:44 smithi098 ceph-mon[127486]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T19:23:45.282 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:23:44 smithi098 ceph-mon[127486]: Deploying daemon mon.b on smithi101 2023-07-14T19:23:45.282 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:23:44 smithi098 ceph-mon[127486]: from='client.? 172.21.15.139:0/2991630039' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-07-14T19:23:45.282 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:23:44 smithi098 ceph-mon[127486]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-07-14T19:23:46.051 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2023-07-14T19:23:46.052 DEBUG:teuthology.orchestra.run.smithi139:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81c9387d2e7576da065747ec9c16d1c7d69016ec shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 9adea648-227b-11ee-9b33-001a4aab830c -- ceph mon dump -f json 2023-07-14T19:23:46.614 INFO:teuthology.orchestra.run.smithi139.stderr:Inferring config /var/lib/ceph/9adea648-227b-11ee-9b33-001a4aab830c/mon.c/config 2023-07-14T19:23:46.816 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:23:46 smithi101 systemd[1]: Starting Ceph mon.b for 9adea648-227b-11ee-9b33-001a4aab830c... 2023-07-14T19:23:47.080 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:23:46 smithi101 podman[128767]: 2023-07-14T19:23:47.080 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:23:47 smithi101 ceph-mon[128807]: set uid:gid to 167:167 (ceph:ceph) 2023-07-14T19:23:47.080 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:23:47 smithi101 ceph-mon[128807]: ceph version 18.0.0-4940-g81c9387d (81c9387d2e7576da065747ec9c16d1c7d69016ec) reef (dev), process ceph-mon, pid 8 2023-07-14T19:23:47.081 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:23:47 smithi101 ceph-mon[128807]: pidfile_write: ignore empty --pid-file 2023-07-14T19:23:47.081 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:23:47 smithi101 ceph-mon[128807]: load: jerasure load: lrc 2023-07-14T19:23:47.081 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:23:47 smithi101 ceph-mon[128807]: rocksdb: RocksDB version: 7.9.2 2023-07-14T19:23:47.081 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:23:47 smithi101 ceph-mon[128807]: rocksdb: Git sha 0 2023-07-14T19:23:47.081 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:23:47 smithi101 ceph-mon[128807]: rocksdb: Compile date 2023-07-14 14:45:42 2023-07-14T19:23:47.081 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:23:47 smithi101 ceph-mon[128807]: rocksdb: DB SUMMARY 2023-07-14T19:23:47.081 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:23:47 smithi101 ceph-mon[128807]: rocksdb: DB Session ID: JDRY8EZ8LM8NL9FB0P67 2023-07-14T19:23:47.082 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:23:47 smithi101 ceph-mon[128807]: rocksdb: CURRENT file: CURRENT 2023-07-14T19:23:47.082 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:23:47 smithi101 ceph-mon[128807]: rocksdb: IDENTITY file: IDENTITY 2023-07-14T19:23:47.082 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:23:47 smithi101 ceph-mon[128807]: rocksdb: MANIFEST file: MANIFEST-000005 size: 59 Bytes 2023-07-14T19:23:47.082 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:23:47 smithi101 ceph-mon[128807]: rocksdb: SST files in /var/lib/ceph/mon/ceph-b/store.db dir, Total Num: 0, files: 2023-07-14T19:23:47.082 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:23:47 smithi101 ceph-mon[128807]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-b/store.db: 000004.log size: 636 ; 2023-07-14T19:23:47.082 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:23:47 smithi101 ceph-mon[128807]: rocksdb: Options.error_if_exists: 0 2023-07-14T19:23:47.082 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:23:47 smithi101 ceph-mon[128807]: rocksdb: Options.create_if_missing: 0 2023-07-14T19:23:47.082 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:23:47 smithi101 ceph-mon[128807]: rocksdb: Options.paranoid_checks: 1 2023-07-14T19:23:47.083 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:23:47 smithi101 ceph-mon[128807]: rocksdb: Options.flush_verify_memtable_count: 1 2023-07-14T19:23:47.083 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:23:47 smithi101 ceph-mon[128807]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2023-07-14T19:23:47.083 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:23:47 smithi101 ceph-mon[128807]: rocksdb: Options.verify_sst_unique_id_in_manifest: 1 2023-07-14T19:23:47.083 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:23:47 smithi101 ceph-mon[128807]: rocksdb: Options.env: 0x564f8ac86c60 2023-07-14T19:23:47.083 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:23:47 smithi101 ceph-mon[128807]: rocksdb: Options.fs: PosixFileSystem 2023-07-14T19:23:47.083 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:23:47 smithi101 ceph-mon[128807]: rocksdb: Options.info_log: 0x564f8b7e72a0 2023-07-14T19:23:47.083 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:23:47 smithi101 ceph-mon[128807]: rocksdb: Options.max_file_opening_threads: 16 2023-07-14T19:23:47.084 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:23:47 smithi101 ceph-mon[128807]: rocksdb: Options.statistics: (nil) 2023-07-14T19:23:47.084 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:23:47 smithi101 ceph-mon[128807]: rocksdb: Options.use_fsync: 0 2023-07-14T19:23:47.084 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:23:47 smithi101 ceph-mon[128807]: rocksdb: Options.max_log_file_size: 0 2023-07-14T19:23:47.084 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:23:47 smithi101 ceph-mon[128807]: rocksdb: Options.max_manifest_file_size: 1073741824 2023-07-14T19:23:47.084 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:23:47 smithi101 ceph-mon[128807]: rocksdb: Options.log_file_time_to_roll: 0 2023-07-14T19:23:47.084 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:23:47 smithi101 ceph-mon[128807]: rocksdb: Options.keep_log_file_num: 1000 2023-07-14T19:23:47.084 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:23:47 smithi101 ceph-mon[128807]: rocksdb: Options.recycle_log_file_num: 0 2023-07-14T19:23:47.084 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:23:47 smithi101 ceph-mon[128807]: rocksdb: Options.allow_fallocate: 1 2023-07-14T19:23:47.085 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:23:47 smithi101 ceph-mon[128807]: rocksdb: Options.allow_mmap_reads: 0 2023-07-14T19:23:47.085 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:23:47 smithi101 ceph-mon[128807]: rocksdb: Options.allow_mmap_writes: 0 2023-07-14T19:23:47.085 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:23:47 smithi101 ceph-mon[128807]: rocksdb: Options.use_direct_reads: 0 2023-07-14T19:23:47.085 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:23:47 smithi101 ceph-mon[128807]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2023-07-14T19:23:47.085 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:23:47 smithi101 ceph-mon[128807]: rocksdb: Options.create_missing_column_families: 0 2023-07-14T19:23:47.085 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:23:47 smithi101 ceph-mon[128807]: rocksdb: Options.db_log_dir: 2023-07-14T19:23:47.085 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:23:47 smithi101 ceph-mon[128807]: rocksdb: Options.wal_dir: 2023-07-14T19:23:47.086 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:23:47 smithi101 ceph-mon[128807]: rocksdb: Options.table_cache_numshardbits: 6 2023-07-14T19:23:47.086 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:23:47 smithi101 ceph-mon[128807]: rocksdb: Options.WAL_ttl_seconds: 0 2023-07-14T19:23:47.086 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:23:47 smithi101 ceph-mon[128807]: rocksdb: Options.WAL_size_limit_MB: 0 2023-07-14T19:23:47.086 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:23:47 smithi101 ceph-mon[128807]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2023-07-14T19:23:47.086 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:23:47 smithi101 ceph-mon[128807]: rocksdb: Options.manifest_preallocation_size: 4194304 2023-07-14T19:23:47.086 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:23:47 smithi101 ceph-mon[128807]: rocksdb: Options.is_fd_close_on_exec: 1 2023-07-14T19:23:47.086 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:23:47 smithi101 ceph-mon[128807]: rocksdb: Options.advise_random_on_open: 1 2023-07-14T19:23:47.086 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:23:47 smithi101 ceph-mon[128807]: rocksdb: Options.db_write_buffer_size: 0 2023-07-14T19:23:47.087 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:23:47 smithi101 ceph-mon[128807]: rocksdb: Options.write_buffer_manager: 0x564f8b7083c0 2023-07-14T19:23:47.087 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:23:47 smithi101 ceph-mon[128807]: rocksdb: Options.access_hint_on_compaction_start: 1 2023-07-14T19:23:47.087 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:23:47 smithi101 ceph-mon[128807]: rocksdb: Options.random_access_max_buffer_size: 1048576 2023-07-14T19:23:47.087 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:23:47 smithi101 ceph-mon[128807]: rocksdb: Options.use_adaptive_mutex: 0 2023-07-14T19:23:47.087 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:23:47 smithi101 ceph-mon[128807]: rocksdb: Options.rate_limiter: (nil) 2023-07-14T19:23:47.087 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:23:47 smithi101 ceph-mon[128807]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2023-07-14T19:23:47.087 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:23:47 smithi101 ceph-mon[128807]: rocksdb: Options.wal_recovery_mode: 2 2023-07-14T19:23:47.087 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:23:47 smithi101 ceph-mon[128807]: rocksdb: Options.enable_thread_tracking: 0 2023-07-14T19:23:47.088 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:23:47 smithi101 ceph-mon[128807]: rocksdb: Options.enable_pipelined_write: 0 2023-07-14T19:23:47.088 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:23:47 smithi101 ceph-mon[128807]: rocksdb: Options.unordered_write: 0 2023-07-14T19:23:47.088 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:23:47 smithi101 ceph-mon[128807]: rocksdb: Options.allow_concurrent_memtable_write: 1 2023-07-14T19:23:47.088 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:23:47 smithi101 ceph-mon[128807]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2023-07-14T19:23:47.088 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:23:47 smithi101 ceph-mon[128807]: rocksdb: Options.write_thread_max_yield_usec: 100 2023-07-14T19:23:47.088 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:23:47 smithi101 ceph-mon[128807]: rocksdb: Options.write_thread_slow_yield_usec: 3 2023-07-14T19:23:47.088 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:23:47 smithi101 ceph-mon[128807]: rocksdb: Options.row_cache: None 2023-07-14T19:23:47.089 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:23:47 smithi101 ceph-mon[128807]: rocksdb: Options.wal_filter: None 2023-07-14T19:23:47.089 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:23:47 smithi101 ceph-mon[128807]: rocksdb: Options.avoid_flush_during_recovery: 0 2023-07-14T19:23:47.089 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:23:47 smithi101 ceph-mon[128807]: rocksdb: Options.allow_ingest_behind: 0 2023-07-14T19:23:47.089 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:23:47 smithi101 ceph-mon[128807]: rocksdb: Options.two_write_queues: 0 2023-07-14T19:23:47.089 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:23:47 smithi101 ceph-mon[128807]: rocksdb: Options.manual_wal_flush: 0 2023-07-14T19:23:47.089 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:23:47 smithi101 ceph-mon[128807]: rocksdb: Options.wal_compression: 0 2023-07-14T19:23:47.089 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:23:47 smithi101 ceph-mon[128807]: rocksdb: Options.atomic_flush: 0 2023-07-14T19:23:47.089 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:23:47 smithi101 ceph-mon[128807]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2023-07-14T19:23:47.090 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:23:47 smithi101 ceph-mon[128807]: rocksdb: Options.persist_stats_to_disk: 0 2023-07-14T19:23:47.090 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:23:47 smithi101 ceph-mon[128807]: rocksdb: Options.write_dbid_to_manifest: 0 2023-07-14T19:23:47.090 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:23:47 smithi101 ceph-mon[128807]: rocksdb: Options.log_readahead_size: 0 2023-07-14T19:23:47.090 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:23:47 smithi101 ceph-mon[128807]: rocksdb: Options.file_checksum_gen_factory: Unknown 2023-07-14T19:23:47.090 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:23:47 smithi101 ceph-mon[128807]: rocksdb: Options.best_efforts_recovery: 0 2023-07-14T19:23:47.090 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:23:47 smithi101 ceph-mon[128807]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2023-07-14T19:23:47.090 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:23:47 smithi101 ceph-mon[128807]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2023-07-14T19:23:47.090 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:23:47 smithi101 ceph-mon[128807]: rocksdb: Options.allow_data_in_errors: 0 2023-07-14T19:23:47.091 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:23:47 smithi101 ceph-mon[128807]: rocksdb: Options.db_host_id: __hostname__ 2023-07-14T19:23:47.091 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:23:47 smithi101 ceph-mon[128807]: rocksdb: Options.enforce_single_del_contracts: true 2023-07-14T19:23:47.091 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:23:47 smithi101 ceph-mon[128807]: rocksdb: Options.max_background_jobs: 2 2023-07-14T19:23:47.091 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:23:47 smithi101 ceph-mon[128807]: rocksdb: Options.max_background_compactions: -1 2023-07-14T19:23:47.091 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:23:47 smithi101 ceph-mon[128807]: rocksdb: Options.max_subcompactions: 1 2023-07-14T19:23:47.091 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:23:47 smithi101 ceph-mon[128807]: rocksdb: Options.avoid_flush_during_shutdown: 0 2023-07-14T19:23:47.091 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:23:47 smithi101 ceph-mon[128807]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2023-07-14T19:23:47.092 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:23:47 smithi101 ceph-mon[128807]: rocksdb: Options.delayed_write_rate : 16777216 2023-07-14T19:23:47.092 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:23:47 smithi101 ceph-mon[128807]: rocksdb: Options.max_total_wal_size: 0 2023-07-14T19:23:47.092 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:23:47 smithi101 ceph-mon[128807]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2023-07-14T19:23:47.092 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:23:47 smithi101 ceph-mon[128807]: rocksdb: Options.stats_dump_period_sec: 600 2023-07-14T19:23:47.092 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:23:47 smithi101 ceph-mon[128807]: rocksdb: Options.stats_persist_period_sec: 600 2023-07-14T19:23:47.092 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:23:47 smithi101 ceph-mon[128807]: rocksdb: Options.stats_history_buffer_size: 1048576 2023-07-14T19:23:47.092 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:23:47 smithi101 ceph-mon[128807]: rocksdb: Options.max_open_files: -1 2023-07-14T19:23:47.093 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:23:47 smithi101 ceph-mon[128807]: rocksdb: Options.bytes_per_sync: 0 2023-07-14T19:23:47.093 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:23:47 smithi101 ceph-mon[128807]: rocksdb: Options.wal_bytes_per_sync: 0 2023-07-14T19:23:47.093 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:23:47 smithi101 ceph-mon[128807]: rocksdb: Options.strict_bytes_per_sync: 0 2023-07-14T19:23:47.093 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:23:47 smithi101 ceph-mon[128807]: rocksdb: Options.compaction_readahead_size: 0 2023-07-14T19:23:47.093 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:23:47 smithi101 ceph-mon[128807]: rocksdb: Options.max_background_flushes: -1 2023-07-14T19:23:47.093 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:23:47 smithi101 ceph-mon[128807]: rocksdb: Compression algorithms supported: 2023-07-14T19:23:47.093 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:23:47 smithi101 ceph-mon[128807]: rocksdb: kZSTDNotFinalCompression supported: 0 2023-07-14T19:23:47.093 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:23:47 smithi101 ceph-mon[128807]: rocksdb: kZSTD supported: 0 2023-07-14T19:23:47.094 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:23:47 smithi101 ceph-mon[128807]: rocksdb: kXpressCompression supported: 0 2023-07-14T19:23:47.094 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:23:47 smithi101 ceph-mon[128807]: rocksdb: kLZ4HCCompression supported: 1 2023-07-14T19:23:47.094 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:23:47 smithi101 ceph-mon[128807]: rocksdb: kZlibCompression supported: 1 2023-07-14T19:23:47.094 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:23:47 smithi101 ceph-mon[128807]: rocksdb: kSnappyCompression supported: 1 2023-07-14T19:23:47.094 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:23:47 smithi101 ceph-mon[128807]: rocksdb: kLZ4Compression supported: 1 2023-07-14T19:23:47.094 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:23:47 smithi101 ceph-mon[128807]: rocksdb: kBZip2Compression supported: 0 2023-07-14T19:23:47.094 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:23:47 smithi101 ceph-mon[128807]: rocksdb: Fast CRC32 supported: Supported on x86 2023-07-14T19:23:47.094 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:23:47 smithi101 ceph-mon[128807]: rocksdb: DMutex implementation: pthread_mutex_t 2023-07-14T19:23:47.095 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:23:47 smithi101 ceph-mon[128807]: rocksdb: [db/version_set.cc:5527] Recovering from manifest file: /var/lib/ceph/mon/ceph-b/store.db/MANIFEST-000005 2023-07-14T19:23:47.095 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:23:47 smithi101 ceph-mon[128807]: rocksdb: [db/column_family.cc:630] --------------- Options for column family [default]: 2023-07-14T19:23:47.095 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:23:47 smithi101 ceph-mon[128807]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2023-07-14T19:23:47.095 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:23:47 smithi101 ceph-mon[128807]: rocksdb: Options.merge_operator: 2023-07-14T19:23:47.095 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:23:47 smithi101 ceph-mon[128807]: rocksdb: Options.compaction_filter: None 2023-07-14T19:23:47.095 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:23:47 smithi101 ceph-mon[128807]: rocksdb: Options.compaction_filter_factory: None 2023-07-14T19:23:47.095 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:23:47 smithi101 ceph-mon[128807]: rocksdb: Options.sst_partitioner_factory: None 2023-07-14T19:23:47.096 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:23:47 smithi101 ceph-mon[128807]: rocksdb: Options.memtable_factory: SkipListFactory 2023-07-14T19:23:47.096 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:23:47 smithi101 bash[128767]: cee07623756b1db6fd8fe34958b4f3a4a032a5c271d84bb5306d12581383a58f 2023-07-14T19:23:47.096 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:23:47 smithi101 ceph-mon[128807]: rocksdb: Options.table_factory: BlockBasedTable 2023-07-14T19:23:47.096 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:23:47 smithi101 ceph-mon[128807]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x564f8b7e73a0) 2023-07-14T19:23:47.096 INFO:journalctl@ceph.mon.b.smithi101.stdout: cache_index_and_filter_blocks: 1 2023-07-14T19:23:47.096 INFO:journalctl@ceph.mon.b.smithi101.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2023-07-14T19:23:47.096 INFO:journalctl@ceph.mon.b.smithi101.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2023-07-14T19:23:47.096 INFO:journalctl@ceph.mon.b.smithi101.stdout: pin_top_level_index_and_filter: 1 2023-07-14T19:23:47.097 INFO:journalctl@ceph.mon.b.smithi101.stdout: index_type: 0 2023-07-14T19:23:47.097 INFO:journalctl@ceph.mon.b.smithi101.stdout: data_block_index_type: 0 2023-07-14T19:23:47.097 INFO:journalctl@ceph.mon.b.smithi101.stdout: index_shortening: 1 2023-07-14T19:23:47.097 INFO:journalctl@ceph.mon.b.smithi101.stdout: data_block_hash_table_util_ratio: 0.750000 2023-07-14T19:23:47.097 INFO:journalctl@ceph.mon.b.smithi101.stdout: checksum: 4 2023-07-14T19:23:47.097 INFO:journalctl@ceph.mon.b.smithi101.stdout: no_block_cache: 0 2023-07-14T19:23:47.097 INFO:journalctl@ceph.mon.b.smithi101.stdout: block_cache: 0x564f8b7a71f0 2023-07-14T19:23:47.097 INFO:journalctl@ceph.mon.b.smithi101.stdout: block_cache_name: BinnedLRUCache 2023-07-14T19:23:47.098 INFO:journalctl@ceph.mon.b.smithi101.stdout: block_cache_options: 2023-07-14T19:23:47.098 INFO:journalctl@ceph.mon.b.smithi101.stdout: capacity : 536870912 2023-07-14T19:23:47.098 INFO:journalctl@ceph.mon.b.smithi101.stdout: num_shard_bits : 4 2023-07-14T19:23:47.098 INFO:journalctl@ceph.mon.b.smithi101.stdout: strict_capacity_limit : 0 2023-07-14T19:23:47.098 INFO:journalctl@ceph.mon.b.smithi101.stdout: high_pri_pool_ratio: 0.000 2023-07-14T19:23:47.098 INFO:journalctl@ceph.mon.b.smithi101.stdout: block_cache_compressed: (nil) 2023-07-14T19:23:47.098 INFO:journalctl@ceph.mon.b.smithi101.stdout: persistent_cache: (nil) 2023-07-14T19:23:47.099 INFO:journalctl@ceph.mon.b.smithi101.stdout: block_size: 4096 2023-07-14T19:23:47.099 INFO:journalctl@ceph.mon.b.smithi101.stdout: block_size_deviation: 10 2023-07-14T19:23:47.099 INFO:journalctl@ceph.mon.b.smithi101.stdout: block_restart_interval: 16 2023-07-14T19:23:47.099 INFO:journalctl@ceph.mon.b.smithi101.stdout: index_block_restart_interval: 1 2023-07-14T19:23:47.099 INFO:journalctl@ceph.mon.b.smithi101.stdout: metadata_block_size: 4096 2023-07-14T19:23:47.099 INFO:journalctl@ceph.mon.b.smithi101.stdout: partition_filters: 0 2023-07-14T19:23:47.099 INFO:journalctl@ceph.mon.b.smithi101.stdout: use_delta_encoding: 1 2023-07-14T19:23:47.099 INFO:journalctl@ceph.mon.b.smithi101.stdout: filter_policy: bloomfilter 2023-07-14T19:23:47.100 INFO:journalctl@ceph.mon.b.smithi101.stdout: whole_key_filtering: 1 2023-07-14T19:23:47.100 INFO:journalctl@ceph.mon.b.smithi101.stdout: verify_compression: 0 2023-07-14T19:23:47.100 INFO:journalctl@ceph.mon.b.smithi101.stdout: read_amp_bytes_per_bit: 0 2023-07-14T19:23:47.100 INFO:journalctl@ceph.mon.b.smithi101.stdout: format_version: 5 2023-07-14T19:23:47.100 INFO:journalctl@ceph.mon.b.smithi101.stdout: enable_index_compression: 1 2023-07-14T19:23:47.100 INFO:journalctl@ceph.mon.b.smithi101.stdout: block_align: 0 2023-07-14T19:23:47.100 INFO:journalctl@ceph.mon.b.smithi101.stdout: max_auto_readahead_size: 262144 2023-07-14T19:23:47.100 INFO:journalctl@ceph.mon.b.smithi101.stdout: prepopulate_block_cache: 0 2023-07-14T19:23:47.101 INFO:journalctl@ceph.mon.b.smithi101.stdout: initial_auto_readahead_size: 8192 2023-07-14T19:23:47.101 INFO:journalctl@ceph.mon.b.smithi101.stdout: num_file_reads_for_auto_readahead: 2 2023-07-14T19:23:47.101 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:23:47 smithi101 ceph-mon[128807]: rocksdb: Options.write_buffer_size: 33554432 2023-07-14T19:23:47.101 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:23:47 smithi101 ceph-mon[128807]: rocksdb: Options.max_write_buffer_number: 2 2023-07-14T19:23:47.101 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:23:47 smithi101 ceph-mon[128807]: rocksdb: Options.compression: NoCompression 2023-07-14T19:23:47.101 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:23:47 smithi101 ceph-mon[128807]: rocksdb: Options.bottommost_compression: Disabled 2023-07-14T19:23:47.101 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:23:47 smithi101 ceph-mon[128807]: rocksdb: Options.prefix_extractor: nullptr 2023-07-14T19:23:47.101 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:23:47 smithi101 ceph-mon[128807]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2023-07-14T19:23:47.102 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:23:47 smithi101 ceph-mon[128807]: rocksdb: Options.num_levels: 7 2023-07-14T19:23:47.102 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:23:47 smithi101 ceph-mon[128807]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2023-07-14T19:23:47.102 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:23:47 smithi101 ceph-mon[128807]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2023-07-14T19:23:47.102 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:23:47 smithi101 ceph-mon[128807]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2023-07-14T19:23:47.102 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:23:47 smithi101 ceph-mon[128807]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2023-07-14T19:23:47.102 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:23:47 smithi101 ceph-mon[128807]: rocksdb: Options.bottommost_compression_opts.level: 32767 2023-07-14T19:23:47.102 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:23:47 smithi101 ceph-mon[128807]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2023-07-14T19:23:47.103 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:23:47 smithi101 ceph-mon[128807]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2023-07-14T19:23:47.103 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:23:47 smithi101 ceph-mon[128807]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2023-07-14T19:23:47.103 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:23:47 smithi101 ceph-mon[128807]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2023-07-14T19:23:47.103 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:23:47 smithi101 ceph-mon[128807]: rocksdb: Options.bottommost_compression_opts.enabled: false 2023-07-14T19:23:47.103 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:23:47 smithi101 ceph-mon[128807]: rocksdb: Options.bottommost_compression_opts.max_dict_buffer_bytes: 0 2023-07-14T19:23:47.103 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:23:47 smithi101 ceph-mon[128807]: rocksdb: Options.bottommost_compression_opts.use_zstd_dict_trainer: true 2023-07-14T19:23:47.103 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:23:47 smithi101 ceph-mon[128807]: rocksdb: Options.compression_opts.window_bits: -14 2023-07-14T19:23:47.103 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:23:47 smithi101 ceph-mon[128807]: rocksdb: Options.compression_opts.level: 32767 2023-07-14T19:23:47.104 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:23:47 smithi101 ceph-mon[128807]: rocksdb: Options.compression_opts.strategy: 0 2023-07-14T19:23:47.104 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:23:47 smithi101 ceph-mon[128807]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2023-07-14T19:23:47.104 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:23:47 smithi101 ceph-mon[128807]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2023-07-14T19:23:47.104 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:23:47 smithi101 ceph-mon[128807]: rocksdb: Options.compression_opts.use_zstd_dict_trainer: true 2023-07-14T19:23:47.104 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:23:47 smithi101 ceph-mon[128807]: rocksdb: Options.compression_opts.parallel_threads: 1 2023-07-14T19:23:47.104 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:23:47 smithi101 ceph-mon[128807]: rocksdb: Options.compression_opts.enabled: false 2023-07-14T19:23:47.104 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:23:47 smithi101 ceph-mon[128807]: rocksdb: Options.compression_opts.max_dict_buffer_bytes: 0 2023-07-14T19:23:47.104 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:23:47 smithi101 ceph-mon[128807]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2023-07-14T19:23:47.105 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:23:47 smithi101 ceph-mon[128807]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2023-07-14T19:23:47.105 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:23:47 smithi101 ceph-mon[128807]: rocksdb: Options.level0_stop_writes_trigger: 36 2023-07-14T19:23:47.105 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:23:47 smithi101 ceph-mon[128807]: rocksdb: Options.target_file_size_base: 67108864 2023-07-14T19:23:47.105 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:23:47 smithi101 ceph-mon[128807]: rocksdb: Options.target_file_size_multiplier: 1 2023-07-14T19:23:47.105 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:23:47 smithi101 ceph-mon[128807]: rocksdb: Options.max_bytes_for_level_base: 268435456 2023-07-14T19:23:47.105 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:23:47 smithi101 ceph-mon[128807]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2023-07-14T19:23:47.105 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:23:47 smithi101 ceph-mon[128807]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2023-07-14T19:23:47.106 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:23:47 smithi101 ceph-mon[128807]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2023-07-14T19:23:47.106 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:23:47 smithi101 ceph-mon[128807]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2023-07-14T19:23:47.106 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:23:47 smithi101 ceph-mon[128807]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2023-07-14T19:23:47.106 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:23:47 smithi101 ceph-mon[128807]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2023-07-14T19:23:47.106 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:23:47 smithi101 ceph-mon[128807]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2023-07-14T19:23:47.106 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:23:47 smithi101 ceph-mon[128807]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2023-07-14T19:23:47.106 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:23:47 smithi101 ceph-mon[128807]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2023-07-14T19:23:47.106 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:23:47 smithi101 ceph-mon[128807]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2023-07-14T19:23:47.107 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:23:47 smithi101 ceph-mon[128807]: rocksdb: Options.max_compaction_bytes: 1677721600 2023-07-14T19:23:47.107 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:23:47 smithi101 ceph-mon[128807]: rocksdb: Options.ignore_max_compaction_bytes_for_input: true 2023-07-14T19:23:47.107 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:23:47 smithi101 ceph-mon[128807]: rocksdb: Options.arena_block_size: 1048576 2023-07-14T19:23:47.107 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:23:47 smithi101 ceph-mon[128807]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2023-07-14T19:23:47.107 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:23:47 smithi101 ceph-mon[128807]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2023-07-14T19:23:47.107 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:23:47 smithi101 ceph-mon[128807]: rocksdb: Options.disable_auto_compactions: 0 2023-07-14T19:23:47.108 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:23:46 smithi139 ceph-mon[129293]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-14T19:23:47.109 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:23:47 smithi101 ceph-mon[128807]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2023-07-14T19:23:47.109 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:23:47 smithi101 ceph-mon[128807]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2023-07-14T19:23:47.109 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:23:47 smithi101 ceph-mon[128807]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2023-07-14T19:23:47.109 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:23:47 smithi101 ceph-mon[128807]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2023-07-14T19:23:47.109 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:23:47 smithi101 ceph-mon[128807]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2023-07-14T19:23:47.109 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:23:47 smithi101 ceph-mon[128807]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2023-07-14T19:23:47.109 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:23:47 smithi101 ceph-mon[128807]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2023-07-14T19:23:47.110 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:23:47 smithi101 ceph-mon[128807]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2023-07-14T19:23:47.110 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:23:47 smithi101 ceph-mon[128807]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2023-07-14T19:23:47.110 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:23:47 smithi101 ceph-mon[128807]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2023-07-14T19:23:47.110 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:23:47 smithi101 ceph-mon[128807]: rocksdb: Options.table_properties_collectors: 2023-07-14T19:23:47.110 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:23:47 smithi101 ceph-mon[128807]: rocksdb: Options.inplace_update_support: 0 2023-07-14T19:23:47.110 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:23:47 smithi101 ceph-mon[128807]: rocksdb: Options.inplace_update_num_locks: 10000 2023-07-14T19:23:47.110 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:23:47 smithi101 ceph-mon[128807]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2023-07-14T19:23:47.110 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:23:47 smithi101 ceph-mon[128807]: rocksdb: Options.memtable_whole_key_filtering: 0 2023-07-14T19:23:47.111 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:23:47 smithi101 ceph-mon[128807]: rocksdb: Options.memtable_huge_page_size: 0 2023-07-14T19:23:47.111 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:23:47 smithi101 ceph-mon[128807]: rocksdb: Options.bloom_locality: 0 2023-07-14T19:23:47.111 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:23:47 smithi101 ceph-mon[128807]: rocksdb: Options.max_successive_merges: 0 2023-07-14T19:23:47.111 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:23:47 smithi101 ceph-mon[128807]: rocksdb: Options.optimize_filters_for_hits: 0 2023-07-14T19:23:47.111 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:23:47 smithi101 ceph-mon[128807]: rocksdb: Options.paranoid_file_checks: 0 2023-07-14T19:23:47.111 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:23:47 smithi101 ceph-mon[128807]: rocksdb: Options.force_consistency_checks: 1 2023-07-14T19:23:47.111 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:23:47 smithi101 ceph-mon[128807]: rocksdb: Options.report_bg_io_stats: 0 2023-07-14T19:23:47.111 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:23:47 smithi101 ceph-mon[128807]: rocksdb: Options.ttl: 2592000 2023-07-14T19:23:47.112 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:23:47 smithi101 ceph-mon[128807]: rocksdb: Options.periodic_compaction_seconds: 0 2023-07-14T19:23:47.112 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:23:47 smithi101 ceph-mon[128807]: rocksdb: Options.preclude_last_level_data_seconds: 0 2023-07-14T19:23:47.112 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:23:47 smithi101 ceph-mon[128807]: rocksdb: Options.preserve_internal_time_seconds: 0 2023-07-14T19:23:47.112 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:23:47 smithi101 ceph-mon[128807]: rocksdb: Options.enable_blob_files: false 2023-07-14T19:23:47.112 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:23:47 smithi101 ceph-mon[128807]: rocksdb: Options.min_blob_size: 0 2023-07-14T19:23:47.112 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:23:47 smithi101 ceph-mon[128807]: rocksdb: Options.blob_file_size: 268435456 2023-07-14T19:23:47.112 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:23:47 smithi101 ceph-mon[128807]: rocksdb: Options.blob_compression_type: NoCompression 2023-07-14T19:23:47.113 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:23:47 smithi101 ceph-mon[128807]: rocksdb: Options.enable_blob_garbage_collection: false 2023-07-14T19:23:47.113 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:23:47 smithi101 ceph-mon[128807]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2023-07-14T19:23:47.113 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:23:47 smithi101 ceph-mon[128807]: rocksdb: Options.blob_garbage_collection_force_threshold: 1.000000 2023-07-14T19:23:47.113 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:23:47 smithi101 ceph-mon[128807]: rocksdb: Options.blob_compaction_readahead_size: 0 2023-07-14T19:23:47.113 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:23:47 smithi101 ceph-mon[128807]: rocksdb: Options.blob_file_starting_level: 0 2023-07-14T19:23:47.113 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:23:47 smithi101 ceph-mon[128807]: rocksdb: Options.experimental_mempurge_threshold: 0.000000 2023-07-14T19:23:47.113 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:23:47 smithi101 ceph-mon[128807]: rocksdb: [db/version_set.cc:5566] Recovered from manifest file:/var/lib/ceph/mon/ceph-b/store.db/MANIFEST-000005 succeeded,manifest_file_number is 5, next_file_number is 7, last_sequence is 0, log_number is 0,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2023-07-14T19:23:47.113 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:23:47 smithi101 ceph-mon[128807]: rocksdb: [db/version_set.cc:5581] Column family [default] (ID 0), log number is 0 2023-07-14T19:23:47.114 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:23:47 smithi101 ceph-mon[128807]: rocksdb: [db/db_impl/db_impl_open.cc:539] DB ID: 2135faa6-2b94-43bf-83cd-76719855b55d 2023-07-14T19:23:47.114 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:23:47 smithi101 ceph-mon[128807]: rocksdb: EVENT_LOG_v1 {"time_micros": 1689362627057365, "job": 1, "event": "recovery_started", "wal_files": [4]} 2023-07-14T19:23:47.114 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:23:47 smithi101 ceph-mon[128807]: rocksdb: [db/db_impl/db_impl_open.cc:1043] Recovering log #4 mode 2 2023-07-14T19:23:47.114 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:23:47 smithi101 ceph-mon[128807]: rocksdb: EVENT_LOG_v1 {"time_micros": 1689362627057775, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 8, "file_size": 1747, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 1, "largest_seqno": 5, "table_properties": {"data_size": 648, "index_size": 31, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 69, "raw_key_size": 115, "raw_average_key_size": 23, "raw_value_size": 526, "raw_average_value_size": 105, "num_data_blocks": 1, "num_entries": 5, "num_filter_entries": 5, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1689362627, "oldest_key_time": 0, "file_creation_time": 0, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "2135faa6-2b94-43bf-83cd-76719855b55d", "db_session_id": "JDRY8EZ8LM8NL9FB0P67", "orig_file_number": 8, "seqno_to_time_mapping": "N/A"}} 2023-07-14T19:23:47.114 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:23:47 smithi101 ceph-mon[128807]: rocksdb: EVENT_LOG_v1 {"time_micros": 1689362627057840, "job": 1, "event": "recovery_finished"} 2023-07-14T19:23:47.114 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:23:47 smithi101 ceph-mon[128807]: rocksdb: [db/version_set.cc:5047] Creating manifest 10 2023-07-14T19:23:47.114 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:23:47 smithi101 ceph-mon[128807]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-b/store.db/000004.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2023-07-14T19:23:47.115 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:23:47 smithi101 ceph-mon[128807]: rocksdb: [db/db_impl/db_impl_open.cc:1987] SstFileManager instance 0x564f8b840000 2023-07-14T19:23:47.115 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:23:47 smithi101 ceph-mon[128807]: rocksdb: DB pointer 0x564f8b82c000 2023-07-14T19:23:47.115 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:23:47 smithi101 ceph-mon[128807]: rocksdb: [db/db_impl/db_impl.cc:1109] ------- DUMPING STATS ------- 2023-07-14T19:23:47.115 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:23:47 smithi101 ceph-mon[128807]: rocksdb: [db/db_impl/db_impl.cc:1111] 2023-07-14T19:23:47.115 INFO:journalctl@ceph.mon.b.smithi101.stdout: ** DB Stats ** 2023-07-14T19:23:47.115 INFO:journalctl@ceph.mon.b.smithi101.stdout: Uptime(secs): 0.0 total, 0.0 interval 2023-07-14T19:23:47.115 INFO:journalctl@ceph.mon.b.smithi101.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2023-07-14T19:23:47.115 INFO:journalctl@ceph.mon.b.smithi101.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2023-07-14T19:23:47.116 INFO:journalctl@ceph.mon.b.smithi101.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2023-07-14T19:23:47.116 INFO:journalctl@ceph.mon.b.smithi101.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2023-07-14T19:23:47.116 INFO:journalctl@ceph.mon.b.smithi101.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2023-07-14T19:23:47.116 INFO:journalctl@ceph.mon.b.smithi101.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2023-07-14T19:23:47.116 INFO:journalctl@ceph.mon.b.smithi101.stdout: 2023-07-14T19:23:47.116 INFO:journalctl@ceph.mon.b.smithi101.stdout: ** Compaction Stats [default] ** 2023-07-14T19:23:47.116 INFO:journalctl@ceph.mon.b.smithi101.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2023-07-14T19:23:47.117 INFO:journalctl@ceph.mon.b.smithi101.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 2023-07-14T19:23:47.117 INFO:journalctl@ceph.mon.b.smithi101.stdout: L0 1/0 1.71 KB 0.2 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 4.2 0.00 0.00 1 0.000 0 0 0.0 0.0 2023-07-14T19:23:47.117 INFO:journalctl@ceph.mon.b.smithi101.stdout: Sum 1/0 1.71 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 4.2 0.00 0.00 1 0.000 0 0 0.0 0.0 2023-07-14T19:23:47.117 INFO:journalctl@ceph.mon.b.smithi101.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 4.2 0.00 0.00 1 0.000 0 0 0.0 0.0 2023-07-14T19:23:47.117 INFO:journalctl@ceph.mon.b.smithi101.stdout: 2023-07-14T19:23:47.117 INFO:journalctl@ceph.mon.b.smithi101.stdout: ** Compaction Stats [default] ** 2023-07-14T19:23:47.117 INFO:journalctl@ceph.mon.b.smithi101.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2023-07-14T19:23:47.117 INFO:journalctl@ceph.mon.b.smithi101.stdout: --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2023-07-14T19:23:47.118 INFO:journalctl@ceph.mon.b.smithi101.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 4.2 0.00 0.00 1 0.000 0 0 0.0 0.0 2023-07-14T19:23:47.118 INFO:journalctl@ceph.mon.b.smithi101.stdout: 2023-07-14T19:23:47.118 INFO:journalctl@ceph.mon.b.smithi101.stdout: Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 2023-07-14T19:23:47.118 INFO:journalctl@ceph.mon.b.smithi101.stdout: 2023-07-14T19:23:47.118 INFO:journalctl@ceph.mon.b.smithi101.stdout: Uptime(secs): 0.0 total, 0.0 interval 2023-07-14T19:23:47.118 INFO:journalctl@ceph.mon.b.smithi101.stdout: Flush(GB): cumulative 0.000, interval 0.000 2023-07-14T19:23:47.118 INFO:journalctl@ceph.mon.b.smithi101.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2023-07-14T19:23:47.119 INFO:journalctl@ceph.mon.b.smithi101.stdout: AddFile(Total Files): cumulative 0, interval 0 2023-07-14T19:23:47.119 INFO:journalctl@ceph.mon.b.smithi101.stdout: AddFile(L0 Files): cumulative 0, interval 0 2023-07-14T19:23:47.119 INFO:journalctl@ceph.mon.b.smithi101.stdout: AddFile(Keys): cumulative 0, interval 0 2023-07-14T19:23:47.119 INFO:journalctl@ceph.mon.b.smithi101.stdout: Cumulative compaction: 0.00 GB write, 0.71 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2023-07-14T19:23:47.119 INFO:journalctl@ceph.mon.b.smithi101.stdout: Interval compaction: 0.00 GB write, 0.71 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2023-07-14T19:23:47.119 INFO:journalctl@ceph.mon.b.smithi101.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2023-07-14T19:23:47.119 INFO:journalctl@ceph.mon.b.smithi101.stdout: Block cache BinnedLRUCache@0x564f8b7a71f0#8 capacity: 512.00 MB usage: 0.22 KB table_size: 0 occupancy: 18446744073709551615 collections: 1 last_copies: 0 last_secs: 9e-06 secs_since: 0 2023-07-14T19:23:47.119 INFO:journalctl@ceph.mon.b.smithi101.stdout: Block cache entry stats(count,size,portion): FilterBlock(1,0.11 KB,2.08616e-05%) IndexBlock(1,0.11 KB,2.08616e-05%) Misc(1,0.00 KB,0%) 2023-07-14T19:23:47.120 INFO:journalctl@ceph.mon.b.smithi101.stdout: 2023-07-14T19:23:47.120 INFO:journalctl@ceph.mon.b.smithi101.stdout: ** File Read Latency Histogram By Level [default] ** 2023-07-14T19:23:47.120 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:23:47 smithi101 ceph-mon[128807]: mon.b does not exist in monmap, will attempt to join an existing cluster 2023-07-14T19:23:47.120 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:23:47 smithi101 ceph-mon[128807]: using public_addr v2:172.21.15.101:0/0 -> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] 2023-07-14T19:23:47.120 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:23:47 smithi101 ceph-mon[128807]: starting mon.b rank -1 at public addrs [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] at bind addrs [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] mon_data /var/lib/ceph/mon/ceph-b fsid 9adea648-227b-11ee-9b33-001a4aab830c 2023-07-14T19:23:47.120 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:23:47 smithi101 ceph-mon[128807]: mon.b@-1(???) e0 preinit fsid 9adea648-227b-11ee-9b33-001a4aab830c 2023-07-14T19:23:47.120 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:23:47 smithi101 ceph-mon[128807]: mon.b@-1(synchronizing).mds e1 new map 2023-07-14T19:23:47.121 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:23:47 smithi101 ceph-mon[128807]: mon.b@-1(synchronizing).mds e1 print_map 2023-07-14T19:23:47.121 INFO:journalctl@ceph.mon.b.smithi101.stdout: e1 2023-07-14T19:23:47.121 INFO:journalctl@ceph.mon.b.smithi101.stdout: enable_multiple, ever_enabled_multiple: 1,1 2023-07-14T19:23:47.121 INFO:journalctl@ceph.mon.b.smithi101.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2023-07-14T19:23:47.121 INFO:journalctl@ceph.mon.b.smithi101.stdout: legacy client fscid: -1 2023-07-14T19:23:47.121 INFO:journalctl@ceph.mon.b.smithi101.stdout: 2023-07-14T19:23:47.121 INFO:journalctl@ceph.mon.b.smithi101.stdout: No filesystems configured 2023-07-14T19:23:47.121 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:23:47 smithi101 ceph-mon[128807]: mon.b@-1(synchronizing).osd e0 _set_cache_ratios kv ratio 0.25 inc ratio 0.375 full ratio 0.375 2023-07-14T19:23:47.122 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:23:47 smithi101 ceph-mon[128807]: mon.b@-1(synchronizing).osd e0 register_cache_with_pcm pcm target: 2147483648 pcm max: 1020054732 pcm min: 134217728 inc_osd_cache size: 1 2023-07-14T19:23:47.122 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:23:47 smithi101 ceph-mon[128807]: mon.b@-1(synchronizing).osd e1 e1: 0 total, 0 up, 0 in 2023-07-14T19:23:47.122 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:23:47 smithi101 ceph-mon[128807]: mon.b@-1(synchronizing).osd e2 e2: 0 total, 0 up, 0 in 2023-07-14T19:23:47.122 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:23:47 smithi101 ceph-mon[128807]: mon.b@-1(synchronizing).osd e3 e3: 0 total, 0 up, 0 in 2023-07-14T19:23:47.122 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:23:47 smithi101 ceph-mon[128807]: mon.b@-1(synchronizing).osd e4 e4: 0 total, 0 up, 0 in 2023-07-14T19:23:47.281 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:23:46 smithi098 ceph-mon[127486]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-14T19:23:47.442 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:23:47 smithi101 ceph-mon[128807]: mon.b@-1(synchronizing).osd e4 crush map has features 3314932999778484224, adjusting msgr requires 2023-07-14T19:23:47.442 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:23:47 smithi101 ceph-mon[128807]: mon.b@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2023-07-14T19:23:47.443 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:23:47 smithi101 ceph-mon[128807]: mon.b@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2023-07-14T19:23:47.443 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:23:47 smithi101 ceph-mon[128807]: mon.b@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2023-07-14T19:23:47.444 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:23:47 smithi101 ceph-mon[128807]: from='client.? 172.21.15.98:0/1384810344' entity='client.admin' cmd=[{"prefix": "osd crush tunables", "profile": "default"}]: dispatch 2023-07-14T19:23:47.444 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:23:47 smithi101 ceph-mon[128807]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-14T19:23:47.444 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:23:47 smithi101 ceph-mon[128807]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' 2023-07-14T19:23:47.445 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:23:47 smithi101 ceph-mon[128807]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' 2023-07-14T19:23:47.445 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:23:47 smithi101 ceph-mon[128807]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' 2023-07-14T19:23:47.445 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:23:47 smithi101 ceph-mon[128807]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' 2023-07-14T19:23:47.445 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:23:47 smithi101 ceph-mon[128807]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi139", "name": "osd_memory_target"}]: dispatch 2023-07-14T19:23:47.445 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:23:47 smithi101 ceph-mon[128807]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T19:23:47.446 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:23:47 smithi101 ceph-mon[128807]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-14T19:23:47.446 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:23:47 smithi101 ceph-mon[128807]: Updating smithi139:/etc/ceph/ceph.conf 2023-07-14T19:23:47.446 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:23:47 smithi101 ceph-mon[128807]: Updating smithi139:/var/lib/ceph/9adea648-227b-11ee-9b33-001a4aab830c/config/ceph.conf 2023-07-14T19:23:47.446 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:23:47 smithi101 ceph-mon[128807]: from='client.? 172.21.15.98:0/1384810344' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2023-07-14T19:23:47.446 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:23:47 smithi101 ceph-mon[128807]: osdmap e4: 0 total, 0 up, 0 in 2023-07-14T19:23:47.447 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:23:47 smithi101 ceph-mon[128807]: Updating smithi139:/etc/ceph/ceph.client.admin.keyring 2023-07-14T19:23:47.447 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:23:47 smithi101 ceph-mon[128807]: Updating smithi139:/var/lib/ceph/9adea648-227b-11ee-9b33-001a4aab830c/config/ceph.client.admin.keyring 2023-07-14T19:23:47.447 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:23:47 smithi101 ceph-mon[128807]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' 2023-07-14T19:23:47.447 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:23:47 smithi101 ceph-mon[128807]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' 2023-07-14T19:23:47.447 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:23:47 smithi101 ceph-mon[128807]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' 2023-07-14T19:23:47.447 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:23:47 smithi101 ceph-mon[128807]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-14T19:23:47.448 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:23:47 smithi101 ceph-mon[128807]: from='client.14188 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi098:172.21.15.98=a;smithi101:172.21.15.101=b;smithi139:172.21.15.139=c", "target": ["mon-mgr", ""]}]: dispatch 2023-07-14T19:23:47.448 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:23:47 smithi101 ceph-mon[128807]: Saving service mon spec with placement smithi098:172.21.15.98=a;smithi101:172.21.15.101=b;smithi139:172.21.15.139=c;count:3 2023-07-14T19:23:47.448 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:23:47 smithi101 ceph-mon[128807]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' 2023-07-14T19:23:47.448 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:23:47 smithi101 ceph-mon[128807]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-14T19:23:47.448 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:23:47 smithi101 ceph-mon[128807]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T19:23:47.448 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:23:47 smithi101 ceph-mon[128807]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-14T19:23:47.449 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:23:47 smithi101 ceph-mon[128807]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' 2023-07-14T19:23:47.449 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:23:47 smithi101 ceph-mon[128807]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2023-07-14T19:23:47.449 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:23:47 smithi101 ceph-mon[128807]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T19:23:47.449 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:23:47 smithi101 ceph-mon[128807]: Deploying daemon mon.c on smithi139 2023-07-14T19:23:47.449 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:23:47 smithi101 ceph-mon[128807]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-14T19:23:47.449 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:23:47 smithi101 ceph-mon[128807]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-14T19:23:47.449 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:23:47 smithi101 ceph-mon[128807]: from='client.? 172.21.15.139:0/4207913916' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-07-14T19:23:47.450 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:23:47 smithi101 ceph-mon[128807]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2023-07-14T19:23:47.450 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:23:47 smithi101 ceph-mon[128807]: mon.a calling monitor election 2023-07-14T19:23:47.450 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:23:47 smithi101 ceph-mon[128807]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-07-14T19:23:47.450 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:23:47 smithi101 ceph-mon[128807]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-14T19:23:47.450 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:23:47 smithi101 ceph-mon[128807]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-07-14T19:23:47.450 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:23:47 smithi101 ceph-mon[128807]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-07-14T19:23:47.450 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:23:47 smithi101 ceph-mon[128807]: mon.c calling monitor election 2023-07-14T19:23:47.451 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:23:47 smithi101 ceph-mon[128807]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-14T19:23:47.451 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:23:47 smithi101 ceph-mon[128807]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-07-14T19:23:47.451 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:23:47 smithi101 ceph-mon[128807]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-07-14T19:23:47.451 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:23:47 smithi101 ceph-mon[128807]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-07-14T19:23:47.451 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:23:47 smithi101 ceph-mon[128807]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2023-07-14T19:23:47.451 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:23:47 smithi101 ceph-mon[128807]: monmap e2: 2 mons at {a=[v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0],c=[v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0]} removed_ranks: {} 2023-07-14T19:23:47.451 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:23:47 smithi101 ceph-mon[128807]: fsmap 2023-07-14T19:23:47.452 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:23:47 smithi101 ceph-mon[128807]: osdmap e4: 0 total, 0 up, 0 in 2023-07-14T19:23:47.452 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:23:47 smithi101 ceph-mon[128807]: mgrmap e13: x(active, since 42s) 2023-07-14T19:23:47.452 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:23:47 smithi101 ceph-mon[128807]: overall HEALTH_OK 2023-07-14T19:23:47.452 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:23:47 smithi101 ceph-mon[128807]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' 2023-07-14T19:23:47.452 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:23:47 smithi101 ceph-mon[128807]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' 2023-07-14T19:23:47.452 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:23:47 smithi101 ceph-mon[128807]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-14T19:23:47.453 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:23:47 smithi101 ceph-mon[128807]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' 2023-07-14T19:23:47.453 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:23:47 smithi101 ceph-mon[128807]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2023-07-14T19:23:47.453 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:23:47 smithi101 ceph-mon[128807]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T19:23:47.453 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:23:47 smithi101 ceph-mon[128807]: Deploying daemon mon.b on smithi101 2023-07-14T19:23:47.453 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:23:47 smithi101 ceph-mon[128807]: from='client.? 172.21.15.139:0/2991630039' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-07-14T19:23:47.453 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:23:47 smithi101 ceph-mon[128807]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-07-14T19:23:47.453 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:23:47 smithi101 ceph-mon[128807]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-14T19:23:47.454 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:23:47 smithi101 ceph-mon[128807]: mon.b@-1(synchronizing).paxosservice(auth 1..3) refresh upgraded, format 0 -> 3 2023-07-14T19:23:47.454 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:23:47 smithi101 ceph-mon[128807]: mon.b@-1(synchronizing) e2 handle_conf_change mon_allow_pool_delete 2023-07-14T19:23:47.454 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:23:47 smithi101 ceph-mon[128807]: expand_channel_meta expand map: {default=false} 2023-07-14T19:23:47.454 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:23:47 smithi101 ceph-mon[128807]: expand_channel_meta from 'false' to 'false' 2023-07-14T19:23:47.454 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:23:47 smithi101 ceph-mon[128807]: expand_channel_meta expanded map: {default=false} 2023-07-14T19:23:47.454 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:23:47 smithi101 ceph-mon[128807]: expand_channel_meta expand map: {default=info} 2023-07-14T19:23:47.454 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:23:47 smithi101 ceph-mon[128807]: expand_channel_meta from 'info' to 'info' 2023-07-14T19:23:47.454 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:23:47 smithi101 ceph-mon[128807]: expand_channel_meta expanded map: {default=info} 2023-07-14T19:23:47.455 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:23:47 smithi101 ceph-mon[128807]: expand_channel_meta expand map: {default=daemon} 2023-07-14T19:23:47.455 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:23:47 smithi101 ceph-mon[128807]: expand_channel_meta from 'daemon' to 'daemon' 2023-07-14T19:23:47.455 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:23:47 smithi101 ceph-mon[128807]: expand_channel_meta expanded map: {default=daemon} 2023-07-14T19:23:47.455 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:23:47 smithi101 ceph-mon[128807]: expand_channel_meta expand map: {default=debug} 2023-07-14T19:23:47.455 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:23:47 smithi101 ceph-mon[128807]: expand_channel_meta from 'debug' to 'debug' 2023-07-14T19:23:47.455 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:23:47 smithi101 ceph-mon[128807]: expand_channel_meta expanded map: {default=debug} 2023-07-14T19:23:47.455 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:23:47 smithi101 systemd[1]: Started Ceph mon.b for 9adea648-227b-11ee-9b33-001a4aab830c. 2023-07-14T19:23:52.189 INFO:teuthology.orchestra.run.smithi139.stderr:2023-07-14T19:23:52.187+0000 7f0d9d398700 1 -- 172.21.15.139:0/1919385001 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0d98102f20 msgr2=0x7f0d80005610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:23:52.189 INFO:teuthology.orchestra.run.smithi139.stderr:2023-07-14T19:23:52.187+0000 7f0d9d398700 1 --2- 172.21.15.139:0/1919385001 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0d98102f20 0x7f0d80005610 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f0d8c005fa0 tx=0x7f0d8c010f80 comp rx=0 tx=0).stop 2023-07-14T19:23:52.189 INFO:teuthology.orchestra.run.smithi139.stderr:2023-07-14T19:23:52.187+0000 7f0d9d398700 1 -- 172.21.15.139:0/1919385001 shutdown_connections 2023-07-14T19:23:52.189 INFO:teuthology.orchestra.run.smithi139.stderr:2023-07-14T19:23:52.187+0000 7f0d9d398700 1 --2- 172.21.15.139:0/1919385001 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0d98102f20 0x7f0d80005610 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:23:52.189 INFO:teuthology.orchestra.run.smithi139.stderr:2023-07-14T19:23:52.187+0000 7f0d9d398700 1 -- 172.21.15.139:0/1919385001 >> 172.21.15.139:0/1919385001 conn(0x7f0d98073ea0 msgr2=0x7f0d980742a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:23:52.190 INFO:teuthology.orchestra.run.smithi139.stderr:2023-07-14T19:23:52.187+0000 7f0d9d398700 1 -- 172.21.15.139:0/1919385001 shutdown_connections 2023-07-14T19:23:52.190 INFO:teuthology.orchestra.run.smithi139.stderr:2023-07-14T19:23:52.187+0000 7f0d9d398700 1 -- 172.21.15.139:0/1919385001 wait complete. 2023-07-14T19:23:52.191 INFO:teuthology.orchestra.run.smithi139.stderr:2023-07-14T19:23:52.188+0000 7f0d9d398700 1 Processor -- start 2023-07-14T19:23:52.192 INFO:teuthology.orchestra.run.smithi139.stderr:2023-07-14T19:23:52.189+0000 7f0d9d398700 1 -- start start 2023-07-14T19:23:52.192 INFO:teuthology.orchestra.run.smithi139.stderr:2023-07-14T19:23:52.189+0000 7f0d9d398700 1 --2- >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f0d98102f20 0x7f0d98198970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:23:52.192 INFO:teuthology.orchestra.run.smithi139.stderr:2023-07-14T19:23:52.189+0000 7f0d9d398700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0d98198eb0 0x7f0d981929f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:23:52.192 INFO:teuthology.orchestra.run.smithi139.stderr:2023-07-14T19:23:52.189+0000 7f0d9d398700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f0d98192f30 0x7f0d981933d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:23:52.192 INFO:teuthology.orchestra.run.smithi139.stderr:2023-07-14T19:23:52.189+0000 7f0d9d398700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0d981994c0 con 0x7f0d98198eb0 2023-07-14T19:23:52.193 INFO:teuthology.orchestra.run.smithi139.stderr:2023-07-14T19:23:52.189+0000 7f0d9d398700 1 -- --> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0d9819f790 con 0x7f0d98102f20 2023-07-14T19:23:52.193 INFO:teuthology.orchestra.run.smithi139.stderr:2023-07-14T19:23:52.189+0000 7f0d9d398700 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0d9819f8d0 con 0x7f0d98192f30 2023-07-14T19:23:52.193 INFO:teuthology.orchestra.run.smithi139.stderr:2023-07-14T19:23:52.189+0000 7f0d977fe700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f0d98192f30 0x7f0d981933d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:23:52.193 INFO:teuthology.orchestra.run.smithi139.stderr:2023-07-14T19:23:52.189+0000 7f0d977fe700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f0d98192f30 0x7f0d981933d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.139:3300/0 says I am v2:172.21.15.139:37050/0 (socket says 172.21.15.139:37050) 2023-07-14T19:23:52.193 INFO:teuthology.orchestra.run.smithi139.stderr:2023-07-14T19:23:52.189+0000 7f0d977fe700 1 -- 172.21.15.139:0/1782633071 learned_addr learned my addr 172.21.15.139:0/1782633071 (peer_addr_for_me v2:172.21.15.139:0/0) 2023-07-14T19:23:52.193 INFO:teuthology.orchestra.run.smithi139.stderr:2023-07-14T19:23:52.189+0000 7f0d967fc700 1 --2- 172.21.15.139:0/1782633071 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0d98198eb0 0x7f0d981929f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:23:52.193 INFO:teuthology.orchestra.run.smithi139.stderr:2023-07-14T19:23:52.190+0000 7f0d96ffd700 1 --2- 172.21.15.139:0/1782633071 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f0d98102f20 0x7f0d98198970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:23:52.194 INFO:teuthology.orchestra.run.smithi139.stderr:2023-07-14T19:23:52.190+0000 7f0d977fe700 1 -- 172.21.15.139:0/1782633071 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f0d98192f30 msgr2=0x7f0d981933d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 14 2023-07-14T19:23:52.194 INFO:teuthology.orchestra.run.smithi139.stderr:2023-07-14T19:23:52.190+0000 7f0d977fe700 1 -- 172.21.15.139:0/1782633071 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f0d98192f30 msgr2=0x7f0d981933d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2023-07-14T19:23:52.194 INFO:teuthology.orchestra.run.smithi139.stderr:2023-07-14T19:23:52.190+0000 7f0d977fe700 1 --2- 172.21.15.139:0/1782633071 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f0d98192f30 0x7f0d981933d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-07-14T19:23:52.194 INFO:teuthology.orchestra.run.smithi139.stderr:2023-07-14T19:23:52.190+0000 7f0d977fe700 1 --2- 172.21.15.139:0/1782633071 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f0d98192f30 0x7f0d981933d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-07-14T19:23:52.194 INFO:teuthology.orchestra.run.smithi139.stderr:2023-07-14T19:23:52.190+0000 7f0d96ffd700 1 -- 172.21.15.139:0/1782633071 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f0d98102f20 msgr2=0x7f0d98198970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 12 2023-07-14T19:23:52.194 INFO:teuthology.orchestra.run.smithi139.stderr:2023-07-14T19:23:52.190+0000 7f0d96ffd700 1 -- 172.21.15.139:0/1782633071 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f0d98102f20 msgr2=0x7f0d98198970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2023-07-14T19:23:52.195 INFO:teuthology.orchestra.run.smithi139.stderr:2023-07-14T19:23:52.190+0000 7f0d96ffd700 1 --2- 172.21.15.139:0/1782633071 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f0d98102f20 0x7f0d98198970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-07-14T19:23:52.195 INFO:teuthology.orchestra.run.smithi139.stderr:2023-07-14T19:23:52.190+0000 7f0d967fc700 1 -- 172.21.15.139:0/1782633071 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f0d98192f30 msgr2=0x7f0d981933d0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-07-14T19:23:52.195 INFO:teuthology.orchestra.run.smithi139.stderr:2023-07-14T19:23:52.190+0000 7f0d967fc700 1 --2- 172.21.15.139:0/1782633071 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f0d98192f30 0x7f0d981933d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:23:52.195 INFO:teuthology.orchestra.run.smithi139.stderr:2023-07-14T19:23:52.190+0000 7f0d96ffd700 1 --2- 172.21.15.139:0/1782633071 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f0d98102f20 0x7f0d98198970 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-07-14T19:23:52.195 INFO:teuthology.orchestra.run.smithi139.stderr:2023-07-14T19:23:52.190+0000 7f0d967fc700 1 -- 172.21.15.139:0/1782633071 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f0d98102f20 msgr2=0x7f0d98198970 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-07-14T19:23:52.195 INFO:teuthology.orchestra.run.smithi139.stderr:2023-07-14T19:23:52.190+0000 7f0d967fc700 1 --2- 172.21.15.139:0/1782633071 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f0d98102f20 0x7f0d98198970 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:23:52.196 INFO:teuthology.orchestra.run.smithi139.stderr:2023-07-14T19:23:52.190+0000 7f0d967fc700 1 -- 172.21.15.139:0/1782633071 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0d8c010980 con 0x7f0d98198eb0 2023-07-14T19:23:52.196 INFO:teuthology.orchestra.run.smithi139.stderr:2023-07-14T19:23:52.190+0000 7f0d967fc700 1 --2- 172.21.15.139:0/1782633071 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0d98198eb0 0x7f0d981929f0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f0d8800b550 tx=0x7f0d8800b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:23:52.196 INFO:teuthology.orchestra.run.smithi139.stderr:2023-07-14T19:23:52.190+0000 7f0d7ffff700 1 -- 172.21.15.139:0/1782633071 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0d88004230 con 0x7f0d98198eb0 2023-07-14T19:23:52.196 INFO:teuthology.orchestra.run.smithi139.stderr:2023-07-14T19:23:52.190+0000 7f0d7ffff700 1 -- 172.21.15.139:0/1782633071 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f0d88004390 con 0x7f0d98198eb0 2023-07-14T19:23:52.196 INFO:teuthology.orchestra.run.smithi139.stderr:2023-07-14T19:23:52.190+0000 7f0d9d398700 1 -- 172.21.15.139:0/1782633071 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0d9819fbb0 con 0x7f0d98198eb0 2023-07-14T19:23:52.196 INFO:teuthology.orchestra.run.smithi139.stderr:2023-07-14T19:23:52.190+0000 7f0d7ffff700 1 -- 172.21.15.139:0/1782633071 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0d88011af0 con 0x7f0d98198eb0 2023-07-14T19:23:52.197 INFO:teuthology.orchestra.run.smithi139.stderr:2023-07-14T19:23:52.190+0000 7f0d9d398700 1 -- 172.21.15.139:0/1782633071 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0d981a0100 con 0x7f0d98198eb0 2023-07-14T19:23:52.197 INFO:teuthology.orchestra.run.smithi139.stderr:2023-07-14T19:23:52.191+0000 7f0d7ffff700 1 -- 172.21.15.139:0/1782633071 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 13) v1 ==== 45539+0+0 (secure 0 0 0) 0x7f0d88011c50 con 0x7f0d98198eb0 2023-07-14T19:23:52.197 INFO:teuthology.orchestra.run.smithi139.stderr:2023-07-14T19:23:52.191+0000 7f0d7ffff700 1 --2- 172.21.15.139:0/1782633071 >> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] conn(0x7f0d84038c30 0x7f0d8403b0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:23:52.198 INFO:teuthology.orchestra.run.smithi139.stderr:2023-07-14T19:23:52.191+0000 7f0d7ffff700 1 -- 172.21.15.139:0/1782633071 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f0d8804ad50 con 0x7f0d98198eb0 2023-07-14T19:23:52.198 INFO:teuthology.orchestra.run.smithi139.stderr:2023-07-14T19:23:52.192+0000 7f0d9d398700 1 -- 172.21.15.139:0/1782633071 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0d98061c70 con 0x7f0d98198eb0 2023-07-14T19:23:52.198 INFO:teuthology.orchestra.run.smithi139.stderr:2023-07-14T19:23:52.192+0000 7f0d96ffd700 1 --2- 172.21.15.139:0/1782633071 >> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] conn(0x7f0d84038c30 0x7f0d8403b0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:23:52.198 INFO:teuthology.orchestra.run.smithi139.stderr:2023-07-14T19:23:52.192+0000 7f0d96ffd700 1 --2- 172.21.15.139:0/1782633071 >> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] conn(0x7f0d84038c30 0x7f0d8403b0e0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f0d8c000c00 tx=0x7f0d8c013bc0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:23:52.199 INFO:teuthology.orchestra.run.smithi139.stderr:2023-07-14T19:23:52.198+0000 7f0d7ffff700 1 -- 172.21.15.139:0/1782633071 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f0d8801a400 con 0x7f0d98198eb0 2023-07-14T19:23:52.470 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:23:52 smithi139 ceph-mon[129293]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2023-07-14T19:23:52.471 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:23:52 smithi139 ceph-mon[129293]: mon.a calling monitor election 2023-07-14T19:23:52.471 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:23:52 smithi139 ceph-mon[129293]: mon.c calling monitor election 2023-07-14T19:23:52.471 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:23:52 smithi139 ceph-mon[129293]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-07-14T19:23:52.471 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:23:52 smithi139 ceph-mon[129293]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-07-14T19:23:52.471 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:23:52 smithi139 ceph-mon[129293]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-14T19:23:52.471 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:23:52 smithi139 ceph-mon[129293]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-07-14T19:23:52.472 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:23:52 smithi139 ceph-mon[129293]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-07-14T19:23:52.472 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:23:52 smithi139 ceph-mon[129293]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-14T19:23:52.472 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:23:52 smithi139 ceph-mon[129293]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-07-14T19:23:52.472 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:23:52 smithi139 ceph-mon[129293]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-07-14T19:23:52.472 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:23:52 smithi139 ceph-mon[129293]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-14T19:23:52.472 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:23:52 smithi139 ceph-mon[129293]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-07-14T19:23:52.472 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:23:52 smithi139 ceph-mon[129293]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2023-07-14T19:23:52.473 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:23:52 smithi139 ceph-mon[129293]: monmap e3: 3 mons at {a=[v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0],b=[v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0],c=[v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0]} removed_ranks: {} 2023-07-14T19:23:52.473 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:23:52 smithi139 ceph-mon[129293]: fsmap 2023-07-14T19:23:52.473 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:23:52 smithi139 ceph-mon[129293]: osdmap e4: 0 total, 0 up, 0 in 2023-07-14T19:23:52.473 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:23:52 smithi139 ceph-mon[129293]: mgrmap e13: x(active, since 50s) 2023-07-14T19:23:52.473 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:23:52 smithi139 ceph-mon[129293]: Health check failed: 1/3 mons down, quorum a,c (MON_DOWN) 2023-07-14T19:23:52.473 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:23:52 smithi139 ceph-mon[129293]: Health detail: HEALTH_WARN 1/3 mons down, quorum a,c 2023-07-14T19:23:52.473 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:23:52 smithi139 ceph-mon[129293]: [WRN] MON_DOWN: 1/3 mons down, quorum a,c 2023-07-14T19:23:52.473 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:23:52 smithi139 ceph-mon[129293]: mon.b (rank 2) addr [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] is down (out of quorum) 2023-07-14T19:23:52.474 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:23:52 smithi139 ceph-mon[129293]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' 2023-07-14T19:23:52.474 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:23:52 smithi139 ceph-mon[129293]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' 2023-07-14T19:23:52.474 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:23:52 smithi139 ceph-mon[129293]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' 2023-07-14T19:23:52.474 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:23:52 smithi139 ceph-mon[129293]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' 2023-07-14T19:23:52.475 INFO:teuthology.orchestra.run.smithi139.stderr:2023-07-14T19:23:52.474+0000 7f0d9d398700 1 -- 172.21.15.139:0/1782633071 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f0d981a0410 con 0x7f0d98198eb0 2023-07-14T19:23:52.475 INFO:teuthology.orchestra.run.smithi139.stderr:2023-07-14T19:23:52.475+0000 7f0d7ffff700 1 -- 172.21.15.139:0/1782633071 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 3 v3) v1 ==== 95+0+1274 (secure 0 0 0) 0x7f0d88048020 con 0x7f0d98198eb0 2023-07-14T19:23:52.476 INFO:teuthology.orchestra.run.smithi139.stdout: 2023-07-14T19:23:52.476 INFO:teuthology.orchestra.run.smithi139.stdout:{"epoch":3,"fsid":"9adea648-227b-11ee-9b33-001a4aab830c","modified":"2023-07-14T19:23:47.088833Z","created":"2023-07-14T19:22:04.850090Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:3300","nonce":0},{"type":"v1","addr":"172.21.15.98:6789","nonce":0}]},"addr":"172.21.15.98:6789/0","public_addr":"172.21.15.98:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:3300","nonce":0},{"type":"v1","addr":"172.21.15.139:6789","nonce":0}]},"addr":"172.21.15.139:6789/0","public_addr":"172.21.15.139:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.101:3300","nonce":0},{"type":"v1","addr":"172.21.15.101:6789","nonce":0}]},"addr":"172.21.15.101:6789/0","public_addr":"172.21.15.101:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0,1]} 2023-07-14T19:23:52.478 INFO:teuthology.orchestra.run.smithi139.stderr:2023-07-14T19:23:52.477+0000 7f0d9d398700 1 -- 172.21.15.139:0/1782633071 >> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] conn(0x7f0d84038c30 msgr2=0x7f0d8403b0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:23:52.478 INFO:teuthology.orchestra.run.smithi139.stderr:2023-07-14T19:23:52.477+0000 7f0d9d398700 1 --2- 172.21.15.139:0/1782633071 >> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] conn(0x7f0d84038c30 0x7f0d8403b0e0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f0d8c000c00 tx=0x7f0d8c013bc0 comp rx=0 tx=0).stop 2023-07-14T19:23:52.478 INFO:teuthology.orchestra.run.smithi139.stderr:2023-07-14T19:23:52.477+0000 7f0d9d398700 1 -- 172.21.15.139:0/1782633071 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0d98198eb0 msgr2=0x7f0d981929f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:23:52.479 INFO:teuthology.orchestra.run.smithi139.stderr:2023-07-14T19:23:52.477+0000 7f0d9d398700 1 --2- 172.21.15.139:0/1782633071 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0d98198eb0 0x7f0d981929f0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f0d8800b550 tx=0x7f0d8800b910 comp rx=0 tx=0).stop 2023-07-14T19:23:52.479 INFO:teuthology.orchestra.run.smithi139.stderr:2023-07-14T19:23:52.477+0000 7f0d9d398700 1 -- 172.21.15.139:0/1782633071 shutdown_connections 2023-07-14T19:23:52.480 INFO:teuthology.orchestra.run.smithi139.stderr:2023-07-14T19:23:52.477+0000 7f0d9d398700 1 --2- 172.21.15.139:0/1782633071 >> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] conn(0x7f0d84038c30 0x7f0d8403b0e0 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:23:52.480 INFO:teuthology.orchestra.run.smithi139.stderr:2023-07-14T19:23:52.477+0000 7f0d9d398700 1 --2- 172.21.15.139:0/1782633071 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f0d98192f30 0x7f0d981933d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:23:52.480 INFO:teuthology.orchestra.run.smithi139.stderr:2023-07-14T19:23:52.477+0000 7f0d9d398700 1 --2- 172.21.15.139:0/1782633071 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f0d98102f20 0x7f0d98198970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:23:52.480 INFO:teuthology.orchestra.run.smithi139.stderr:2023-07-14T19:23:52.477+0000 7f0d9d398700 1 --2- 172.21.15.139:0/1782633071 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0d98198eb0 0x7f0d981929f0 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:23:52.480 INFO:teuthology.orchestra.run.smithi139.stderr:2023-07-14T19:23:52.477+0000 7f0d9d398700 1 -- 172.21.15.139:0/1782633071 >> 172.21.15.139:0/1782633071 conn(0x7f0d98073ea0 msgr2=0x7f0d9818ba40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:23:52.481 INFO:teuthology.orchestra.run.smithi139.stderr:2023-07-14T19:23:52.477+0000 7f0d9d398700 1 -- 172.21.15.139:0/1782633071 shutdown_connections 2023-07-14T19:23:52.481 INFO:teuthology.orchestra.run.smithi139.stderr:2023-07-14T19:23:52.478+0000 7f0d9d398700 1 -- 172.21.15.139:0/1782633071 wait complete. 2023-07-14T19:23:52.481 INFO:teuthology.orchestra.run.smithi139.stderr:dumped monmap epoch 3 2023-07-14T19:23:52.531 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:23:52 smithi098 ceph-mon[127486]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2023-07-14T19:23:52.532 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:23:52 smithi098 ceph-mon[127486]: mon.a calling monitor election 2023-07-14T19:23:52.532 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:23:52 smithi098 ceph-mon[127486]: mon.c calling monitor election 2023-07-14T19:23:52.532 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:23:52 smithi098 ceph-mon[127486]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-07-14T19:23:52.532 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:23:52 smithi098 ceph-mon[127486]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-07-14T19:23:52.532 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:23:52 smithi098 ceph-mon[127486]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-14T19:23:52.532 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:23:52 smithi098 ceph-mon[127486]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-07-14T19:23:52.533 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:23:52 smithi098 ceph-mon[127486]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-07-14T19:23:52.533 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:23:52 smithi098 ceph-mon[127486]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-14T19:23:52.533 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:23:52 smithi098 ceph-mon[127486]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-07-14T19:23:52.533 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:23:52 smithi098 ceph-mon[127486]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-07-14T19:23:52.533 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:23:52 smithi098 ceph-mon[127486]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-14T19:23:52.533 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:23:52 smithi098 ceph-mon[127486]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-07-14T19:23:52.533 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:23:52 smithi098 ceph-mon[127486]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2023-07-14T19:23:52.533 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:23:52 smithi098 ceph-mon[127486]: monmap e3: 3 mons at {a=[v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0],b=[v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0],c=[v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0]} removed_ranks: {} 2023-07-14T19:23:52.534 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:23:52 smithi098 ceph-mon[127486]: fsmap 2023-07-14T19:23:52.534 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:23:52 smithi098 ceph-mon[127486]: osdmap e4: 0 total, 0 up, 0 in 2023-07-14T19:23:52.534 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:23:52 smithi098 ceph-mon[127486]: mgrmap e13: x(active, since 50s) 2023-07-14T19:23:52.534 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:23:52 smithi098 ceph-mon[127486]: Health check failed: 1/3 mons down, quorum a,c (MON_DOWN) 2023-07-14T19:23:52.534 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:23:52 smithi098 ceph-mon[127486]: Health detail: HEALTH_WARN 1/3 mons down, quorum a,c 2023-07-14T19:23:52.534 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:23:52 smithi098 ceph-mon[127486]: [WRN] MON_DOWN: 1/3 mons down, quorum a,c 2023-07-14T19:23:52.534 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:23:52 smithi098 ceph-mon[127486]: mon.b (rank 2) addr [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] is down (out of quorum) 2023-07-14T19:23:52.535 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:23:52 smithi098 ceph-mon[127486]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' 2023-07-14T19:23:52.535 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:23:52 smithi098 ceph-mon[127486]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' 2023-07-14T19:23:52.535 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:23:52 smithi098 ceph-mon[127486]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' 2023-07-14T19:23:52.535 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:23:52 smithi098 ceph-mon[127486]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' 2023-07-14T19:23:52.865 INFO:tasks.cephadm:Generating final ceph.conf file... 2023-07-14T19:23:52.865 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81c9387d2e7576da065747ec9c16d1c7d69016ec shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 9adea648-227b-11ee-9b33-001a4aab830c -- ceph config generate-minimal-conf 2023-07-14T19:23:53.431 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/9adea648-227b-11ee-9b33-001a4aab830c/mon.a/config 2023-07-14T19:23:53.470 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:23:53 smithi139 ceph-mon[129293]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-14T19:23:53.470 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:23:53 smithi139 ceph-mon[129293]: from='client.? 172.21.15.139:0/1782633071' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-07-14T19:23:53.471 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:23:53 smithi139 ceph-mon[129293]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-07-14T19:23:53.531 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:23:53 smithi098 ceph-mon[127486]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-14T19:23:53.532 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:23:53 smithi098 ceph-mon[127486]: from='client.? 172.21.15.139:0/1782633071' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-07-14T19:23:53.532 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:23:53 smithi098 ceph-mon[127486]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-07-14T19:23:54.441 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:23:54 smithi101 ceph-mon[128807]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2023-07-14T19:23:54.442 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:23:54 smithi101 ceph-mon[128807]: mon.a calling monitor election 2023-07-14T19:23:54.442 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:23:54 smithi101 ceph-mon[128807]: mon.c calling monitor election 2023-07-14T19:23:54.442 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:23:54 smithi101 ceph-mon[128807]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-07-14T19:23:54.442 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:23:54 smithi101 ceph-mon[128807]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-07-14T19:23:54.442 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:23:54 smithi101 ceph-mon[128807]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-14T19:23:54.443 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:23:54 smithi101 ceph-mon[128807]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-07-14T19:23:54.443 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:23:54 smithi101 ceph-mon[128807]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-07-14T19:23:54.443 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:23:54 smithi101 ceph-mon[128807]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-14T19:23:54.443 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:23:54 smithi101 ceph-mon[128807]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-07-14T19:23:54.443 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:23:54 smithi101 ceph-mon[128807]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-07-14T19:23:54.443 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:23:54 smithi101 ceph-mon[128807]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-14T19:23:54.443 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:23:54 smithi101 ceph-mon[128807]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-07-14T19:23:54.444 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:23:54 smithi101 ceph-mon[128807]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2023-07-14T19:23:54.444 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:23:54 smithi101 ceph-mon[128807]: monmap e3: 3 mons at {a=[v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0],b=[v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0],c=[v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0]} removed_ranks: {} 2023-07-14T19:23:54.444 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:23:54 smithi101 ceph-mon[128807]: fsmap 2023-07-14T19:23:54.444 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:23:54 smithi101 ceph-mon[128807]: osdmap e4: 0 total, 0 up, 0 in 2023-07-14T19:23:54.444 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:23:54 smithi101 ceph-mon[128807]: mgrmap e13: x(active, since 50s) 2023-07-14T19:23:54.445 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:23:54 smithi101 ceph-mon[128807]: Health check failed: 1/3 mons down, quorum a,c (MON_DOWN) 2023-07-14T19:23:54.445 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:23:54 smithi101 ceph-mon[128807]: Health detail: HEALTH_WARN 1/3 mons down, quorum a,c 2023-07-14T19:23:54.445 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:23:54 smithi101 ceph-mon[128807]: [WRN] MON_DOWN: 1/3 mons down, quorum a,c 2023-07-14T19:23:54.445 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:23:54 smithi101 ceph-mon[128807]: mon.b (rank 2) addr [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] is down (out of quorum) 2023-07-14T19:23:54.445 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:23:54 smithi101 ceph-mon[128807]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' 2023-07-14T19:23:54.445 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:23:54 smithi101 ceph-mon[128807]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' 2023-07-14T19:23:54.445 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:23:54 smithi101 ceph-mon[128807]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' 2023-07-14T19:23:54.446 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:23:54 smithi101 ceph-mon[128807]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' 2023-07-14T19:23:54.446 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:23:54 smithi101 ceph-mon[128807]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-14T19:23:54.446 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:23:54 smithi101 ceph-mon[128807]: from='client.? 172.21.15.139:0/1782633071' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-07-14T19:23:54.446 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:23:54 smithi101 ceph-mon[128807]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-07-14T19:23:54.742 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:23:54.739+0000 7f8b03569700 1 -- 172.21.15.98:0/2211017713 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8afc0fe580 msgr2=0x7f8afc0fe950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:23:54.743 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:23:54.739+0000 7f8b03569700 1 --2- 172.21.15.98:0/2211017713 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8afc0fe580 0x7f8afc0fe950 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f8af8009910 tx=0x7f8af8009c20 comp rx=0 tx=0).stop 2023-07-14T19:23:54.743 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:23:54.739+0000 7f8b03569700 1 -- 172.21.15.98:0/2211017713 shutdown_connections 2023-07-14T19:23:54.743 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:23:54.739+0000 7f8b03569700 1 --2- 172.21.15.98:0/2211017713 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8afc0fe580 0x7f8afc0fe950 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:23:54.743 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:23:54.739+0000 7f8b03569700 1 -- 172.21.15.98:0/2211017713 >> 172.21.15.98:0/2211017713 conn(0x7f8afc075040 msgr2=0x7f8afc075440 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:23:54.743 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:23:54.739+0000 7f8b03569700 1 -- 172.21.15.98:0/2211017713 shutdown_connections 2023-07-14T19:23:54.743 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:23:54.739+0000 7f8b03569700 1 -- 172.21.15.98:0/2211017713 wait complete. 2023-07-14T19:23:54.744 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:23:54.740+0000 7f8b03569700 1 Processor -- start 2023-07-14T19:23:54.744 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:23:54.740+0000 7f8b03569700 1 -- start start 2023-07-14T19:23:54.744 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:23:54.740+0000 7f8b03569700 1 --2- >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f8afc0fe580 0x7f8afc194cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:23:54.744 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:23:54.740+0000 7f8b03569700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8afc195210 0x7f8afc1985d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:23:54.744 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:23:54.740+0000 7f8b03569700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f8afc198b10 0x7f8afc198f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:23:54.744 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:23:54.740+0000 7f8b03569700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8afc1958b0 con 0x7f8afc195210 2023-07-14T19:23:54.744 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:23:54.740+0000 7f8b03569700 1 -- --> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8afc1a1960 con 0x7f8afc0fe580 2023-07-14T19:23:54.744 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:23:54.740+0000 7f8b03569700 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8afc1a1aa0 con 0x7f8afc198b10 2023-07-14T19:23:54.745 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:23:54.740+0000 7f8b00b04700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8afc195210 0x7f8afc1985d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:23:54.745 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:23:54.740+0000 7f8b00b04700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8afc195210 0x7f8afc1985d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:44744/0 (socket says 172.21.15.98:44744) 2023-07-14T19:23:54.745 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:23:54.740+0000 7f8b00b04700 1 -- 172.21.15.98:0/2407443534 learned_addr learned my addr 172.21.15.98:0/2407443534 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-07-14T19:23:54.745 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:23:54.740+0000 7f8b01305700 1 --2- 172.21.15.98:0/2407443534 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f8afc0fe580 0x7f8afc194cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:23:54.745 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:23:54.740+0000 7f8b01b06700 1 --2- 172.21.15.98:0/2407443534 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f8afc198b10 0x7f8afc198f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:23:54.745 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:23:54.740+0000 7f8b00b04700 1 -- 172.21.15.98:0/2407443534 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f8afc198b10 msgr2=0x7f8afc198f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:23:54.745 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:23:54.740+0000 7f8b00b04700 1 --2- 172.21.15.98:0/2407443534 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f8afc198b10 0x7f8afc198f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:23:54.746 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:23:54.740+0000 7f8b00b04700 1 -- 172.21.15.98:0/2407443534 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f8afc0fe580 msgr2=0x7f8afc194cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:23:54.746 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:23:54.740+0000 7f8b00b04700 1 --2- 172.21.15.98:0/2407443534 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f8afc0fe580 0x7f8afc194cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:23:54.746 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:23:54.740+0000 7f8b00b04700 1 -- 172.21.15.98:0/2407443534 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8af80095f0 con 0x7f8afc195210 2023-07-14T19:23:54.746 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:23:54.740+0000 7f8b01b06700 1 --2- 172.21.15.98:0/2407443534 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f8afc198b10 0x7f8afc198f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-07-14T19:23:54.746 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:23:54.740+0000 7f8b00b04700 1 --2- 172.21.15.98:0/2407443534 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8afc195210 0x7f8afc1985d0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f8aec00b780 tx=0x7f8aec00ba90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:23:54.746 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:23:54.741+0000 7f8af27fc700 1 -- 172.21.15.98:0/2407443534 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8aec004230 con 0x7f8afc195210 2023-07-14T19:23:54.746 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:23:54.741+0000 7f8b03569700 1 -- 172.21.15.98:0/2407443534 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8afc1a1d30 con 0x7f8afc195210 2023-07-14T19:23:54.747 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:23:54.741+0000 7f8b03569700 1 -- 172.21.15.98:0/2407443534 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8afc1a22a0 con 0x7f8afc195210 2023-07-14T19:23:54.747 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:23:54.741+0000 7f8af27fc700 1 -- 172.21.15.98:0/2407443534 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f8aec004390 con 0x7f8afc195210 2023-07-14T19:23:54.747 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:23:54.741+0000 7f8af27fc700 1 -- 172.21.15.98:0/2407443534 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8aec011ad0 con 0x7f8afc195210 2023-07-14T19:23:54.748 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:23:54.742+0000 7f8b03569700 1 -- 172.21.15.98:0/2407443534 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8ae0005320 con 0x7f8afc195210 2023-07-14T19:23:54.748 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:23:54.742+0000 7f8af27fc700 1 -- 172.21.15.98:0/2407443534 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 13) v1 ==== 45539+0+0 (secure 0 0 0) 0x7f8aec011cd0 con 0x7f8afc195210 2023-07-14T19:23:54.748 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:23:54.742+0000 7f8af27fc700 1 --2- 172.21.15.98:0/2407443534 >> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] conn(0x7f8ae8038c30 0x7f8ae803b0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:23:54.748 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:23:54.742+0000 7f8af27fc700 1 -- 172.21.15.98:0/2407443534 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f8aec04ac80 con 0x7f8afc195210 2023-07-14T19:23:54.748 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:23:54.742+0000 7f8b01305700 1 --2- 172.21.15.98:0/2407443534 >> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] conn(0x7f8ae8038c30 0x7f8ae803b0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:23:54.748 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:23:54.743+0000 7f8b01305700 1 --2- 172.21.15.98:0/2407443534 >> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] conn(0x7f8ae8038c30 0x7f8ae803b0e0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f8af8006010 tx=0x7f8af800b540 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:23:54.753 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:23:54.751+0000 7f8af27fc700 1 -- 172.21.15.98:0/2407443534 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f8aec01ae40 con 0x7f8afc195210 2023-07-14T19:23:54.934 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:23:54 smithi098 ceph-mon[127486]: mon.b calling monitor election 2023-07-14T19:23:54.935 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:23:54 smithi098 ceph-mon[127486]: mon.b calling monitor election 2023-07-14T19:23:54.935 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:23:54 smithi098 ceph-mon[127486]: mon.a calling monitor election 2023-07-14T19:23:54.935 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:23:54 smithi098 ceph-mon[127486]: mon.c calling monitor election 2023-07-14T19:23:54.936 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:23:54 smithi098 ceph-mon[127486]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2023-07-14T19:23:54.936 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:23:54 smithi098 ceph-mon[127486]: monmap e3: 3 mons at {a=[v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0],b=[v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0],c=[v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0]} removed_ranks: {} 2023-07-14T19:23:54.936 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:23:54 smithi098 ceph-mon[127486]: fsmap 2023-07-14T19:23:54.936 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:23:54 smithi098 ceph-mon[127486]: osdmap e4: 0 total, 0 up, 0 in 2023-07-14T19:23:54.936 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:23:54 smithi098 ceph-mon[127486]: mgrmap e13: x(active, since 52s) 2023-07-14T19:23:54.936 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:23:54 smithi098 ceph-mon[127486]: Health check cleared: MON_DOWN (was: 1/3 mons down, quorum a,c) 2023-07-14T19:23:54.936 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:23:54 smithi098 ceph-mon[127486]: Cluster is now healthy 2023-07-14T19:23:54.937 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:23:54 smithi098 ceph-mon[127486]: overall HEALTH_OK 2023-07-14T19:23:54.937 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:23:54 smithi098 ceph-mon[127486]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' 2023-07-14T19:23:54.937 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:23:54 smithi098 ceph-mon[127486]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' 2023-07-14T19:23:54.937 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:23:54 smithi098 ceph-mon[127486]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' 2023-07-14T19:23:54.937 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:23:54 smithi098 ceph-mon[127486]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' 2023-07-14T19:23:54.937 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:23:54 smithi098 ceph-mon[127486]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' 2023-07-14T19:23:54.937 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:23:54 smithi098 ceph-mon[127486]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' 2023-07-14T19:23:54.938 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:23:54 smithi098 ceph-mon[127486]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' 2023-07-14T19:23:54.938 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:23:54 smithi098 ceph-mon[127486]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2023-07-14T19:23:54.938 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:23:54 smithi098 ceph-mon[127486]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2023-07-14T19:23:54.938 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:23:54 smithi098 ceph-mon[127486]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T19:23:54.951 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:23:54.948+0000 7f8b03569700 1 -- 172.21.15.98:0/2407443534 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "config generate-minimal-conf"} v 0) v1 -- 0x7f8ae0005190 con 0x7f8afc195210 2023-07-14T19:23:54.951 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:23:54.949+0000 7f8af27fc700 1 -- 172.21.15.98:0/2407443534 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "config generate-minimal-conf"}]=0 v9) v1 ==== 76+0+275 (secure 0 0 0) 0x7f8aec0212f0 con 0x7f8afc195210 2023-07-14T19:23:54.952 INFO:teuthology.orchestra.run.smithi098.stdout:# minimal ceph.conf for 9adea648-227b-11ee-9b33-001a4aab830c 2023-07-14T19:23:54.952 INFO:teuthology.orchestra.run.smithi098.stdout:[global] 2023-07-14T19:23:54.952 INFO:teuthology.orchestra.run.smithi098.stdout: fsid = 9adea648-227b-11ee-9b33-001a4aab830c 2023-07-14T19:23:54.952 INFO:teuthology.orchestra.run.smithi098.stdout: mon_host = [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] 2023-07-14T19:23:54.953 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:23:54.951+0000 7f8b03569700 1 -- 172.21.15.98:0/2407443534 >> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] conn(0x7f8ae8038c30 msgr2=0x7f8ae803b0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:23:54.953 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:23:54.951+0000 7f8b03569700 1 --2- 172.21.15.98:0/2407443534 >> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] conn(0x7f8ae8038c30 0x7f8ae803b0e0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f8af8006010 tx=0x7f8af800b540 comp rx=0 tx=0).stop 2023-07-14T19:23:54.953 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:23:54.951+0000 7f8b03569700 1 -- 172.21.15.98:0/2407443534 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8afc195210 msgr2=0x7f8afc1985d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:23:54.953 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:23:54.951+0000 7f8b03569700 1 --2- 172.21.15.98:0/2407443534 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8afc195210 0x7f8afc1985d0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f8aec00b780 tx=0x7f8aec00ba90 comp rx=0 tx=0).stop 2023-07-14T19:23:54.954 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:23:54.951+0000 7f8b03569700 1 -- 172.21.15.98:0/2407443534 shutdown_connections 2023-07-14T19:23:54.954 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:23:54.951+0000 7f8b03569700 1 --2- 172.21.15.98:0/2407443534 >> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] conn(0x7f8ae8038c30 0x7f8ae803b0e0 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:23:54.954 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:23:54.951+0000 7f8b03569700 1 --2- 172.21.15.98:0/2407443534 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f8afc198b10 0x7f8afc198f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:23:54.954 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:23:54.951+0000 7f8b03569700 1 --2- 172.21.15.98:0/2407443534 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f8afc0fe580 0x7f8afc194cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:23:54.954 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:23:54.951+0000 7f8b03569700 1 --2- 172.21.15.98:0/2407443534 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8afc195210 0x7f8afc1985d0 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:23:54.954 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:23:54.951+0000 7f8b03569700 1 -- 172.21.15.98:0/2407443534 >> 172.21.15.98:0/2407443534 conn(0x7f8afc075040 msgr2=0x7f8afc104160 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:23:54.954 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:23:54.951+0000 7f8b03569700 1 -- 172.21.15.98:0/2407443534 shutdown_connections 2023-07-14T19:23:54.955 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:23:54.951+0000 7f8b03569700 1 -- 172.21.15.98:0/2407443534 wait complete. 2023-07-14T19:23:55.191 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:23:54 smithi101 ceph-mon[128807]: mon.b calling monitor election 2023-07-14T19:23:55.192 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:23:54 smithi101 ceph-mon[128807]: mon.b calling monitor election 2023-07-14T19:23:55.192 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:23:54 smithi101 ceph-mon[128807]: mon.a calling monitor election 2023-07-14T19:23:55.192 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:23:54 smithi101 ceph-mon[128807]: mon.c calling monitor election 2023-07-14T19:23:55.192 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:23:54 smithi101 ceph-mon[128807]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2023-07-14T19:23:55.192 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:23:54 smithi101 ceph-mon[128807]: monmap e3: 3 mons at {a=[v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0],b=[v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0],c=[v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0]} removed_ranks: {} 2023-07-14T19:23:55.193 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:23:54 smithi101 ceph-mon[128807]: fsmap 2023-07-14T19:23:55.193 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:23:54 smithi101 ceph-mon[128807]: osdmap e4: 0 total, 0 up, 0 in 2023-07-14T19:23:55.193 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:23:54 smithi101 ceph-mon[128807]: mgrmap e13: x(active, since 52s) 2023-07-14T19:23:55.193 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:23:54 smithi101 ceph-mon[128807]: Health check cleared: MON_DOWN (was: 1/3 mons down, quorum a,c) 2023-07-14T19:23:55.193 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:23:54 smithi101 ceph-mon[128807]: Cluster is now healthy 2023-07-14T19:23:55.193 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:23:54 smithi101 ceph-mon[128807]: overall HEALTH_OK 2023-07-14T19:23:55.193 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:23:54 smithi101 ceph-mon[128807]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' 2023-07-14T19:23:55.193 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:23:54 smithi101 ceph-mon[128807]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' 2023-07-14T19:23:55.194 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:23:54 smithi101 ceph-mon[128807]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' 2023-07-14T19:23:55.194 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:23:54 smithi101 ceph-mon[128807]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' 2023-07-14T19:23:55.194 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:23:54 smithi101 ceph-mon[128807]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' 2023-07-14T19:23:55.194 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:23:54 smithi101 ceph-mon[128807]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' 2023-07-14T19:23:55.194 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:23:54 smithi101 ceph-mon[128807]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' 2023-07-14T19:23:55.194 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:23:54 smithi101 ceph-mon[128807]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2023-07-14T19:23:55.194 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:23:54 smithi101 ceph-mon[128807]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2023-07-14T19:23:55.195 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:23:54 smithi101 ceph-mon[128807]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T19:23:55.220 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:23:54 smithi139 ceph-mon[129293]: mon.b calling monitor election 2023-07-14T19:23:55.221 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:23:54 smithi139 ceph-mon[129293]: mon.b calling monitor election 2023-07-14T19:23:55.221 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:23:54 smithi139 ceph-mon[129293]: mon.a calling monitor election 2023-07-14T19:23:55.221 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:23:54 smithi139 ceph-mon[129293]: mon.c calling monitor election 2023-07-14T19:23:55.221 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:23:54 smithi139 ceph-mon[129293]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2023-07-14T19:23:55.221 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:23:54 smithi139 ceph-mon[129293]: monmap e3: 3 mons at {a=[v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0],b=[v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0],c=[v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0]} removed_ranks: {} 2023-07-14T19:23:55.222 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:23:54 smithi139 ceph-mon[129293]: fsmap 2023-07-14T19:23:55.222 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:23:54 smithi139 ceph-mon[129293]: osdmap e4: 0 total, 0 up, 0 in 2023-07-14T19:23:55.222 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:23:54 smithi139 ceph-mon[129293]: mgrmap e13: x(active, since 52s) 2023-07-14T19:23:55.222 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:23:54 smithi139 ceph-mon[129293]: Health check cleared: MON_DOWN (was: 1/3 mons down, quorum a,c) 2023-07-14T19:23:55.222 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:23:54 smithi139 ceph-mon[129293]: Cluster is now healthy 2023-07-14T19:23:55.222 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:23:54 smithi139 ceph-mon[129293]: overall HEALTH_OK 2023-07-14T19:23:55.222 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:23:54 smithi139 ceph-mon[129293]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' 2023-07-14T19:23:55.222 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:23:54 smithi139 ceph-mon[129293]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' 2023-07-14T19:23:55.223 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:23:54 smithi139 ceph-mon[129293]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' 2023-07-14T19:23:55.223 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:23:54 smithi139 ceph-mon[129293]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' 2023-07-14T19:23:55.223 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:23:54 smithi139 ceph-mon[129293]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' 2023-07-14T19:23:55.223 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:23:54 smithi139 ceph-mon[129293]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' 2023-07-14T19:23:55.223 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:23:54 smithi139 ceph-mon[129293]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' 2023-07-14T19:23:55.223 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:23:54 smithi139 ceph-mon[129293]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2023-07-14T19:23:55.223 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:23:54 smithi139 ceph-mon[129293]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2023-07-14T19:23:55.224 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:23:54 smithi139 ceph-mon[129293]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T19:23:55.476 INFO:tasks.cephadm:Distributing (final) config and client.admin keyring... 2023-07-14T19:23:55.477 DEBUG:teuthology.orchestra.run.smithi098:> set -ex 2023-07-14T19:23:55.477 DEBUG:teuthology.orchestra.run.smithi098:> sudo dd of=/etc/ceph/ceph.conf 2023-07-14T19:23:55.505 DEBUG:teuthology.orchestra.run.smithi098:> set -ex 2023-07-14T19:23:55.506 DEBUG:teuthology.orchestra.run.smithi098:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2023-07-14T19:23:55.572 DEBUG:teuthology.orchestra.run.smithi101:> set -ex 2023-07-14T19:23:55.572 DEBUG:teuthology.orchestra.run.smithi101:> sudo dd of=/etc/ceph/ceph.conf 2023-07-14T19:23:55.610 DEBUG:teuthology.orchestra.run.smithi101:> set -ex 2023-07-14T19:23:55.610 DEBUG:teuthology.orchestra.run.smithi101:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2023-07-14T19:23:55.679 DEBUG:teuthology.orchestra.run.smithi139:> set -ex 2023-07-14T19:23:55.680 DEBUG:teuthology.orchestra.run.smithi139:> sudo dd of=/etc/ceph/ceph.conf 2023-07-14T19:23:55.717 DEBUG:teuthology.orchestra.run.smithi139:> set -ex 2023-07-14T19:23:55.717 DEBUG:teuthology.orchestra.run.smithi139:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2023-07-14T19:23:55.789 INFO:tasks.cephadm:Adding mgr.x on smithi098 2023-07-14T19:23:55.789 INFO:tasks.cephadm:Adding mgr.y on smithi101 2023-07-14T19:23:55.790 INFO:tasks.cephadm:Adding mgr.z on smithi139 2023-07-14T19:23:55.790 DEBUG:teuthology.orchestra.run.smithi139:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81c9387d2e7576da065747ec9c16d1c7d69016ec shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 9adea648-227b-11ee-9b33-001a4aab830c -- ceph orch apply mgr '3;smithi098=x;smithi101=y;smithi139=z' 2023-07-14T19:23:55.942 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:23:55 smithi101 ceph-mon[128807]: Updating smithi101:/var/lib/ceph/9adea648-227b-11ee-9b33-001a4aab830c/config/ceph.conf 2023-07-14T19:23:55.942 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:23:55 smithi101 ceph-mon[128807]: Updating smithi098:/var/lib/ceph/9adea648-227b-11ee-9b33-001a4aab830c/config/ceph.conf 2023-07-14T19:23:55.942 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:23:55 smithi101 ceph-mon[128807]: Updating smithi139:/var/lib/ceph/9adea648-227b-11ee-9b33-001a4aab830c/config/ceph.conf 2023-07-14T19:23:55.942 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:23:55 smithi101 ceph-mon[128807]: Reconfiguring mon.a (unknown last config time)... 2023-07-14T19:23:55.942 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:23:55 smithi101 ceph-mon[128807]: Reconfiguring daemon mon.a on smithi098 2023-07-14T19:23:55.943 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:23:55 smithi101 ceph-mon[128807]: from='client.? 172.21.15.98:0/2407443534' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T19:23:55.943 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:23:55 smithi101 ceph-mon[128807]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-07-14T19:23:55.943 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:23:55 smithi101 ceph-mon[128807]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' 2023-07-14T19:23:55.943 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:23:55 smithi101 ceph-mon[128807]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' 2023-07-14T19:23:55.943 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:23:55 smithi101 ceph-mon[128807]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2023-07-14T19:23:55.943 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:23:55 smithi101 ceph-mon[128807]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2023-07-14T19:23:55.943 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:23:55 smithi101 ceph-mon[128807]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T19:23:55.970 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:23:55 smithi139 ceph-mon[129293]: Updating smithi101:/var/lib/ceph/9adea648-227b-11ee-9b33-001a4aab830c/config/ceph.conf 2023-07-14T19:23:55.971 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:23:55 smithi139 ceph-mon[129293]: Updating smithi098:/var/lib/ceph/9adea648-227b-11ee-9b33-001a4aab830c/config/ceph.conf 2023-07-14T19:23:55.971 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:23:55 smithi139 ceph-mon[129293]: Updating smithi139:/var/lib/ceph/9adea648-227b-11ee-9b33-001a4aab830c/config/ceph.conf 2023-07-14T19:23:55.971 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:23:55 smithi139 ceph-mon[129293]: Reconfiguring mon.a (unknown last config time)... 2023-07-14T19:23:55.971 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:23:55 smithi139 ceph-mon[129293]: Reconfiguring daemon mon.a on smithi098 2023-07-14T19:23:55.971 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:23:55 smithi139 ceph-mon[129293]: from='client.? 172.21.15.98:0/2407443534' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T19:23:55.971 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:23:55 smithi139 ceph-mon[129293]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-07-14T19:23:55.972 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:23:55 smithi139 ceph-mon[129293]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' 2023-07-14T19:23:55.972 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:23:55 smithi139 ceph-mon[129293]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' 2023-07-14T19:23:55.972 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:23:55 smithi139 ceph-mon[129293]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2023-07-14T19:23:55.972 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:23:55 smithi139 ceph-mon[129293]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2023-07-14T19:23:55.972 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:23:55 smithi139 ceph-mon[129293]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T19:23:56.281 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:23:55 smithi098 ceph-mon[127486]: Updating smithi101:/var/lib/ceph/9adea648-227b-11ee-9b33-001a4aab830c/config/ceph.conf 2023-07-14T19:23:56.281 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:23:55 smithi098 ceph-mon[127486]: Updating smithi098:/var/lib/ceph/9adea648-227b-11ee-9b33-001a4aab830c/config/ceph.conf 2023-07-14T19:23:56.282 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:23:55 smithi098 ceph-mon[127486]: Updating smithi139:/var/lib/ceph/9adea648-227b-11ee-9b33-001a4aab830c/config/ceph.conf 2023-07-14T19:23:56.282 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:23:55 smithi098 ceph-mon[127486]: Reconfiguring mon.a (unknown last config time)... 2023-07-14T19:23:56.282 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:23:55 smithi098 ceph-mon[127486]: Reconfiguring daemon mon.a on smithi098 2023-07-14T19:23:56.282 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:23:55 smithi098 ceph-mon[127486]: from='client.? 172.21.15.98:0/2407443534' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T19:23:56.282 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:23:55 smithi098 ceph-mon[127486]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-07-14T19:23:56.282 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:23:55 smithi098 ceph-mon[127486]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' 2023-07-14T19:23:56.282 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:23:55 smithi098 ceph-mon[127486]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' 2023-07-14T19:23:56.282 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:23:55 smithi098 ceph-mon[127486]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2023-07-14T19:23:56.283 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:23:55 smithi098 ceph-mon[127486]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2023-07-14T19:23:56.283 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:23:55 smithi098 ceph-mon[127486]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T19:23:56.378 INFO:teuthology.orchestra.run.smithi139.stderr:Inferring config /var/lib/ceph/9adea648-227b-11ee-9b33-001a4aab830c/mon.c/config 2023-07-14T19:23:56.906 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:23:56 smithi139 ceph-mon[129293]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-14T19:23:56.906 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:23:56 smithi139 ceph-mon[129293]: Reconfiguring mon.b (monmap changed)... 2023-07-14T19:23:56.907 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:23:56 smithi139 ceph-mon[129293]: Reconfiguring daemon mon.b on smithi101 2023-07-14T19:23:56.907 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:23:56 smithi139 ceph-mon[129293]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' 2023-07-14T19:23:56.944 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:23:56 smithi101 ceph-mon[128807]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-14T19:23:56.945 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:23:56 smithi101 ceph-mon[128807]: Reconfiguring mon.b (monmap changed)... 2023-07-14T19:23:56.945 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:23:56 smithi101 ceph-mon[128807]: Reconfiguring daemon mon.b on smithi101 2023-07-14T19:23:56.945 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:23:56 smithi101 ceph-mon[128807]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' 2023-07-14T19:23:57.281 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:23:56 smithi098 ceph-mon[127486]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-14T19:23:57.282 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:23:56 smithi098 ceph-mon[127486]: Reconfiguring mon.b (monmap changed)... 2023-07-14T19:23:57.282 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:23:56 smithi098 ceph-mon[127486]: Reconfiguring daemon mon.b on smithi101 2023-07-14T19:23:57.282 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:23:56 smithi098 ceph-mon[127486]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' 2023-07-14T19:23:57.646 INFO:teuthology.orchestra.run.smithi139.stderr:2023-07-14T19:23:57.645+0000 7f871ad2e700 1 -- 172.21.15.139:0/1520612353 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f87141002c0 msgr2=0x7f87141026a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:23:57.646 INFO:teuthology.orchestra.run.smithi139.stderr:2023-07-14T19:23:57.645+0000 7f871ad2e700 1 --2- 172.21.15.139:0/1520612353 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f87141002c0 0x7f87141026a0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f8708009910 tx=0x7f8708009c20 comp rx=0 tx=0).stop 2023-07-14T19:23:57.646 INFO:teuthology.orchestra.run.smithi139.stderr:2023-07-14T19:23:57.645+0000 7f871ad2e700 1 -- 172.21.15.139:0/1520612353 shutdown_connections 2023-07-14T19:23:57.647 INFO:teuthology.orchestra.run.smithi139.stderr:2023-07-14T19:23:57.645+0000 7f871ad2e700 1 --2- 172.21.15.139:0/1520612353 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f87141002c0 0x7f87141026a0 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:23:57.647 INFO:teuthology.orchestra.run.smithi139.stderr:2023-07-14T19:23:57.645+0000 7f871ad2e700 1 -- 172.21.15.139:0/1520612353 >> 172.21.15.139:0/1520612353 conn(0x7f87140f9ce0 msgr2=0x7f87140fc110 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:23:57.647 INFO:teuthology.orchestra.run.smithi139.stderr:2023-07-14T19:23:57.645+0000 7f871ad2e700 1 -- 172.21.15.139:0/1520612353 shutdown_connections 2023-07-14T19:23:57.647 INFO:teuthology.orchestra.run.smithi139.stderr:2023-07-14T19:23:57.645+0000 7f871ad2e700 1 -- 172.21.15.139:0/1520612353 wait complete. 2023-07-14T19:23:57.650 INFO:teuthology.orchestra.run.smithi139.stderr:2023-07-14T19:23:57.646+0000 7f871ad2e700 1 Processor -- start 2023-07-14T19:23:57.650 INFO:teuthology.orchestra.run.smithi139.stderr:2023-07-14T19:23:57.646+0000 7f871ad2e700 1 -- start start 2023-07-14T19:23:57.650 INFO:teuthology.orchestra.run.smithi139.stderr:2023-07-14T19:23:57.646+0000 7f871ad2e700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f87141002c0 0x7f87140ffc20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:23:57.650 INFO:teuthology.orchestra.run.smithi139.stderr:2023-07-14T19:23:57.646+0000 7f871ad2e700 1 --2- >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f87140fe270 0x7f87140fe6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:23:57.650 INFO:teuthology.orchestra.run.smithi139.stderr:2023-07-14T19:23:57.646+0000 7f871ad2e700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f87140fec20 0x7f871419eb20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:23:57.651 INFO:teuthology.orchestra.run.smithi139.stderr:2023-07-14T19:23:57.646+0000 7f871ad2e700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f871419f060 con 0x7f87141002c0 2023-07-14T19:23:57.651 INFO:teuthology.orchestra.run.smithi139.stderr:2023-07-14T19:23:57.646+0000 7f871ad2e700 1 -- --> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f871419f1a0 con 0x7f87140fe270 2023-07-14T19:23:57.651 INFO:teuthology.orchestra.run.smithi139.stderr:2023-07-14T19:23:57.646+0000 7f871ad2e700 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f871419f2e0 con 0x7f87140fec20 2023-07-14T19:23:57.651 INFO:teuthology.orchestra.run.smithi139.stderr:2023-07-14T19:23:57.646+0000 7f87192cb700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f87140fec20 0x7f871419eb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:23:57.651 INFO:teuthology.orchestra.run.smithi139.stderr:2023-07-14T19:23:57.646+0000 7f87192cb700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f87140fec20 0x7f871419eb20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.139:3300/0 says I am v2:172.21.15.139:37060/0 (socket says 172.21.15.139:37060) 2023-07-14T19:23:57.651 INFO:teuthology.orchestra.run.smithi139.stderr:2023-07-14T19:23:57.646+0000 7f87192cb700 1 -- 172.21.15.139:0/1476874183 learned_addr learned my addr 172.21.15.139:0/1476874183 (peer_addr_for_me v2:172.21.15.139:0/0) 2023-07-14T19:23:57.652 INFO:teuthology.orchestra.run.smithi139.stderr:2023-07-14T19:23:57.646+0000 7f8718aca700 1 --2- 172.21.15.139:0/1476874183 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f87141002c0 0x7f87140ffc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:23:57.652 INFO:teuthology.orchestra.run.smithi139.stderr:2023-07-14T19:23:57.646+0000 7f8713fff700 1 --2- 172.21.15.139:0/1476874183 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f87140fe270 0x7f87140fe6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:23:57.652 INFO:teuthology.orchestra.run.smithi139.stderr:2023-07-14T19:23:57.646+0000 7f87192cb700 1 -- 172.21.15.139:0/1476874183 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f87140fe270 msgr2=0x7f87140fe6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:23:57.652 INFO:teuthology.orchestra.run.smithi139.stderr:2023-07-14T19:23:57.646+0000 7f87192cb700 1 --2- 172.21.15.139:0/1476874183 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f87140fe270 0x7f87140fe6e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:23:57.652 INFO:teuthology.orchestra.run.smithi139.stderr:2023-07-14T19:23:57.646+0000 7f87192cb700 1 -- 172.21.15.139:0/1476874183 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f87141002c0 msgr2=0x7f87140ffc20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:23:57.653 INFO:teuthology.orchestra.run.smithi139.stderr:2023-07-14T19:23:57.646+0000 7f87192cb700 1 --2- 172.21.15.139:0/1476874183 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f87141002c0 0x7f87140ffc20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:23:57.653 INFO:teuthology.orchestra.run.smithi139.stderr:2023-07-14T19:23:57.646+0000 7f87192cb700 1 -- 172.21.15.139:0/1476874183 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f87080095f0 con 0x7f87140fec20 2023-07-14T19:23:57.653 INFO:teuthology.orchestra.run.smithi139.stderr:2023-07-14T19:23:57.647+0000 7f87192cb700 1 --2- 172.21.15.139:0/1476874183 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f87140fec20 0x7f871419eb20 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f870c00ba50 tx=0x7f870c00bd60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:23:57.653 INFO:teuthology.orchestra.run.smithi139.stderr:2023-07-14T19:23:57.647+0000 7f8711ffb700 1 -- 172.21.15.139:0/1476874183 <== mon.1 v2:172.21.15.139:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f870c00c930 con 0x7f87140fec20 2023-07-14T19:23:57.655 INFO:teuthology.orchestra.run.smithi139.stderr:2023-07-14T19:23:57.647+0000 7f8711ffb700 1 -- 172.21.15.139:0/1476874183 <== mon.1 v2:172.21.15.139:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f870c014440 con 0x7f87140fec20 2023-07-14T19:23:57.655 INFO:teuthology.orchestra.run.smithi139.stderr:2023-07-14T19:23:57.647+0000 7f8711ffb700 1 -- 172.21.15.139:0/1476874183 <== mon.1 v2:172.21.15.139:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f870c0075b0 con 0x7f87140fec20 2023-07-14T19:23:57.655 INFO:teuthology.orchestra.run.smithi139.stderr:2023-07-14T19:23:57.647+0000 7f871ad2e700 1 -- 172.21.15.139:0/1476874183 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f871419f5c0 con 0x7f87140fec20 2023-07-14T19:23:57.655 INFO:teuthology.orchestra.run.smithi139.stderr:2023-07-14T19:23:57.647+0000 7f871ad2e700 1 -- 172.21.15.139:0/1476874183 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f871419faf0 con 0x7f87140fec20 2023-07-14T19:23:57.655 INFO:teuthology.orchestra.run.smithi139.stderr:2023-07-14T19:23:57.648+0000 7f8711ffb700 1 -- 172.21.15.139:0/1476874183 <== mon.1 v2:172.21.15.139:3300/0 4 ==== mgrmap(e 13) v1 ==== 45539+0+0 (secure 0 0 0) 0x7f870c00ca90 con 0x7f87140fec20 2023-07-14T19:23:57.655 INFO:teuthology.orchestra.run.smithi139.stderr:2023-07-14T19:23:57.648+0000 7f8711ffb700 1 --2- 172.21.15.139:0/1476874183 >> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] conn(0x7f8704038c80 0x7f870403b130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:23:57.656 INFO:teuthology.orchestra.run.smithi139.stderr:2023-07-14T19:23:57.648+0000 7f871ad2e700 1 -- 172.21.15.139:0/1476874183 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f871418e400 con 0x7f87140fec20 2023-07-14T19:23:57.656 INFO:teuthology.orchestra.run.smithi139.stderr:2023-07-14T19:23:57.648+0000 7f8711ffb700 1 -- 172.21.15.139:0/1476874183 <== mon.1 v2:172.21.15.139:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f870c04a990 con 0x7f87140fec20 2023-07-14T19:23:57.656 INFO:teuthology.orchestra.run.smithi139.stderr:2023-07-14T19:23:57.648+0000 7f8718aca700 1 --2- 172.21.15.139:0/1476874183 >> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] conn(0x7f8704038c80 0x7f870403b130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:23:57.656 INFO:teuthology.orchestra.run.smithi139.stderr:2023-07-14T19:23:57.649+0000 7f8718aca700 1 --2- 172.21.15.139:0/1476874183 >> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] conn(0x7f8704038c80 0x7f870403b130 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f8708005950 tx=0x7f87080058e0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:23:57.657 INFO:teuthology.orchestra.run.smithi139.stderr:2023-07-14T19:23:57.653+0000 7f8711ffb700 1 -- 172.21.15.139:0/1476874183 <== mon.1 v2:172.21.15.139:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f870c0148b0 con 0x7f87140fec20 2023-07-14T19:23:57.860 INFO:teuthology.orchestra.run.smithi139.stderr:2023-07-14T19:23:57.860+0000 7f871ad2e700 1 -- 172.21.15.139:0/1476874183 --> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mgr", "placement": "3;smithi098=x;smithi101=y;smithi139=z", "target": ["mon-mgr", ""]}) v1 -- 0x7f8714060b10 con 0x7f8704038c80 2023-07-14T19:23:57.867 INFO:teuthology.orchestra.run.smithi139.stderr:2023-07-14T19:23:57.866+0000 7f8711ffb700 1 -- 172.21.15.139:0/1476874183 <== mgr.14152 v2:172.21.15.98:6800/452042164 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+24 (secure 0 0 0) 0x7f8714060b10 con 0x7f8704038c80 2023-07-14T19:23:57.867 INFO:teuthology.orchestra.run.smithi139.stdout:Scheduled mgr update... 2023-07-14T19:23:57.870 INFO:teuthology.orchestra.run.smithi139.stderr:2023-07-14T19:23:57.869+0000 7f871ad2e700 1 -- 172.21.15.139:0/1476874183 >> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] conn(0x7f8704038c80 msgr2=0x7f870403b130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:23:57.870 INFO:teuthology.orchestra.run.smithi139.stderr:2023-07-14T19:23:57.869+0000 7f871ad2e700 1 --2- 172.21.15.139:0/1476874183 >> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] conn(0x7f8704038c80 0x7f870403b130 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f8708005950 tx=0x7f87080058e0 comp rx=0 tx=0).stop 2023-07-14T19:23:57.870 INFO:teuthology.orchestra.run.smithi139.stderr:2023-07-14T19:23:57.869+0000 7f871ad2e700 1 -- 172.21.15.139:0/1476874183 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f87140fec20 msgr2=0x7f871419eb20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:23:57.870 INFO:teuthology.orchestra.run.smithi139.stderr:2023-07-14T19:23:57.869+0000 7f871ad2e700 1 --2- 172.21.15.139:0/1476874183 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f87140fec20 0x7f871419eb20 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f870c00ba50 tx=0x7f870c00bd60 comp rx=0 tx=0).stop 2023-07-14T19:23:57.871 INFO:teuthology.orchestra.run.smithi139.stderr:2023-07-14T19:23:57.869+0000 7f871ad2e700 1 -- 172.21.15.139:0/1476874183 shutdown_connections 2023-07-14T19:23:57.871 INFO:teuthology.orchestra.run.smithi139.stderr:2023-07-14T19:23:57.869+0000 7f871ad2e700 1 --2- 172.21.15.139:0/1476874183 >> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] conn(0x7f8704038c80 0x7f870403b130 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:23:57.871 INFO:teuthology.orchestra.run.smithi139.stderr:2023-07-14T19:23:57.869+0000 7f871ad2e700 1 --2- 172.21.15.139:0/1476874183 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f87140fec20 0x7f871419eb20 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:23:57.871 INFO:teuthology.orchestra.run.smithi139.stderr:2023-07-14T19:23:57.869+0000 7f871ad2e700 1 --2- 172.21.15.139:0/1476874183 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f87141002c0 0x7f87140ffc20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:23:57.871 INFO:teuthology.orchestra.run.smithi139.stderr:2023-07-14T19:23:57.869+0000 7f871ad2e700 1 --2- 172.21.15.139:0/1476874183 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f87140fe270 0x7f87140fe6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:23:57.871 INFO:teuthology.orchestra.run.smithi139.stderr:2023-07-14T19:23:57.869+0000 7f871ad2e700 1 -- 172.21.15.139:0/1476874183 >> 172.21.15.139:0/1476874183 conn(0x7f87140f9ce0 msgr2=0x7f87140fa990 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:23:57.872 INFO:teuthology.orchestra.run.smithi139.stderr:2023-07-14T19:23:57.869+0000 7f871ad2e700 1 -- 172.21.15.139:0/1476874183 shutdown_connections 2023-07-14T19:23:57.872 INFO:teuthology.orchestra.run.smithi139.stderr:2023-07-14T19:23:57.869+0000 7f871ad2e700 1 -- 172.21.15.139:0/1476874183 wait complete. 2023-07-14T19:23:58.192 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:23:57 smithi101 ceph-mon[128807]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' 2023-07-14T19:23:58.192 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:23:57 smithi101 ceph-mon[128807]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' 2023-07-14T19:23:58.192 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:23:57 smithi101 ceph-mon[128807]: Reconfiguring mon.c (monmap changed)... 2023-07-14T19:23:58.192 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:23:57 smithi101 ceph-mon[128807]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2023-07-14T19:23:58.192 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:23:57 smithi101 ceph-mon[128807]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2023-07-14T19:23:58.192 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:23:57 smithi101 ceph-mon[128807]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T19:23:58.193 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:23:57 smithi101 ceph-mon[128807]: Reconfiguring daemon mon.c on smithi139 2023-07-14T19:23:58.193 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:23:57 smithi101 ceph-mon[128807]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' 2023-07-14T19:23:58.220 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:23:57 smithi139 ceph-mon[129293]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' 2023-07-14T19:23:58.227 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:23:57 smithi139 ceph-mon[129293]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' 2023-07-14T19:23:58.227 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:23:57 smithi139 ceph-mon[129293]: Reconfiguring mon.c (monmap changed)... 2023-07-14T19:23:58.227 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:23:57 smithi139 ceph-mon[129293]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2023-07-14T19:23:58.227 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:23:57 smithi139 ceph-mon[129293]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2023-07-14T19:23:58.228 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:23:57 smithi139 ceph-mon[129293]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T19:23:58.228 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:23:57 smithi139 ceph-mon[129293]: Reconfiguring daemon mon.c on smithi139 2023-07-14T19:23:58.228 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:23:57 smithi139 ceph-mon[129293]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' 2023-07-14T19:23:58.281 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:23:57 smithi098 ceph-mon[127486]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' 2023-07-14T19:23:58.282 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:23:57 smithi098 ceph-mon[127486]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' 2023-07-14T19:23:58.282 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:23:57 smithi098 ceph-mon[127486]: Reconfiguring mon.c (monmap changed)... 2023-07-14T19:23:58.282 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:23:57 smithi098 ceph-mon[127486]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2023-07-14T19:23:58.282 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:23:57 smithi098 ceph-mon[127486]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2023-07-14T19:23:58.282 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:23:57 smithi098 ceph-mon[127486]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T19:23:58.283 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:23:57 smithi098 ceph-mon[127486]: Reconfiguring daemon mon.c on smithi139 2023-07-14T19:23:58.283 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:23:57 smithi098 ceph-mon[127486]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' 2023-07-14T19:23:58.421 DEBUG:teuthology.orchestra.run.smithi101:mgr.y> sudo journalctl -f -n 0 -u ceph-9adea648-227b-11ee-9b33-001a4aab830c@mgr.y.service 2023-07-14T19:23:58.424 DEBUG:teuthology.orchestra.run.smithi139:mgr.z> sudo journalctl -f -n 0 -u ceph-9adea648-227b-11ee-9b33-001a4aab830c@mgr.z.service 2023-07-14T19:23:58.426 INFO:tasks.cephadm:Deploying OSDs... 2023-07-14T19:23:58.427 DEBUG:teuthology.orchestra.run.smithi098:> set -ex 2023-07-14T19:23:58.427 DEBUG:teuthology.orchestra.run.smithi098:> dd if=/scratch_devs of=/dev/stdout 2023-07-14T19:23:58.448 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2023-07-14T19:23:58.449 DEBUG:teuthology.orchestra.run.smithi098:> stat /dev/vg_nvme/lv_1 2023-07-14T19:23:58.455 INFO:journalctl@ceph.mgr.z.smithi139.stdout:-- Logs begin at Fri 2023-07-14 18:56:10 UTC. -- 2023-07-14T19:23:58.458 INFO:journalctl@ceph.mgr.y.smithi101.stdout:-- Logs begin at Fri 2023-07-14 18:56:58 UTC. -- 2023-07-14T19:23:58.507 INFO:teuthology.orchestra.run.smithi098.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2023-07-14T19:23:58.508 INFO:teuthology.orchestra.run.smithi098.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-07-14T19:23:58.508 INFO:teuthology.orchestra.run.smithi098.stdout:Device: 6h/6d Inode: 379802 Links: 1 2023-07-14T19:23:58.508 INFO:teuthology.orchestra.run.smithi098.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-07-14T19:23:58.508 INFO:teuthology.orchestra.run.smithi098.stdout:Context: system_u:object_r:device_t:s0 2023-07-14T19:23:58.509 INFO:teuthology.orchestra.run.smithi098.stdout:Access: 2023-07-14 19:23:11.695422047 +0000 2023-07-14T19:23:58.509 INFO:teuthology.orchestra.run.smithi098.stdout:Modify: 2023-07-14 19:10:36.039987068 +0000 2023-07-14T19:23:58.509 INFO:teuthology.orchestra.run.smithi098.stdout:Change: 2023-07-14 19:10:36.039987068 +0000 2023-07-14T19:23:58.509 INFO:teuthology.orchestra.run.smithi098.stdout: Birth: - 2023-07-14T19:23:58.510 DEBUG:teuthology.orchestra.run.smithi098:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2023-07-14T19:23:58.576 INFO:teuthology.orchestra.run.smithi098.stderr:1+0 records in 2023-07-14T19:23:58.576 INFO:teuthology.orchestra.run.smithi098.stderr:1+0 records out 2023-07-14T19:23:58.576 INFO:teuthology.orchestra.run.smithi098.stderr:512 bytes copied, 0.000255793 s, 2.0 MB/s 2023-07-14T19:23:58.577 DEBUG:teuthology.orchestra.run.smithi098:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2023-07-14T19:23:58.636 DEBUG:teuthology.orchestra.run.smithi098:> stat /dev/vg_nvme/lv_2 2023-07-14T19:23:58.695 INFO:teuthology.orchestra.run.smithi098.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2023-07-14T19:23:58.695 INFO:teuthology.orchestra.run.smithi098.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-07-14T19:23:58.695 INFO:teuthology.orchestra.run.smithi098.stdout:Device: 6h/6d Inode: 381712 Links: 1 2023-07-14T19:23:58.695 INFO:teuthology.orchestra.run.smithi098.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-07-14T19:23:58.695 INFO:teuthology.orchestra.run.smithi098.stdout:Context: system_u:object_r:device_t:s0 2023-07-14T19:23:58.696 INFO:teuthology.orchestra.run.smithi098.stdout:Access: 2023-07-14 19:23:11.695422047 +0000 2023-07-14T19:23:58.696 INFO:teuthology.orchestra.run.smithi098.stdout:Modify: 2023-07-14 19:10:36.549980667 +0000 2023-07-14T19:23:58.696 INFO:teuthology.orchestra.run.smithi098.stdout:Change: 2023-07-14 19:10:36.549980667 +0000 2023-07-14T19:23:58.696 INFO:teuthology.orchestra.run.smithi098.stdout: Birth: - 2023-07-14T19:23:58.696 DEBUG:teuthology.orchestra.run.smithi098:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2023-07-14T19:23:58.761 INFO:teuthology.orchestra.run.smithi098.stderr:1+0 records in 2023-07-14T19:23:58.762 INFO:teuthology.orchestra.run.smithi098.stderr:1+0 records out 2023-07-14T19:23:58.762 INFO:teuthology.orchestra.run.smithi098.stderr:512 bytes copied, 0.000229687 s, 2.2 MB/s 2023-07-14T19:23:58.763 DEBUG:teuthology.orchestra.run.smithi098:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2023-07-14T19:23:58.822 DEBUG:teuthology.orchestra.run.smithi098:> stat /dev/vg_nvme/lv_3 2023-07-14T19:23:58.880 INFO:teuthology.orchestra.run.smithi098.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2023-07-14T19:23:58.880 INFO:teuthology.orchestra.run.smithi098.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-07-14T19:23:58.880 INFO:teuthology.orchestra.run.smithi098.stdout:Device: 6h/6d Inode: 382310 Links: 1 2023-07-14T19:23:58.881 INFO:teuthology.orchestra.run.smithi098.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-07-14T19:23:58.881 INFO:teuthology.orchestra.run.smithi098.stdout:Context: system_u:object_r:device_t:s0 2023-07-14T19:23:58.881 INFO:teuthology.orchestra.run.smithi098.stdout:Access: 2023-07-14 19:23:11.695422047 +0000 2023-07-14T19:23:58.881 INFO:teuthology.orchestra.run.smithi098.stdout:Modify: 2023-07-14 19:10:37.065974191 +0000 2023-07-14T19:23:58.882 INFO:teuthology.orchestra.run.smithi098.stdout:Change: 2023-07-14 19:10:37.065974191 +0000 2023-07-14T19:23:58.882 INFO:teuthology.orchestra.run.smithi098.stdout: Birth: - 2023-07-14T19:23:58.882 DEBUG:teuthology.orchestra.run.smithi098:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2023-07-14T19:23:58.945 INFO:teuthology.orchestra.run.smithi098.stderr:1+0 records in 2023-07-14T19:23:58.946 INFO:teuthology.orchestra.run.smithi098.stderr:1+0 records out 2023-07-14T19:23:58.946 INFO:teuthology.orchestra.run.smithi098.stderr:512 bytes copied, 0.000200339 s, 2.6 MB/s 2023-07-14T19:23:58.947 DEBUG:teuthology.orchestra.run.smithi098:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2023-07-14T19:23:59.007 DEBUG:teuthology.orchestra.run.smithi098:> stat /dev/vg_nvme/lv_4 2023-07-14T19:23:59.065 INFO:teuthology.orchestra.run.smithi098.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2023-07-14T19:23:59.066 INFO:teuthology.orchestra.run.smithi098.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-07-14T19:23:59.066 INFO:teuthology.orchestra.run.smithi098.stdout:Device: 6h/6d Inode: 382395 Links: 1 2023-07-14T19:23:59.066 INFO:teuthology.orchestra.run.smithi098.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-07-14T19:23:59.066 INFO:teuthology.orchestra.run.smithi098.stdout:Context: system_u:object_r:device_t:s0 2023-07-14T19:23:59.066 INFO:teuthology.orchestra.run.smithi098.stdout:Access: 2023-07-14 19:23:11.695422047 +0000 2023-07-14T19:23:59.067 INFO:teuthology.orchestra.run.smithi098.stdout:Modify: 2023-07-14 19:10:37.583967691 +0000 2023-07-14T19:23:59.067 INFO:teuthology.orchestra.run.smithi098.stdout:Change: 2023-07-14 19:10:37.583967691 +0000 2023-07-14T19:23:59.067 INFO:teuthology.orchestra.run.smithi098.stdout: Birth: - 2023-07-14T19:23:59.067 DEBUG:teuthology.orchestra.run.smithi098:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2023-07-14T19:23:59.131 INFO:teuthology.orchestra.run.smithi098.stderr:1+0 records in 2023-07-14T19:23:59.131 INFO:teuthology.orchestra.run.smithi098.stderr:1+0 records out 2023-07-14T19:23:59.131 INFO:teuthology.orchestra.run.smithi098.stderr:512 bytes copied, 0.00019867 s, 2.6 MB/s 2023-07-14T19:23:59.132 DEBUG:teuthology.orchestra.run.smithi098:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2023-07-14T19:23:59.190 DEBUG:teuthology.orchestra.run.smithi101:> set -ex 2023-07-14T19:23:59.191 DEBUG:teuthology.orchestra.run.smithi101:> dd if=/scratch_devs of=/dev/stdout 2023-07-14T19:23:59.191 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:23:58 smithi101 ceph-mon[128807]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-14T19:23:59.192 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:23:58 smithi101 ceph-mon[128807]: from='client.24100 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "3;smithi098=x;smithi101=y;smithi139=z", "target": ["mon-mgr", ""]}]: dispatch 2023-07-14T19:23:59.192 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:23:58 smithi101 ceph-mon[128807]: Saving service mgr spec with placement smithi098=x;smithi101=y;smithi139=z;count:3 2023-07-14T19:23:59.192 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:23:58 smithi101 ceph-mon[128807]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' 2023-07-14T19:23:59.192 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:23:58 smithi101 ceph-mon[128807]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' 2023-07-14T19:23:59.192 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:23:58 smithi101 ceph-mon[128807]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-14T19:23:59.192 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:23:58 smithi101 ceph-mon[128807]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T19:23:59.193 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:23:58 smithi101 ceph-mon[128807]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-14T19:23:59.193 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:23:58 smithi101 ceph-mon[128807]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' 2023-07-14T19:23:59.193 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:23:58 smithi101 ceph-mon[128807]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2023-07-14T19:23:59.193 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:23:58 smithi101 ceph-mon[128807]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2023-07-14T19:23:59.193 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:23:58 smithi101 ceph-mon[128807]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2023-07-14T19:23:59.193 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:23:58 smithi101 ceph-mon[128807]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T19:23:59.194 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:23:58 smithi139 ceph-mon[129293]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-14T19:23:59.195 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:23:58 smithi139 ceph-mon[129293]: from='client.24100 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "3;smithi098=x;smithi101=y;smithi139=z", "target": ["mon-mgr", ""]}]: dispatch 2023-07-14T19:23:59.195 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:23:58 smithi139 ceph-mon[129293]: Saving service mgr spec with placement smithi098=x;smithi101=y;smithi139=z;count:3 2023-07-14T19:23:59.195 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:23:58 smithi139 ceph-mon[129293]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' 2023-07-14T19:23:59.195 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:23:58 smithi139 ceph-mon[129293]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' 2023-07-14T19:23:59.195 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:23:58 smithi139 ceph-mon[129293]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-14T19:23:59.195 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:23:58 smithi139 ceph-mon[129293]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T19:23:59.195 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:23:58 smithi139 ceph-mon[129293]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-14T19:23:59.196 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:23:58 smithi139 ceph-mon[129293]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' 2023-07-14T19:23:59.196 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:23:58 smithi139 ceph-mon[129293]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2023-07-14T19:23:59.196 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:23:58 smithi139 ceph-mon[129293]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2023-07-14T19:23:59.196 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:23:58 smithi139 ceph-mon[129293]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2023-07-14T19:23:59.196 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:23:58 smithi139 ceph-mon[129293]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T19:23:59.214 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2023-07-14T19:23:59.215 DEBUG:teuthology.orchestra.run.smithi101:> stat /dev/vg_nvme/lv_1 2023-07-14T19:23:59.274 INFO:teuthology.orchestra.run.smithi101.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2023-07-14T19:23:59.275 INFO:teuthology.orchestra.run.smithi101.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-07-14T19:23:59.275 INFO:teuthology.orchestra.run.smithi101.stdout:Device: 6h/6d Inode: 374246 Links: 1 2023-07-14T19:23:59.275 INFO:teuthology.orchestra.run.smithi101.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-07-14T19:23:59.276 INFO:teuthology.orchestra.run.smithi101.stdout:Context: system_u:object_r:device_t:s0 2023-07-14T19:23:59.276 INFO:teuthology.orchestra.run.smithi101.stdout:Access: 2023-07-14 19:23:46.497250838 +0000 2023-07-14T19:23:59.277 INFO:teuthology.orchestra.run.smithi101.stdout:Modify: 2023-07-14 19:14:48.781402352 +0000 2023-07-14T19:23:59.277 INFO:teuthology.orchestra.run.smithi101.stdout:Change: 2023-07-14 19:14:48.781402352 +0000 2023-07-14T19:23:59.277 INFO:teuthology.orchestra.run.smithi101.stdout: Birth: - 2023-07-14T19:23:59.278 DEBUG:teuthology.orchestra.run.smithi101:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2023-07-14T19:23:59.281 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:23:58 smithi098 ceph-mon[127486]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-14T19:23:59.281 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:23:58 smithi098 ceph-mon[127486]: from='client.24100 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "3;smithi098=x;smithi101=y;smithi139=z", "target": ["mon-mgr", ""]}]: dispatch 2023-07-14T19:23:59.282 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:23:58 smithi098 ceph-mon[127486]: Saving service mgr spec with placement smithi098=x;smithi101=y;smithi139=z;count:3 2023-07-14T19:23:59.282 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:23:58 smithi098 ceph-mon[127486]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' 2023-07-14T19:23:59.282 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:23:58 smithi098 ceph-mon[127486]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' 2023-07-14T19:23:59.282 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:23:58 smithi098 ceph-mon[127486]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-14T19:23:59.282 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:23:58 smithi098 ceph-mon[127486]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T19:23:59.282 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:23:58 smithi098 ceph-mon[127486]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-14T19:23:59.282 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:23:58 smithi098 ceph-mon[127486]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' 2023-07-14T19:23:59.283 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:23:58 smithi098 ceph-mon[127486]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2023-07-14T19:23:59.283 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:23:58 smithi098 ceph-mon[127486]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2023-07-14T19:23:59.283 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:23:58 smithi098 ceph-mon[127486]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2023-07-14T19:23:59.283 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:23:58 smithi098 ceph-mon[127486]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T19:23:59.344 INFO:teuthology.orchestra.run.smithi101.stderr:1+0 records in 2023-07-14T19:23:59.345 INFO:teuthology.orchestra.run.smithi101.stderr:1+0 records out 2023-07-14T19:23:59.345 INFO:teuthology.orchestra.run.smithi101.stderr:512 bytes copied, 0.000262042 s, 2.0 MB/s 2023-07-14T19:23:59.346 DEBUG:teuthology.orchestra.run.smithi101:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2023-07-14T19:23:59.406 DEBUG:teuthology.orchestra.run.smithi101:> stat /dev/vg_nvme/lv_2 2023-07-14T19:23:59.464 INFO:teuthology.orchestra.run.smithi101.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2023-07-14T19:23:59.464 INFO:teuthology.orchestra.run.smithi101.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-07-14T19:23:59.465 INFO:teuthology.orchestra.run.smithi101.stdout:Device: 6h/6d Inode: 382252 Links: 1 2023-07-14T19:23:59.465 INFO:teuthology.orchestra.run.smithi101.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-07-14T19:23:59.465 INFO:teuthology.orchestra.run.smithi101.stdout:Context: system_u:object_r:device_t:s0 2023-07-14T19:23:59.465 INFO:teuthology.orchestra.run.smithi101.stdout:Access: 2023-07-14 19:23:46.497250838 +0000 2023-07-14T19:23:59.465 INFO:teuthology.orchestra.run.smithi101.stdout:Modify: 2023-07-14 19:14:49.292390455 +0000 2023-07-14T19:23:59.465 INFO:teuthology.orchestra.run.smithi101.stdout:Change: 2023-07-14 19:14:49.292390455 +0000 2023-07-14T19:23:59.465 INFO:teuthology.orchestra.run.smithi101.stdout: Birth: - 2023-07-14T19:23:59.466 DEBUG:teuthology.orchestra.run.smithi101:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2023-07-14T19:23:59.532 INFO:teuthology.orchestra.run.smithi101.stderr:1+0 records in 2023-07-14T19:23:59.532 INFO:teuthology.orchestra.run.smithi101.stderr:1+0 records out 2023-07-14T19:23:59.532 INFO:teuthology.orchestra.run.smithi101.stderr:512 bytes copied, 0.00023895 s, 2.1 MB/s 2023-07-14T19:23:59.533 DEBUG:teuthology.orchestra.run.smithi101:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2023-07-14T19:23:59.592 DEBUG:teuthology.orchestra.run.smithi101:> stat /dev/vg_nvme/lv_3 2023-07-14T19:23:59.650 INFO:teuthology.orchestra.run.smithi101.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2023-07-14T19:23:59.651 INFO:teuthology.orchestra.run.smithi101.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-07-14T19:23:59.651 INFO:teuthology.orchestra.run.smithi101.stdout:Device: 6h/6d Inode: 372566 Links: 1 2023-07-14T19:23:59.651 INFO:teuthology.orchestra.run.smithi101.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-07-14T19:23:59.651 INFO:teuthology.orchestra.run.smithi101.stdout:Context: system_u:object_r:device_t:s0 2023-07-14T19:23:59.651 INFO:teuthology.orchestra.run.smithi101.stdout:Access: 2023-07-14 19:23:46.497250838 +0000 2023-07-14T19:23:59.651 INFO:teuthology.orchestra.run.smithi101.stdout:Modify: 2023-07-14 19:14:49.849377488 +0000 2023-07-14T19:23:59.651 INFO:teuthology.orchestra.run.smithi101.stdout:Change: 2023-07-14 19:14:49.849377488 +0000 2023-07-14T19:23:59.652 INFO:teuthology.orchestra.run.smithi101.stdout: Birth: - 2023-07-14T19:23:59.652 DEBUG:teuthology.orchestra.run.smithi101:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2023-07-14T19:23:59.718 INFO:teuthology.orchestra.run.smithi101.stderr:1+0 records in 2023-07-14T19:23:59.719 INFO:teuthology.orchestra.run.smithi101.stderr:1+0 records out 2023-07-14T19:23:59.719 INFO:teuthology.orchestra.run.smithi101.stderr:512 bytes copied, 0.000208804 s, 2.5 MB/s 2023-07-14T19:23:59.719 DEBUG:teuthology.orchestra.run.smithi101:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2023-07-14T19:23:59.776 DEBUG:teuthology.orchestra.run.smithi101:> stat /dev/vg_nvme/lv_4 2023-07-14T19:23:59.834 INFO:teuthology.orchestra.run.smithi101.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2023-07-14T19:23:59.834 INFO:teuthology.orchestra.run.smithi101.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-07-14T19:23:59.834 INFO:teuthology.orchestra.run.smithi101.stdout:Device: 6h/6d Inode: 379682 Links: 1 2023-07-14T19:23:59.834 INFO:teuthology.orchestra.run.smithi101.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-07-14T19:23:59.834 INFO:teuthology.orchestra.run.smithi101.stdout:Context: system_u:object_r:device_t:s0 2023-07-14T19:23:59.835 INFO:teuthology.orchestra.run.smithi101.stdout:Access: 2023-07-14 19:23:46.497250838 +0000 2023-07-14T19:23:59.835 INFO:teuthology.orchestra.run.smithi101.stdout:Modify: 2023-07-14 19:14:50.376365219 +0000 2023-07-14T19:23:59.835 INFO:teuthology.orchestra.run.smithi101.stdout:Change: 2023-07-14 19:14:50.376365219 +0000 2023-07-14T19:23:59.835 INFO:teuthology.orchestra.run.smithi101.stdout: Birth: - 2023-07-14T19:23:59.835 DEBUG:teuthology.orchestra.run.smithi101:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2023-07-14T19:23:59.899 INFO:teuthology.orchestra.run.smithi101.stderr:1+0 records in 2023-07-14T19:23:59.900 INFO:teuthology.orchestra.run.smithi101.stderr:1+0 records out 2023-07-14T19:23:59.901 INFO:teuthology.orchestra.run.smithi101.stderr:512 bytes copied, 0.000189762 s, 2.7 MB/s 2023-07-14T19:23:59.902 DEBUG:teuthology.orchestra.run.smithi101:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2023-07-14T19:23:59.960 DEBUG:teuthology.orchestra.run.smithi139:> set -ex 2023-07-14T19:23:59.961 DEBUG:teuthology.orchestra.run.smithi139:> dd if=/scratch_devs of=/dev/stdout 2023-07-14T19:23:59.980 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2023-07-14T19:23:59.980 DEBUG:teuthology.orchestra.run.smithi139:> stat /dev/vg_nvme/lv_1 2023-07-14T19:24:00.037 INFO:teuthology.orchestra.run.smithi139.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2023-07-14T19:24:00.038 INFO:teuthology.orchestra.run.smithi139.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-07-14T19:24:00.038 INFO:teuthology.orchestra.run.smithi139.stdout:Device: 6h/6d Inode: 374719 Links: 1 2023-07-14T19:24:00.038 INFO:teuthology.orchestra.run.smithi139.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-07-14T19:24:00.038 INFO:teuthology.orchestra.run.smithi139.stdout:Context: system_u:object_r:device_t:s0 2023-07-14T19:24:00.038 INFO:teuthology.orchestra.run.smithi139.stdout:Access: 2023-07-14 19:23:59.798156757 +0000 2023-07-14T19:24:00.039 INFO:teuthology.orchestra.run.smithi139.stdout:Modify: 2023-07-14 19:10:06.149492528 +0000 2023-07-14T19:24:00.039 INFO:teuthology.orchestra.run.smithi139.stdout:Change: 2023-07-14 19:10:06.149492528 +0000 2023-07-14T19:24:00.039 INFO:teuthology.orchestra.run.smithi139.stdout: Birth: - 2023-07-14T19:24:00.039 DEBUG:teuthology.orchestra.run.smithi139:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2023-07-14T19:24:00.102 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:23:59 smithi139 ceph-mon[129293]: Deploying daemon mgr.z on smithi139 2023-07-14T19:24:00.103 INFO:journalctl@ceph.mgr.z.smithi139.stdout:Jul 14 19:23:59 smithi139 systemd[1]: Starting Ceph mgr.z for 9adea648-227b-11ee-9b33-001a4aab830c... 2023-07-14T19:24:00.107 INFO:teuthology.orchestra.run.smithi139.stderr:1+0 records in 2023-07-14T19:24:00.108 INFO:teuthology.orchestra.run.smithi139.stderr:1+0 records out 2023-07-14T19:24:00.108 INFO:teuthology.orchestra.run.smithi139.stderr:512 bytes copied, 0.00064303 s, 796 kB/s 2023-07-14T19:24:00.109 DEBUG:teuthology.orchestra.run.smithi139:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2023-07-14T19:24:00.167 DEBUG:teuthology.orchestra.run.smithi139:> stat /dev/vg_nvme/lv_2 2023-07-14T19:24:00.192 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:23:59 smithi101 ceph-mon[128807]: Deploying daemon mgr.z on smithi139 2023-07-14T19:24:00.225 INFO:teuthology.orchestra.run.smithi139.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2023-07-14T19:24:00.226 INFO:teuthology.orchestra.run.smithi139.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-07-14T19:24:00.226 INFO:teuthology.orchestra.run.smithi139.stdout:Device: 6h/6d Inode: 379185 Links: 1 2023-07-14T19:24:00.226 INFO:teuthology.orchestra.run.smithi139.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-07-14T19:24:00.226 INFO:teuthology.orchestra.run.smithi139.stdout:Context: system_u:object_r:device_t:s0 2023-07-14T19:24:00.226 INFO:teuthology.orchestra.run.smithi139.stdout:Access: 2023-07-14 19:23:59.798156757 +0000 2023-07-14T19:24:00.226 INFO:teuthology.orchestra.run.smithi139.stdout:Modify: 2023-07-14 19:10:06.712480356 +0000 2023-07-14T19:24:00.226 INFO:teuthology.orchestra.run.smithi139.stdout:Change: 2023-07-14 19:10:06.712480356 +0000 2023-07-14T19:24:00.227 INFO:teuthology.orchestra.run.smithi139.stdout: Birth: - 2023-07-14T19:24:00.227 DEBUG:teuthology.orchestra.run.smithi139:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2023-07-14T19:24:00.281 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:23:59 smithi098 ceph-mon[127486]: Deploying daemon mgr.z on smithi139 2023-07-14T19:24:00.293 INFO:teuthology.orchestra.run.smithi139.stderr:1+0 records in 2023-07-14T19:24:00.293 INFO:teuthology.orchestra.run.smithi139.stderr:1+0 records out 2023-07-14T19:24:00.294 INFO:teuthology.orchestra.run.smithi139.stderr:512 bytes copied, 0.000202126 s, 2.5 MB/s 2023-07-14T19:24:00.294 DEBUG:teuthology.orchestra.run.smithi139:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2023-07-14T19:24:00.351 DEBUG:teuthology.orchestra.run.smithi139:> stat /dev/vg_nvme/lv_3 2023-07-14T19:24:00.395 INFO:journalctl@ceph.mgr.z.smithi139.stdout:Jul 14 19:24:00 smithi139 podman[131147]: 2023-07-14T19:24:00.408 INFO:teuthology.orchestra.run.smithi139.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2023-07-14T19:24:00.409 INFO:teuthology.orchestra.run.smithi139.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-07-14T19:24:00.409 INFO:teuthology.orchestra.run.smithi139.stdout:Device: 6h/6d Inode: 383150 Links: 1 2023-07-14T19:24:00.409 INFO:teuthology.orchestra.run.smithi139.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-07-14T19:24:00.409 INFO:teuthology.orchestra.run.smithi139.stdout:Context: system_u:object_r:device_t:s0 2023-07-14T19:24:00.409 INFO:teuthology.orchestra.run.smithi139.stdout:Access: 2023-07-14 19:23:59.798156757 +0000 2023-07-14T19:24:00.409 INFO:teuthology.orchestra.run.smithi139.stdout:Modify: 2023-07-14 19:10:07.236469028 +0000 2023-07-14T19:24:00.410 INFO:teuthology.orchestra.run.smithi139.stdout:Change: 2023-07-14 19:10:07.236469028 +0000 2023-07-14T19:24:00.410 INFO:teuthology.orchestra.run.smithi139.stdout: Birth: - 2023-07-14T19:24:00.410 DEBUG:teuthology.orchestra.run.smithi139:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2023-07-14T19:24:00.476 INFO:teuthology.orchestra.run.smithi139.stderr:1+0 records in 2023-07-14T19:24:00.476 INFO:teuthology.orchestra.run.smithi139.stderr:1+0 records out 2023-07-14T19:24:00.477 INFO:teuthology.orchestra.run.smithi139.stderr:512 bytes copied, 0.000227586 s, 2.2 MB/s 2023-07-14T19:24:00.478 DEBUG:teuthology.orchestra.run.smithi139:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2023-07-14T19:24:00.536 DEBUG:teuthology.orchestra.run.smithi139:> stat /dev/vg_nvme/lv_4 2023-07-14T19:24:00.593 INFO:teuthology.orchestra.run.smithi139.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2023-07-14T19:24:00.593 INFO:teuthology.orchestra.run.smithi139.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-07-14T19:24:00.593 INFO:teuthology.orchestra.run.smithi139.stdout:Device: 6h/6d Inode: 385069 Links: 1 2023-07-14T19:24:00.594 INFO:teuthology.orchestra.run.smithi139.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-07-14T19:24:00.594 INFO:teuthology.orchestra.run.smithi139.stdout:Context: system_u:object_r:device_t:s0 2023-07-14T19:24:00.594 INFO:teuthology.orchestra.run.smithi139.stdout:Access: 2023-07-14 19:23:59.798156757 +0000 2023-07-14T19:24:00.594 INFO:teuthology.orchestra.run.smithi139.stdout:Modify: 2023-07-14 19:10:07.760457699 +0000 2023-07-14T19:24:00.594 INFO:teuthology.orchestra.run.smithi139.stdout:Change: 2023-07-14 19:10:07.760457699 +0000 2023-07-14T19:24:00.594 INFO:teuthology.orchestra.run.smithi139.stdout: Birth: - 2023-07-14T19:24:00.595 DEBUG:teuthology.orchestra.run.smithi139:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2023-07-14T19:24:00.656 INFO:journalctl@ceph.mgr.z.smithi139.stdout:Jul 14 19:24:00 smithi139 bash[131147]: faa8fef8c92399b07bd39dffda78802247855ab2c289a152cbc7c6fb4cb5fe9b 2023-07-14T19:24:00.657 INFO:journalctl@ceph.mgr.z.smithi139.stdout:Jul 14 19:24:00 smithi139 ceph-mgr[131311]: -- 172.21.15.139:0/2018618033 <== mon.1 v2:172.21.15.139:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x559ec83f31e0 con 0x559ec83c8800 2023-07-14T19:24:00.657 INFO:journalctl@ceph.mgr.z.smithi139.stdout:Jul 14 19:24:00 smithi139 systemd[1]: Started Ceph mgr.z for 9adea648-227b-11ee-9b33-001a4aab830c. 2023-07-14T19:24:00.657 INFO:journalctl@ceph.mgr.z.smithi139.stdout:Jul 14 19:24:00 smithi139 ceph-9adea648-227b-11ee-9b33-001a4aab830c-mgr-z[131225]: 2023-07-14T19:24:00.520+0000 7f658f46f200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-07-14T19:24:00.657 INFO:journalctl@ceph.mgr.z.smithi139.stdout:Jul 14 19:24:00 smithi139 ceph-9adea648-227b-11ee-9b33-001a4aab830c-mgr-z[131225]: 2023-07-14T19:24:00.603+0000 7f658f46f200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-07-14T19:24:00.659 INFO:teuthology.orchestra.run.smithi139.stderr:1+0 records in 2023-07-14T19:24:00.659 INFO:teuthology.orchestra.run.smithi139.stderr:1+0 records out 2023-07-14T19:24:00.659 INFO:teuthology.orchestra.run.smithi139.stderr:512 bytes copied, 0.000189631 s, 2.7 MB/s 2023-07-14T19:24:00.660 DEBUG:teuthology.orchestra.run.smithi139:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2023-07-14T19:24:00.720 INFO:tasks.cephadm:Deploying osd.0 on smithi098 with /dev/vg_nvme/lv_4... 2023-07-14T19:24:00.720 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81c9387d2e7576da065747ec9c16d1c7d69016ec shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 9adea648-227b-11ee-9b33-001a4aab830c -- ceph-volume lvm zap /dev/vg_nvme/lv_4 2023-07-14T19:24:00.964 INFO:journalctl@ceph.mgr.z.smithi139.stdout:Jul 14 19:24:00 smithi139 ceph-9adea648-227b-11ee-9b33-001a4aab830c-mgr-z[131225]: 2023-07-14T19:24:00.695+0000 7f658f46f200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-07-14T19:24:00.965 INFO:journalctl@ceph.mgr.z.smithi139.stdout:Jul 14 19:24:00 smithi139 ceph-9adea648-227b-11ee-9b33-001a4aab830c-mgr-z[131225]: 2023-07-14T19:24:00.794+0000 7f658f46f200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-07-14T19:24:00.965 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:24:00 smithi139 ceph-mon[129293]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-14T19:24:00.965 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:24:00 smithi139 ceph-mon[129293]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' 2023-07-14T19:24:00.965 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:24:00 smithi139 ceph-mon[129293]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' 2023-07-14T19:24:00.965 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:24:00 smithi139 ceph-mon[129293]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' 2023-07-14T19:24:00.965 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:24:00 smithi139 ceph-mon[129293]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2023-07-14T19:24:00.966 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:24:00 smithi139 ceph-mon[129293]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2023-07-14T19:24:00.966 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:24:00 smithi139 ceph-mon[129293]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2023-07-14T19:24:00.966 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:24:00 smithi139 ceph-mon[129293]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T19:24:01.031 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:24:00 smithi098 ceph-mon[127486]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-14T19:24:01.032 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:24:00 smithi098 ceph-mon[127486]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' 2023-07-14T19:24:01.032 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:24:00 smithi098 ceph-mon[127486]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' 2023-07-14T19:24:01.032 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:24:00 smithi098 ceph-mon[127486]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' 2023-07-14T19:24:01.032 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:24:00 smithi098 ceph-mon[127486]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2023-07-14T19:24:01.032 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:24:00 smithi098 ceph-mon[127486]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2023-07-14T19:24:01.033 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:24:00 smithi098 ceph-mon[127486]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2023-07-14T19:24:01.033 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:24:00 smithi098 ceph-mon[127486]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T19:24:01.148 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:24:00 smithi101 ceph-mon[128807]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-14T19:24:01.148 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:24:00 smithi101 ceph-mon[128807]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' 2023-07-14T19:24:01.148 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:24:00 smithi101 ceph-mon[128807]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' 2023-07-14T19:24:01.149 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:24:00 smithi101 ceph-mon[128807]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' 2023-07-14T19:24:01.149 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:24:00 smithi101 ceph-mon[128807]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2023-07-14T19:24:01.149 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:24:00 smithi101 ceph-mon[128807]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2023-07-14T19:24:01.149 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:24:00 smithi101 ceph-mon[128807]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2023-07-14T19:24:01.149 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:24:00 smithi101 ceph-mon[128807]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T19:24:01.220 INFO:journalctl@ceph.mgr.z.smithi139.stdout:Jul 14 19:24:00 smithi139 ceph-9adea648-227b-11ee-9b33-001a4aab830c-mgr-z[131225]: 2023-07-14T19:24:00.963+0000 7f658f46f200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-07-14T19:24:01.281 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/9adea648-227b-11ee-9b33-001a4aab830c/mon.a/config 2023-07-14T19:24:01.579 INFO:journalctl@ceph.mgr.z.smithi139.stdout:Jul 14 19:24:01 smithi139 ceph-9adea648-227b-11ee-9b33-001a4aab830c-mgr-z[131225]: 2023-07-14T19:24:01.269+0000 7f658f46f200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-07-14T19:24:01.579 INFO:journalctl@ceph.mgr.z.smithi139.stdout:Jul 14 19:24:01 smithi139 ceph-9adea648-227b-11ee-9b33-001a4aab830c-mgr-z[131225]: 2023-07-14T19:24:01.357+0000 7f658f46f200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-07-14T19:24:01.579 INFO:journalctl@ceph.mgr.z.smithi139.stdout:Jul 14 19:24:01 smithi139 ceph-9adea648-227b-11ee-9b33-001a4aab830c-mgr-z[131225]: 2023-07-14T19:24:01.485+0000 7f658f46f200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-07-14T19:24:01.892 INFO:journalctl@ceph.mgr.z.smithi139.stdout:Jul 14 19:24:01 smithi139 ceph-9adea648-227b-11ee-9b33-001a4aab830c-mgr-z[131225]: 2023-07-14T19:24:01.578+0000 7f658f46f200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-07-14T19:24:01.991 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:24:01 smithi098 ceph-mon[127486]: Deploying daemon mgr.y on smithi101 2023-07-14T19:24:02.192 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:24:01 smithi101 ceph-mon[128807]: Deploying daemon mgr.y on smithi101 2023-07-14T19:24:02.192 INFO:journalctl@ceph.mgr.y.smithi101.stdout:Jul 14 19:24:01 smithi101 systemd[1]: Starting Ceph mgr.y for 9adea648-227b-11ee-9b33-001a4aab830c... 2023-07-14T19:24:02.220 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:24:01 smithi139 ceph-mon[129293]: Deploying daemon mgr.y on smithi101 2023-07-14T19:24:02.221 INFO:journalctl@ceph.mgr.z.smithi139.stdout:Jul 14 19:24:01 smithi139 ceph-9adea648-227b-11ee-9b33-001a4aab830c-mgr-z[131225]: 2023-07-14T19:24:01.998+0000 7f658f46f200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-07-14T19:24:02.570 INFO:journalctl@ceph.mgr.y.smithi101.stdout:Jul 14 19:24:02 smithi101 podman[130240]: 2023-07-14T19:24:02.570 INFO:journalctl@ceph.mgr.y.smithi101.stdout:Jul 14 19:24:02 smithi101 bash[130240]: 615bd99c6c7ff34d5cee67a1bd829a66b48cc86a02f0640efc1911285ffb3b8e 2023-07-14T19:24:02.570 INFO:journalctl@ceph.mgr.y.smithi101.stdout:Jul 14 19:24:02 smithi101 ceph-mgr[130280]: -- 172.21.15.101:0/3595695981 <== mon.2 v2:172.21.15.101:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x563b623331e0 con 0x563b62308800 2023-07-14T19:24:02.707 INFO:teuthology.orchestra.run.smithi098.stderr:--> Zapping: /dev/vg_nvme/lv_4 2023-07-14T19:24:02.708 INFO:teuthology.orchestra.run.smithi098.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 conv=fsync 2023-07-14T19:24:02.708 INFO:teuthology.orchestra.run.smithi098.stderr: stderr: 10+0 records in 2023-07-14T19:24:02.708 INFO:teuthology.orchestra.run.smithi098.stderr:10+0 records out 2023-07-14T19:24:02.708 INFO:teuthology.orchestra.run.smithi098.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.011673 s, 898 MB/s 2023-07-14T19:24:02.708 INFO:teuthology.orchestra.run.smithi098.stderr: stderr: 2023-07-14T19:24:02.709 INFO:teuthology.orchestra.run.smithi098.stderr:--> Zapping successful for: 2023-07-14T19:24:02.894 INFO:journalctl@ceph.mgr.y.smithi101.stdout:Jul 14 19:24:02 smithi101 systemd[1]: Started Ceph mgr.y for 9adea648-227b-11ee-9b33-001a4aab830c. 2023-07-14T19:24:02.894 INFO:journalctl@ceph.mgr.y.smithi101.stdout:Jul 14 19:24:02 smithi101 ceph-9adea648-227b-11ee-9b33-001a4aab830c-mgr-y[130256]: 2023-07-14T19:24:02.644+0000 7f845a459200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-07-14T19:24:02.894 INFO:journalctl@ceph.mgr.y.smithi101.stdout:Jul 14 19:24:02 smithi101 ceph-9adea648-227b-11ee-9b33-001a4aab830c-mgr-y[130256]: 2023-07-14T19:24:02.723+0000 7f845a459200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-07-14T19:24:02.895 INFO:journalctl@ceph.mgr.y.smithi101.stdout:Jul 14 19:24:02 smithi101 ceph-9adea648-227b-11ee-9b33-001a4aab830c-mgr-y[130256]: 2023-07-14T19:24:02.816+0000 7f845a459200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-07-14T19:24:02.895 INFO:journalctl@ceph.mgr.z.smithi139.stdout:Jul 14 19:24:02 smithi139 ceph-9adea648-227b-11ee-9b33-001a4aab830c-mgr-z[131225]: 2023-07-14T19:24:02.577+0000 7f658f46f200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2023-07-14T19:24:02.895 INFO:journalctl@ceph.mgr.z.smithi139.stdout:Jul 14 19:24:02 smithi139 ceph-9adea648-227b-11ee-9b33-001a4aab830c-mgr-z[131225]: 2023-07-14T19:24:02.748+0000 7f658f46f200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-07-14T19:24:03.032 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:24:02 smithi098 ceph-mon[127486]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-14T19:24:03.032 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:24:02 smithi098 ceph-mon[127486]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' 2023-07-14T19:24:03.032 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:24:02 smithi098 ceph-mon[127486]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' 2023-07-14T19:24:03.033 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:24:02 smithi098 ceph-mon[127486]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' 2023-07-14T19:24:03.033 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:24:02 smithi098 ceph-mon[127486]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' 2023-07-14T19:24:03.033 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:24:02 smithi098 ceph-mon[127486]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-14T19:24:03.136 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81c9387d2e7576da065747ec9c16d1c7d69016ec shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 9adea648-227b-11ee-9b33-001a4aab830c -- ceph orch daemon add osd smithi098:vg_nvme/lv_4 2023-07-14T19:24:03.192 INFO:journalctl@ceph.mgr.y.smithi101.stdout:Jul 14 19:24:02 smithi101 ceph-9adea648-227b-11ee-9b33-001a4aab830c-mgr-y[130256]: 2023-07-14T19:24:02.915+0000 7f845a459200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-07-14T19:24:03.192 INFO:journalctl@ceph.mgr.y.smithi101.stdout:Jul 14 19:24:03 smithi101 ceph-9adea648-227b-11ee-9b33-001a4aab830c-mgr-y[130256]: 2023-07-14T19:24:03.091+0000 7f845a459200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-07-14T19:24:03.192 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:24:02 smithi101 ceph-mon[128807]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-14T19:24:03.193 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:24:02 smithi101 ceph-mon[128807]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' 2023-07-14T19:24:03.193 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:24:02 smithi101 ceph-mon[128807]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' 2023-07-14T19:24:03.193 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:24:02 smithi101 ceph-mon[128807]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' 2023-07-14T19:24:03.193 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:24:02 smithi101 ceph-mon[128807]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' 2023-07-14T19:24:03.193 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:24:02 smithi101 ceph-mon[128807]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-14T19:24:03.220 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:24:02 smithi139 ceph-mon[129293]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-14T19:24:03.221 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:24:02 smithi139 ceph-mon[129293]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' 2023-07-14T19:24:03.221 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:24:02 smithi139 ceph-mon[129293]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' 2023-07-14T19:24:03.221 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:24:02 smithi139 ceph-mon[129293]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' 2023-07-14T19:24:03.221 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:24:02 smithi139 ceph-mon[129293]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' 2023-07-14T19:24:03.221 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:24:02 smithi139 ceph-mon[129293]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-14T19:24:03.686 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/9adea648-227b-11ee-9b33-001a4aab830c/mon.a/config 2023-07-14T19:24:03.691 INFO:journalctl@ceph.mgr.y.smithi101.stdout:Jul 14 19:24:03 smithi101 ceph-9adea648-227b-11ee-9b33-001a4aab830c-mgr-y[130256]: 2023-07-14T19:24:03.410+0000 7f845a459200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-07-14T19:24:03.692 INFO:journalctl@ceph.mgr.y.smithi101.stdout:Jul 14 19:24:03 smithi101 ceph-9adea648-227b-11ee-9b33-001a4aab830c-mgr-y[130256]: 2023-07-14T19:24:03.503+0000 7f845a459200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-07-14T19:24:03.692 INFO:journalctl@ceph.mgr.y.smithi101.stdout:Jul 14 19:24:03 smithi101 ceph-9adea648-227b-11ee-9b33-001a4aab830c-mgr-y[130256]: 2023-07-14T19:24:03.646+0000 7f845a459200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-07-14T19:24:03.850 INFO:journalctl@ceph.mgr.z.smithi139.stdout:Jul 14 19:24:03 smithi139 ceph-9adea648-227b-11ee-9b33-001a4aab830c-mgr-z[131225]: 2023-07-14T19:24:03.568+0000 7f658f46f200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-07-14T19:24:03.850 INFO:journalctl@ceph.mgr.z.smithi139.stdout:Jul 14 19:24:03 smithi139 ceph-9adea648-227b-11ee-9b33-001a4aab830c-mgr-z[131225]: 2023-07-14T19:24:03.664+0000 7f658f46f200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-07-14T19:24:04.150 INFO:journalctl@ceph.mgr.z.smithi139.stdout:Jul 14 19:24:03 smithi139 ceph-9adea648-227b-11ee-9b33-001a4aab830c-mgr-z[131225]: 2023-07-14T19:24:03.849+0000 7f658f46f200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-07-14T19:24:04.191 INFO:journalctl@ceph.mgr.y.smithi101.stdout:Jul 14 19:24:03 smithi101 ceph-9adea648-227b-11ee-9b33-001a4aab830c-mgr-y[130256]: 2023-07-14T19:24:03.745+0000 7f845a459200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-07-14T19:24:04.401 INFO:journalctl@ceph.mgr.z.smithi139.stdout:Jul 14 19:24:04 smithi139 ceph-9adea648-227b-11ee-9b33-001a4aab830c-mgr-z[131225]: 2023-07-14T19:24:04.149+0000 7f658f46f200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2023-07-14T19:24:04.691 INFO:journalctl@ceph.mgr.y.smithi101.stdout:Jul 14 19:24:04 smithi101 ceph-9adea648-227b-11ee-9b33-001a4aab830c-mgr-y[130256]: 2023-07-14T19:24:04.192+0000 7f845a459200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-07-14T19:24:04.720 INFO:journalctl@ceph.mgr.z.smithi139.stdout:Jul 14 19:24:04 smithi139 ceph-9adea648-227b-11ee-9b33-001a4aab830c-mgr-z[131225]: 2023-07-14T19:24:04.400+0000 7f658f46f200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-07-14T19:24:04.721 INFO:journalctl@ceph.mgr.z.smithi139.stdout:Jul 14 19:24:04 smithi139 ceph-9adea648-227b-11ee-9b33-001a4aab830c-mgr-z[131225]: 2023-07-14T19:24:04.551+0000 7f658f46f200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-07-14T19:24:05.010 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:24:05.007+0000 7f13ab627700 1 -- 172.21.15.98:0/1434737227 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f13a4100740 msgr2=0x7f13a4102b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:24:05.010 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:24:05.007+0000 7f13ab627700 1 --2- 172.21.15.98:0/1434737227 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f13a4100740 0x7f13a4102b20 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f13a0009990 tx=0x7f13a0009ca0 comp rx=0 tx=0).stop 2023-07-14T19:24:05.010 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:24:05.007+0000 7f13ab627700 1 -- 172.21.15.98:0/1434737227 shutdown_connections 2023-07-14T19:24:05.010 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:24:05.007+0000 7f13ab627700 1 --2- 172.21.15.98:0/1434737227 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f13a4105980 0x7f13a4107de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:24:05.011 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:24:05.007+0000 7f13ab627700 1 --2- 172.21.15.98:0/1434737227 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f13a4100740 0x7f13a4102b20 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:24:05.011 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:24:05.007+0000 7f13ab627700 1 --2- 172.21.15.98:0/1434737227 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f13a4103060 0x7f13a4105440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:24:05.011 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:24:05.007+0000 7f13ab627700 1 -- 172.21.15.98:0/1434737227 >> 172.21.15.98:0/1434737227 conn(0x7f13a40fa1e0 msgr2=0x7f13a40fc630 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:24:05.011 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:24:05.007+0000 7f13ab627700 1 -- 172.21.15.98:0/1434737227 shutdown_connections 2023-07-14T19:24:05.012 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:24:05.007+0000 7f13ab627700 1 -- 172.21.15.98:0/1434737227 wait complete. 2023-07-14T19:24:05.012 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:24:05.008+0000 7f13ab627700 1 Processor -- start 2023-07-14T19:24:05.012 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:24:05.008+0000 7f13ab627700 1 -- start start 2023-07-14T19:24:05.012 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:24:05.008+0000 7f13ab627700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f13a4100740 0x7f13a41959c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:24:05.013 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:24:05.008+0000 7f13ab627700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f13a4103060 0x7f13a4195f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:24:05.014 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:24:05.008+0000 7f13ab627700 1 --2- >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f13a4105980 0x7f13a419cf80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:24:05.014 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:24:05.008+0000 7f13ab627700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f13a419d550 con 0x7f13a4100740 2023-07-14T19:24:05.014 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:24:05.008+0000 7f13ab627700 1 -- --> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f13a419d690 con 0x7f13a4105980 2023-07-14T19:24:05.014 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:24:05.008+0000 7f13ab627700 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f13a419d820 con 0x7f13a4103060 2023-07-14T19:24:05.014 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:24:05.008+0000 7f13a93c3700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f13a4100740 0x7f13a41959c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:24:05.015 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:24:05.008+0000 7f13a93c3700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f13a4100740 0x7f13a41959c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:44754/0 (socket says 172.21.15.98:44754) 2023-07-14T19:24:05.015 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:24:05.008+0000 7f13a93c3700 1 -- 172.21.15.98:0/202011673 learned_addr learned my addr 172.21.15.98:0/202011673 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-07-14T19:24:05.015 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:24:05.008+0000 7f13a9bc4700 1 --2- 172.21.15.98:0/202011673 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f13a4105980 0x7f13a419cf80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:24:05.015 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:24:05.008+0000 7f13a8bc2700 1 --2- 172.21.15.98:0/202011673 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f13a4103060 0x7f13a4195f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:24:05.015 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:24:05.008+0000 7f13a93c3700 1 -- 172.21.15.98:0/202011673 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f13a4103060 msgr2=0x7f13a4195f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:24:05.015 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:24:05.008+0000 7f13a93c3700 1 --2- 172.21.15.98:0/202011673 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f13a4103060 0x7f13a4195f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:24:05.015 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:24:05.008+0000 7f13a93c3700 1 -- 172.21.15.98:0/202011673 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f13a4105980 msgr2=0x7f13a419cf80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:24:05.016 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:24:05.008+0000 7f13a93c3700 1 --2- 172.21.15.98:0/202011673 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f13a4105980 0x7f13a419cf80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:24:05.016 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:24:05.008+0000 7f13a93c3700 1 -- 172.21.15.98:0/202011673 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f13a00095f0 con 0x7f13a4100740 2023-07-14T19:24:05.016 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:24:05.008+0000 7f13a93c3700 1 --2- 172.21.15.98:0/202011673 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f13a4100740 0x7f13a41959c0 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f13a0000c00 tx=0x7f13a000f690 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:24:05.017 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:24:05.009+0000 7f139a7fc700 1 -- 172.21.15.98:0/202011673 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f13a001d070 con 0x7f13a4100740 2023-07-14T19:24:05.017 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:24:05.009+0000 7f139a7fc700 1 -- 172.21.15.98:0/202011673 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f13a0022470 con 0x7f13a4100740 2023-07-14T19:24:05.017 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:24:05.009+0000 7f139a7fc700 1 -- 172.21.15.98:0/202011673 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f13a0017650 con 0x7f13a4100740 2023-07-14T19:24:05.017 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:24:05.011+0000 7f13ab627700 1 -- 172.21.15.98:0/202011673 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f13a419daa0 con 0x7f13a4100740 2023-07-14T19:24:05.018 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:24:05.011+0000 7f13ab627700 1 -- 172.21.15.98:0/202011673 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f13a419de60 con 0x7f13a4100740 2023-07-14T19:24:05.018 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:24:05.012+0000 7f139a7fc700 1 -- 172.21.15.98:0/202011673 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 13) v1 ==== 45539+0+0 (secure 0 0 0) 0x7f13a00225e0 con 0x7f13a4100740 2023-07-14T19:24:05.018 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:24:05.012+0000 7f139a7fc700 1 --2- 172.21.15.98:0/202011673 >> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] conn(0x7f13900389e0 0x7f139003ae90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:24:05.018 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:24:05.012+0000 7f139a7fc700 1 -- 172.21.15.98:0/202011673 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f13a004c300 con 0x7f13a4100740 2023-07-14T19:24:05.018 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:24:05.012+0000 7f13a8bc2700 1 --2- 172.21.15.98:0/202011673 >> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] conn(0x7f13900389e0 0x7f139003ae90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:24:05.018 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:24:05.012+0000 7f13a8bc2700 1 --2- 172.21.15.98:0/202011673 >> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] conn(0x7f13900389e0 0x7f139003ae90 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f1394005950 tx=0x7f139400b410 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:24:05.019 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:24:05.013+0000 7f13ab627700 1 -- 172.21.15.98:0/202011673 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f13a40fbd50 con 0x7f13a4100740 2023-07-14T19:24:05.019 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:24:05.017+0000 7f139a7fc700 1 -- 172.21.15.98:0/202011673 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f13a0027020 con 0x7f13a4100740 2023-07-14T19:24:05.080 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:24:04 smithi098 ceph-mon[127486]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-14T19:24:05.080 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:24:04 smithi098 ceph-mon[127486]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' 2023-07-14T19:24:05.080 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:24:04 smithi098 ceph-mon[127486]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' 2023-07-14T19:24:05.080 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:24:04 smithi098 ceph-mon[127486]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' 2023-07-14T19:24:05.081 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:24:04 smithi098 ceph-mon[127486]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' 2023-07-14T19:24:05.081 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:24:04 smithi098 ceph-mon[127486]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T19:24:05.081 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:24:04 smithi098 ceph-mon[127486]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-14T19:24:05.081 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:24:04 smithi098 ceph-mon[127486]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' 2023-07-14T19:24:05.081 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:24:04 smithi098 ceph-mon[127486]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2023-07-14T19:24:05.081 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:24:04 smithi098 ceph-mon[127486]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2023-07-14T19:24:05.082 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:24:04 smithi098 ceph-mon[127486]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T19:24:05.192 INFO:journalctl@ceph.mgr.y.smithi101.stdout:Jul 14 19:24:04 smithi101 ceph-9adea648-227b-11ee-9b33-001a4aab830c-mgr-y[130256]: 2023-07-14T19:24:04.771+0000 7f845a459200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2023-07-14T19:24:05.192 INFO:journalctl@ceph.mgr.y.smithi101.stdout:Jul 14 19:24:04 smithi101 ceph-9adea648-227b-11ee-9b33-001a4aab830c-mgr-y[130256]: 2023-07-14T19:24:04.939+0000 7f845a459200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-07-14T19:24:05.193 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:24:04 smithi101 ceph-mon[128807]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-14T19:24:05.193 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:24:04 smithi101 ceph-mon[128807]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' 2023-07-14T19:24:05.193 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:24:04 smithi101 ceph-mon[128807]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' 2023-07-14T19:24:05.194 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:24:04 smithi101 ceph-mon[128807]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' 2023-07-14T19:24:05.194 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:24:04 smithi101 ceph-mon[128807]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' 2023-07-14T19:24:05.194 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:24:04 smithi101 ceph-mon[128807]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T19:24:05.195 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:24:04 smithi101 ceph-mon[128807]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-14T19:24:05.195 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:24:04 smithi101 ceph-mon[128807]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' 2023-07-14T19:24:05.195 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:24:04 smithi101 ceph-mon[128807]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2023-07-14T19:24:05.195 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:24:04 smithi101 ceph-mon[128807]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2023-07-14T19:24:05.196 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:24:04 smithi101 ceph-mon[128807]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T19:24:05.221 INFO:journalctl@ceph.mgr.z.smithi139.stdout:Jul 14 19:24:04 smithi139 ceph-9adea648-227b-11ee-9b33-001a4aab830c-mgr-z[131225]: 2023-07-14T19:24:04.809+0000 7f658f46f200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-07-14T19:24:05.221 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:24:04 smithi139 ceph-mon[129293]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-14T19:24:05.221 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:24:04 smithi139 ceph-mon[129293]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' 2023-07-14T19:24:05.221 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:24:04 smithi139 ceph-mon[129293]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' 2023-07-14T19:24:05.222 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:24:04 smithi139 ceph-mon[129293]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' 2023-07-14T19:24:05.222 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:24:04 smithi139 ceph-mon[129293]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' 2023-07-14T19:24:05.222 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:24:04 smithi139 ceph-mon[129293]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T19:24:05.222 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:24:04 smithi139 ceph-mon[129293]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-14T19:24:05.222 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:24:04 smithi139 ceph-mon[129293]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' 2023-07-14T19:24:05.223 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:24:04 smithi139 ceph-mon[129293]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2023-07-14T19:24:05.223 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:24:04 smithi139 ceph-mon[129293]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2023-07-14T19:24:05.223 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:24:04 smithi139 ceph-mon[129293]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T19:24:05.224 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:24:05.222+0000 7f13ab627700 1 -- 172.21.15.98:0/202011673 --> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi098:vg_nvme/lv_4", "target": ["mon-mgr", ""]}) v1 -- 0x7f13a4060a80 con 0x7f13900389e0 2023-07-14T19:24:05.827 INFO:journalctl@ceph.mgr.z.smithi139.stdout:Jul 14 19:24:05 smithi139 ceph-9adea648-227b-11ee-9b33-001a4aab830c-mgr-z[131225]: 2023-07-14T19:24:05.508+0000 7f658f46f200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-07-14T19:24:05.827 INFO:journalctl@ceph.mgr.z.smithi139.stdout:Jul 14 19:24:05 smithi139 ceph-9adea648-227b-11ee-9b33-001a4aab830c-mgr-z[131225]: 2023-07-14T19:24:05.636+0000 7f658f46f200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-07-14T19:24:05.957 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:24:05 smithi101 ceph-mon[128807]: Reconfiguring mgr.x (unknown last config time)... 2023-07-14T19:24:05.957 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:24:05 smithi101 ceph-mon[128807]: Reconfiguring daemon mgr.x on smithi098 2023-07-14T19:24:05.958 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:24:05 smithi101 ceph-mon[128807]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-07-14T19:24:05.958 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:24:05 smithi101 ceph-mon[128807]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-07-14T19:24:05.958 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:24:05 smithi101 ceph-mon[128807]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T19:24:05.958 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:24:05 smithi101 ceph-mon[128807]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' 2023-07-14T19:24:05.958 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:24:05 smithi101 ceph-mon[128807]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' 2023-07-14T19:24:05.958 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:24:05 smithi101 ceph-mon[128807]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-14T19:24:05.959 INFO:journalctl@ceph.mgr.y.smithi101.stdout:Jul 14 19:24:05 smithi101 ceph-9adea648-227b-11ee-9b33-001a4aab830c-mgr-y[130256]: 2023-07-14T19:24:05.693+0000 7f845a459200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-07-14T19:24:05.959 INFO:journalctl@ceph.mgr.y.smithi101.stdout:Jul 14 19:24:05 smithi101 ceph-9adea648-227b-11ee-9b33-001a4aab830c-mgr-y[130256]: 2023-07-14T19:24:05.785+0000 7f845a459200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-07-14T19:24:06.152 INFO:journalctl@ceph.mgr.z.smithi139.stdout:Jul 14 19:24:05 smithi139 ceph-9adea648-227b-11ee-9b33-001a4aab830c-mgr-z[131225]: 2023-07-14T19:24:05.826+0000 7f658f46f200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-07-14T19:24:06.153 INFO:journalctl@ceph.mgr.z.smithi139.stdout:Jul 14 19:24:05 smithi139 ceph-9adea648-227b-11ee-9b33-001a4aab830c-mgr-z[131225]: 2023-07-14T19:24:05.938+0000 7f658f46f200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-07-14T19:24:06.153 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:24:05 smithi139 ceph-mon[129293]: Reconfiguring mgr.x (unknown last config time)... 2023-07-14T19:24:06.153 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:24:05 smithi139 ceph-mon[129293]: Reconfiguring daemon mgr.x on smithi098 2023-07-14T19:24:06.153 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:24:05 smithi139 ceph-mon[129293]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-07-14T19:24:06.153 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:24:05 smithi139 ceph-mon[129293]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-07-14T19:24:06.154 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:24:05 smithi139 ceph-mon[129293]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T19:24:06.154 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:24:05 smithi139 ceph-mon[129293]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' 2023-07-14T19:24:06.154 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:24:05 smithi139 ceph-mon[129293]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' 2023-07-14T19:24:06.154 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:24:05 smithi139 ceph-mon[129293]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-14T19:24:06.157 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:24:05 smithi098 ceph-mon[127486]: Reconfiguring mgr.x (unknown last config time)... 2023-07-14T19:24:06.157 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:24:05 smithi098 ceph-mon[127486]: Reconfiguring daemon mgr.x on smithi098 2023-07-14T19:24:06.158 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:24:05 smithi098 ceph-mon[127486]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-07-14T19:24:06.158 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:24:05 smithi098 ceph-mon[127486]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-07-14T19:24:06.158 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:24:05 smithi098 ceph-mon[127486]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T19:24:06.158 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:24:05 smithi098 ceph-mon[127486]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' 2023-07-14T19:24:06.158 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:24:05 smithi098 ceph-mon[127486]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' 2023-07-14T19:24:06.158 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:24:05 smithi098 ceph-mon[127486]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-14T19:24:06.232 INFO:journalctl@ceph.mgr.y.smithi101.stdout:Jul 14 19:24:05 smithi101 ceph-9adea648-227b-11ee-9b33-001a4aab830c-mgr-y[130256]: 2023-07-14T19:24:05.955+0000 7f845a459200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-07-14T19:24:06.470 INFO:journalctl@ceph.mgr.z.smithi139.stdout:Jul 14 19:24:06 smithi139 ceph-9adea648-227b-11ee-9b33-001a4aab830c-mgr-z[131225]: 2023-07-14T19:24:06.151+0000 7f658f46f200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-07-14T19:24:06.604 INFO:journalctl@ceph.mgr.y.smithi101.stdout:Jul 14 19:24:06 smithi101 ceph-9adea648-227b-11ee-9b33-001a4aab830c-mgr-y[130256]: 2023-07-14T19:24:06.230+0000 7f845a459200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2023-07-14T19:24:06.604 INFO:journalctl@ceph.mgr.y.smithi101.stdout:Jul 14 19:24:06 smithi101 ceph-9adea648-227b-11ee-9b33-001a4aab830c-mgr-y[130256]: 2023-07-14T19:24:06.457+0000 7f845a459200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-07-14T19:24:06.861 INFO:journalctl@ceph.mgr.y.smithi101.stdout:Jul 14 19:24:06 smithi101 ceph-9adea648-227b-11ee-9b33-001a4aab830c-mgr-y[130256]: 2023-07-14T19:24:06.602+0000 7f845a459200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-07-14T19:24:07.191 INFO:journalctl@ceph.mgr.y.smithi101.stdout:Jul 14 19:24:06 smithi101 ceph-9adea648-227b-11ee-9b33-001a4aab830c-mgr-y[130256]: 2023-07-14T19:24:06.858+0000 7f845a459200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-07-14T19:24:07.192 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:24:06 smithi101 ceph-mon[128807]: from='client.14226 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi098:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2023-07-14T19:24:07.192 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:24:06 smithi101 ceph-mon[128807]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-14T19:24:07.192 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:24:06 smithi101 ceph-mon[128807]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T19:24:07.192 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:24:06 smithi101 ceph-mon[128807]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-14T19:24:07.192 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:24:06 smithi101 ceph-mon[128807]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' 2023-07-14T19:24:07.193 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:24:06 smithi101 ceph-mon[128807]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' 2023-07-14T19:24:07.221 INFO:journalctl@ceph.mgr.z.smithi139.stdout:Jul 14 19:24:07 smithi139 ceph-9adea648-227b-11ee-9b33-001a4aab830c-mgr-z[131225]: 2023-07-14T19:24:07.006+0000 7f658f46f200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-07-14T19:24:07.221 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:24:06 smithi139 ceph-mon[129293]: from='client.14226 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi098:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2023-07-14T19:24:07.221 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:24:06 smithi139 ceph-mon[129293]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-14T19:24:07.221 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:24:06 smithi139 ceph-mon[129293]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T19:24:07.221 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:24:06 smithi139 ceph-mon[129293]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-14T19:24:07.222 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:24:06 smithi139 ceph-mon[129293]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' 2023-07-14T19:24:07.222 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:24:06 smithi139 ceph-mon[129293]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' 2023-07-14T19:24:07.282 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:24:06 smithi098 ceph-mon[127486]: from='client.14226 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi098:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2023-07-14T19:24:07.282 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:24:06 smithi098 ceph-mon[127486]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-14T19:24:07.282 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:24:06 smithi098 ceph-mon[127486]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T19:24:07.283 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:24:06 smithi098 ceph-mon[127486]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-14T19:24:07.283 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:24:06 smithi098 ceph-mon[127486]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' 2023-07-14T19:24:07.283 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:24:06 smithi098 ceph-mon[127486]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' 2023-07-14T19:24:07.873 INFO:journalctl@ceph.mgr.y.smithi101.stdout:Jul 14 19:24:07 smithi101 ceph-9adea648-227b-11ee-9b33-001a4aab830c-mgr-y[130256]: 2023-07-14T19:24:07.553+0000 7f845a459200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-07-14T19:24:07.873 INFO:journalctl@ceph.mgr.y.smithi101.stdout:Jul 14 19:24:07 smithi101 ceph-9adea648-227b-11ee-9b33-001a4aab830c-mgr-y[130256]: 2023-07-14T19:24:07.682+0000 7f845a459200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-07-14T19:24:07.910 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:24:07.908+0000 7f139a7fc700 1 -- 172.21.15.98:0/202011673 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mgrmap(e 14) v1 ==== 90599+0+0 (secure 0 0 0) 0x7f13a00177b0 con 0x7f13a4100740 2023-07-14T19:24:08.191 INFO:journalctl@ceph.mgr.y.smithi101.stdout:Jul 14 19:24:07 smithi101 ceph-9adea648-227b-11ee-9b33-001a4aab830c-mgr-y[130256]: 2023-07-14T19:24:07.870+0000 7f845a459200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-07-14T19:24:08.192 INFO:journalctl@ceph.mgr.y.smithi101.stdout:Jul 14 19:24:07 smithi101 ceph-9adea648-227b-11ee-9b33-001a4aab830c-mgr-y[130256]: 2023-07-14T19:24:07.983+0000 7f845a459200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-07-14T19:24:08.192 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:24:07 smithi101 ceph-mon[128807]: Standby manager daemon z started 2023-07-14T19:24:08.192 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:24:07 smithi101 ceph-mon[128807]: from='mgr.? 172.21.15.139:0/1937053169' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2023-07-14T19:24:08.193 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:24:07 smithi101 ceph-mon[128807]: from='mgr.? 172.21.15.139:0/1937053169' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2023-07-14T19:24:08.193 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:24:07 smithi101 ceph-mon[128807]: from='mgr.? 172.21.15.139:0/1937053169' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2023-07-14T19:24:08.193 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:24:07 smithi101 ceph-mon[128807]: from='mgr.? 172.21.15.139:0/1937053169' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2023-07-14T19:24:08.193 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:24:07 smithi101 ceph-mon[128807]: from='client.? 172.21.15.98:0/3342608815' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "a3797e89-1754-41cf-9517-227e06ffdd72"}]: dispatch 2023-07-14T19:24:08.193 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:24:07 smithi101 ceph-mon[128807]: from='client.? 172.21.15.98:0/3342608815' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "a3797e89-1754-41cf-9517-227e06ffdd72"}]': finished 2023-07-14T19:24:08.193 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:24:07 smithi101 ceph-mon[128807]: osdmap e5: 1 total, 0 up, 1 in 2023-07-14T19:24:08.193 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:24:07 smithi101 ceph-mon[128807]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2023-07-14T19:24:08.221 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:24:07 smithi139 ceph-mon[129293]: Standby manager daemon z started 2023-07-14T19:24:08.221 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:24:07 smithi139 ceph-mon[129293]: from='mgr.? 172.21.15.139:0/1937053169' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2023-07-14T19:24:08.221 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:24:07 smithi139 ceph-mon[129293]: from='mgr.? 172.21.15.139:0/1937053169' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2023-07-14T19:24:08.221 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:24:07 smithi139 ceph-mon[129293]: from='mgr.? 172.21.15.139:0/1937053169' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2023-07-14T19:24:08.222 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:24:07 smithi139 ceph-mon[129293]: from='mgr.? 172.21.15.139:0/1937053169' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2023-07-14T19:24:08.222 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:24:07 smithi139 ceph-mon[129293]: from='client.? 172.21.15.98:0/3342608815' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "a3797e89-1754-41cf-9517-227e06ffdd72"}]: dispatch 2023-07-14T19:24:08.222 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:24:07 smithi139 ceph-mon[129293]: from='client.? 172.21.15.98:0/3342608815' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "a3797e89-1754-41cf-9517-227e06ffdd72"}]': finished 2023-07-14T19:24:08.222 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:24:07 smithi139 ceph-mon[129293]: osdmap e5: 1 total, 0 up, 1 in 2023-07-14T19:24:08.222 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:24:07 smithi139 ceph-mon[129293]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2023-07-14T19:24:08.281 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:24:07 smithi098 ceph-mon[127486]: Standby manager daemon z started 2023-07-14T19:24:08.282 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:24:07 smithi098 ceph-mon[127486]: from='mgr.? 172.21.15.139:0/1937053169' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2023-07-14T19:24:08.282 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:24:07 smithi098 ceph-mon[127486]: from='mgr.? 172.21.15.139:0/1937053169' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2023-07-14T19:24:08.282 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:24:07 smithi098 ceph-mon[127486]: from='mgr.? 172.21.15.139:0/1937053169' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2023-07-14T19:24:08.282 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:24:07 smithi098 ceph-mon[127486]: from='mgr.? 172.21.15.139:0/1937053169' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2023-07-14T19:24:08.282 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:24:07 smithi098 ceph-mon[127486]: from='client.? 172.21.15.98:0/3342608815' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "a3797e89-1754-41cf-9517-227e06ffdd72"}]: dispatch 2023-07-14T19:24:08.282 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:24:07 smithi098 ceph-mon[127486]: from='client.? 172.21.15.98:0/3342608815' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "a3797e89-1754-41cf-9517-227e06ffdd72"}]': finished 2023-07-14T19:24:08.283 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:24:07 smithi098 ceph-mon[127486]: osdmap e5: 1 total, 0 up, 1 in 2023-07-14T19:24:08.283 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:24:07 smithi098 ceph-mon[127486]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2023-07-14T19:24:08.691 INFO:journalctl@ceph.mgr.y.smithi101.stdout:Jul 14 19:24:08 smithi101 ceph-9adea648-227b-11ee-9b33-001a4aab830c-mgr-y[130256]: 2023-07-14T19:24:08.197+0000 7f845a459200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-07-14T19:24:09.191 INFO:journalctl@ceph.mgr.y.smithi101.stdout:Jul 14 19:24:09 smithi101 ceph-9adea648-227b-11ee-9b33-001a4aab830c-mgr-y[130256]: 2023-07-14T19:24:09.041+0000 7f845a459200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-07-14T19:24:09.192 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:24:08 smithi101 ceph-mon[128807]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-14T19:24:09.192 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:24:08 smithi101 ceph-mon[128807]: mgrmap e14: x(active, since 66s), standbys: z 2023-07-14T19:24:09.192 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:24:08 smithi101 ceph-mon[128807]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2023-07-14T19:24:09.192 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:24:08 smithi101 ceph-mon[128807]: from='client.? 172.21.15.98:0/3564377581' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-07-14T19:24:09.220 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:24:08 smithi139 ceph-mon[129293]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-14T19:24:09.221 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:24:08 smithi139 ceph-mon[129293]: mgrmap e14: x(active, since 66s), standbys: z 2023-07-14T19:24:09.221 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:24:08 smithi139 ceph-mon[129293]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2023-07-14T19:24:09.221 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:24:08 smithi139 ceph-mon[129293]: from='client.? 172.21.15.98:0/3564377581' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-07-14T19:24:09.281 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:24:08 smithi098 ceph-mon[127486]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-14T19:24:09.282 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:24:08 smithi098 ceph-mon[127486]: mgrmap e14: x(active, since 66s), standbys: z 2023-07-14T19:24:09.282 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:24:08 smithi098 ceph-mon[127486]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2023-07-14T19:24:09.282 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:24:08 smithi098 ceph-mon[127486]: from='client.? 172.21.15.98:0/3564377581' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-07-14T19:24:09.920 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:24:09.918+0000 7f139a7fc700 1 -- 172.21.15.98:0/202011673 <== mon.0 v2:172.21.15.98:3300/0 8 ==== mgrmap(e 15) v1 ==== 135659+0+0 (secure 0 0 0) 0x7f13a000fa40 con 0x7f13a4100740 2023-07-14T19:24:10.191 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:24:09 smithi101 ceph-mon[128807]: Standby manager daemon y started 2023-07-14T19:24:10.192 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:24:09 smithi101 ceph-mon[128807]: from='mgr.? 172.21.15.101:0/1418233775' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2023-07-14T19:24:10.192 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:24:09 smithi101 ceph-mon[128807]: from='mgr.? 172.21.15.101:0/1418233775' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2023-07-14T19:24:10.192 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:24:09 smithi101 ceph-mon[128807]: from='mgr.? 172.21.15.101:0/1418233775' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2023-07-14T19:24:10.192 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:24:09 smithi101 ceph-mon[128807]: from='mgr.? 172.21.15.101:0/1418233775' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2023-07-14T19:24:10.220 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:24:09 smithi139 ceph-mon[129293]: Standby manager daemon y started 2023-07-14T19:24:10.221 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:24:09 smithi139 ceph-mon[129293]: from='mgr.? 172.21.15.101:0/1418233775' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2023-07-14T19:24:10.221 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:24:09 smithi139 ceph-mon[129293]: from='mgr.? 172.21.15.101:0/1418233775' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2023-07-14T19:24:10.221 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:24:09 smithi139 ceph-mon[129293]: from='mgr.? 172.21.15.101:0/1418233775' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2023-07-14T19:24:10.221 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:24:09 smithi139 ceph-mon[129293]: from='mgr.? 172.21.15.101:0/1418233775' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2023-07-14T19:24:10.281 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:24:09 smithi098 ceph-mon[127486]: Standby manager daemon y started 2023-07-14T19:24:10.281 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:24:09 smithi098 ceph-mon[127486]: from='mgr.? 172.21.15.101:0/1418233775' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2023-07-14T19:24:10.282 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:24:09 smithi098 ceph-mon[127486]: from='mgr.? 172.21.15.101:0/1418233775' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2023-07-14T19:24:10.282 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:24:09 smithi098 ceph-mon[127486]: from='mgr.? 172.21.15.101:0/1418233775' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2023-07-14T19:24:10.282 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:24:09 smithi098 ceph-mon[127486]: from='mgr.? 172.21.15.101:0/1418233775' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2023-07-14T19:24:11.191 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:24:10 smithi101 ceph-mon[128807]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-14T19:24:11.192 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:24:10 smithi101 ceph-mon[128807]: mgrmap e15: x(active, since 68s), standbys: z, y 2023-07-14T19:24:11.192 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:24:10 smithi101 ceph-mon[128807]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2023-07-14T19:24:11.220 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:24:10 smithi139 ceph-mon[129293]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-14T19:24:11.221 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:24:10 smithi139 ceph-mon[129293]: mgrmap e15: x(active, since 68s), standbys: z, y 2023-07-14T19:24:11.221 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:24:10 smithi139 ceph-mon[129293]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2023-07-14T19:24:11.281 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:24:10 smithi098 ceph-mon[127486]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-14T19:24:11.281 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:24:10 smithi098 ceph-mon[127486]: mgrmap e15: x(active, since 68s), standbys: z, y 2023-07-14T19:24:11.282 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:24:10 smithi098 ceph-mon[127486]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2023-07-14T19:24:12.531 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:24:12 smithi098 ceph-mon[127486]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-14T19:24:12.691 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:24:12 smithi101 ceph-mon[128807]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-14T19:24:12.720 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:24:12 smithi139 ceph-mon[129293]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-14T19:24:14.781 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:24:14 smithi098 ceph-mon[127486]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-14T19:24:14.781 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:24:14 smithi098 ceph-mon[127486]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2023-07-14T19:24:14.782 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:24:14 smithi098 ceph-mon[127486]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T19:24:14.941 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:24:14 smithi101 ceph-mon[128807]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-14T19:24:14.942 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:24:14 smithi101 ceph-mon[128807]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2023-07-14T19:24:14.942 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:24:14 smithi101 ceph-mon[128807]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T19:24:14.970 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:24:14 smithi139 ceph-mon[129293]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-14T19:24:14.971 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:24:14 smithi139 ceph-mon[129293]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2023-07-14T19:24:14.971 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:24:14 smithi139 ceph-mon[129293]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T19:24:15.852 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:24:15 smithi098 ceph-mon[127486]: Deploying daemon osd.0 on smithi098 2023-07-14T19:24:15.941 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:24:15 smithi101 ceph-mon[128807]: Deploying daemon osd.0 on smithi098 2023-07-14T19:24:15.970 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:24:15 smithi139 ceph-mon[129293]: Deploying daemon osd.0 on smithi098 2023-07-14T19:24:16.779 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:24:16 smithi098 ceph-mon[127486]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-14T19:24:16.941 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:24:16 smithi101 ceph-mon[128807]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-14T19:24:16.970 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:24:16 smithi139 ceph-mon[129293]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-14T19:24:18.731 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:24:18 smithi098 ceph-mon[127486]: pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-14T19:24:18.941 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:24:18 smithi101 ceph-mon[128807]: pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-14T19:24:18.970 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:24:18 smithi139 ceph-mon[129293]: pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-14T19:24:20.031 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:24:19 smithi098 ceph-mon[127486]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' 2023-07-14T19:24:20.032 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:24:19 smithi098 ceph-mon[127486]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' 2023-07-14T19:24:20.032 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:24:19 smithi098 ceph-mon[127486]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-14T19:24:20.191 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:24:19 smithi101 ceph-mon[128807]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' 2023-07-14T19:24:20.192 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:24:19 smithi101 ceph-mon[128807]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' 2023-07-14T19:24:20.192 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:24:19 smithi101 ceph-mon[128807]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-14T19:24:20.220 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:24:19 smithi139 ceph-mon[129293]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' 2023-07-14T19:24:20.221 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:24:19 smithi139 ceph-mon[129293]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' 2023-07-14T19:24:20.221 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:24:19 smithi139 ceph-mon[129293]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-14T19:24:21.191 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:24:20 smithi101 ceph-mon[128807]: pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-14T19:24:21.192 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:24:20 smithi101 ceph-mon[128807]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' 2023-07-14T19:24:21.220 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:24:20 smithi139 ceph-mon[129293]: pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-14T19:24:21.221 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:24:20 smithi139 ceph-mon[129293]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' 2023-07-14T19:24:21.282 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:24:20 smithi098 ceph-mon[127486]: pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-14T19:24:21.282 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:24:20 smithi098 ceph-mon[127486]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' 2023-07-14T19:24:21.779 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:24:21.777+0000 7f139a7fc700 1 -- 172.21.15.98:0/202011673 <== mgr.14152 v2:172.21.15.98:6800/452042164 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7f13a4060a80 con 0x7f13900389e0 2023-07-14T19:24:21.780 INFO:teuthology.orchestra.run.smithi098.stdout:Created osd(s) 0 on host 'smithi098' 2023-07-14T19:24:21.782 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:24:21.780+0000 7f13ab627700 1 -- 172.21.15.98:0/202011673 >> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] conn(0x7f13900389e0 msgr2=0x7f139003ae90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:24:21.782 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:24:21.780+0000 7f13ab627700 1 --2- 172.21.15.98:0/202011673 >> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] conn(0x7f13900389e0 0x7f139003ae90 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f1394005950 tx=0x7f139400b410 comp rx=0 tx=0).stop 2023-07-14T19:24:21.782 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:24:21.780+0000 7f13ab627700 1 -- 172.21.15.98:0/202011673 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f13a4100740 msgr2=0x7f13a41959c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:24:21.782 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:24:21.780+0000 7f13ab627700 1 --2- 172.21.15.98:0/202011673 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f13a4100740 0x7f13a41959c0 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f13a0000c00 tx=0x7f13a000f690 comp rx=0 tx=0).stop 2023-07-14T19:24:21.783 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:24:21.780+0000 7f13ab627700 1 -- 172.21.15.98:0/202011673 shutdown_connections 2023-07-14T19:24:21.783 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:24:21.780+0000 7f13ab627700 1 --2- 172.21.15.98:0/202011673 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f13a4105980 0x7f13a419cf80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:24:21.783 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:24:21.780+0000 7f13ab627700 1 --2- 172.21.15.98:0/202011673 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f13a4100740 0x7f13a41959c0 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:24:21.783 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:24:21.780+0000 7f13ab627700 1 --2- 172.21.15.98:0/202011673 >> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] conn(0x7f13900389e0 0x7f139003ae90 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:24:21.783 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:24:21.780+0000 7f13ab627700 1 --2- 172.21.15.98:0/202011673 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f13a4103060 0x7f13a4195f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:24:21.783 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:24:21.780+0000 7f13ab627700 1 -- 172.21.15.98:0/202011673 >> 172.21.15.98:0/202011673 conn(0x7f13a40fa1e0 msgr2=0x7f13a41013a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:24:21.783 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:24:21.780+0000 7f13ab627700 1 -- 172.21.15.98:0/202011673 shutdown_connections 2023-07-14T19:24:21.783 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:24:21.780+0000 7f13ab627700 1 -- 172.21.15.98:0/202011673 wait complete. 2023-07-14T19:24:21.933 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:24:21 smithi098 ceph-mon[127486]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' 2023-07-14T19:24:21.934 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:24:21 smithi098 ceph-mon[127486]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T19:24:21.934 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:24:21 smithi098 ceph-mon[127486]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-14T19:24:21.934 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:24:21 smithi098 ceph-mon[127486]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' 2023-07-14T19:24:21.934 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:24:21 smithi098 ceph-mon[127486]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' 2023-07-14T19:24:21.934 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:24:21 smithi098 ceph-mon[127486]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' 2023-07-14T19:24:21.935 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:24:21 smithi098 ceph-mon[127486]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-14T19:24:22.191 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:24:21 smithi101 ceph-mon[128807]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' 2023-07-14T19:24:22.192 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:24:21 smithi101 ceph-mon[128807]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T19:24:22.192 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:24:21 smithi101 ceph-mon[128807]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-14T19:24:22.192 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:24:21 smithi101 ceph-mon[128807]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' 2023-07-14T19:24:22.192 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:24:21 smithi101 ceph-mon[128807]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' 2023-07-14T19:24:22.192 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:24:21 smithi101 ceph-mon[128807]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' 2023-07-14T19:24:22.192 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:24:21 smithi101 ceph-mon[128807]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-14T19:24:22.220 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:24:21 smithi139 ceph-mon[129293]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' 2023-07-14T19:24:22.221 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:24:21 smithi139 ceph-mon[129293]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T19:24:22.221 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:24:21 smithi139 ceph-mon[129293]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-14T19:24:22.221 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:24:21 smithi139 ceph-mon[129293]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' 2023-07-14T19:24:22.221 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:24:21 smithi139 ceph-mon[129293]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' 2023-07-14T19:24:22.221 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:24:21 smithi139 ceph-mon[129293]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' 2023-07-14T19:24:22.221 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:24:21 smithi139 ceph-mon[129293]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-14T19:24:22.281 DEBUG:teuthology.orchestra.run.smithi098:osd.0> sudo journalctl -f -n 0 -u ceph-9adea648-227b-11ee-9b33-001a4aab830c@osd.0.service 2023-07-14T19:24:22.284 INFO:tasks.cephadm:Deploying osd.1 on smithi101 with /dev/vg_nvme/lv_4... 2023-07-14T19:24:22.284 DEBUG:teuthology.orchestra.run.smithi101:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81c9387d2e7576da065747ec9c16d1c7d69016ec shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 9adea648-227b-11ee-9b33-001a4aab830c -- ceph-volume lvm zap /dev/vg_nvme/lv_4 2023-07-14T19:24:22.313 INFO:journalctl@ceph.osd.0.smithi098.stdout:-- Logs begin at Fri 2023-07-14 18:57:10 UTC. -- 2023-07-14T19:24:22.832 INFO:teuthology.orchestra.run.smithi101.stderr:Inferring config /var/lib/ceph/9adea648-227b-11ee-9b33-001a4aab830c/mon.b/config 2023-07-14T19:24:23.016 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:24:22 smithi098 ceph-mon[127486]: pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-14T19:24:23.017 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:24:22 smithi098 ceph-mon[127486]: from='osd.0 [v2:172.21.15.98:6802/120507228,v1:172.21.15.98:6803/120507228]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2023-07-14T19:24:23.017 INFO:journalctl@ceph.osd.0.smithi098.stdout:Jul 14 19:24:22 smithi098 ceph-9adea648-227b-11ee-9b33-001a4aab830c-osd-0[140489]: 2023-07-14T19:24:22.659+0000 7fdc64fab640 -1 osd.0 0 log_to_monitors true 2023-07-14T19:24:23.088 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:24:22 smithi101 ceph-mon[128807]: pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-14T19:24:23.088 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:24:22 smithi101 ceph-mon[128807]: from='osd.0 [v2:172.21.15.98:6802/120507228,v1:172.21.15.98:6803/120507228]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2023-07-14T19:24:23.220 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:24:22 smithi139 ceph-mon[129293]: pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-14T19:24:23.221 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:24:22 smithi139 ceph-mon[129293]: from='osd.0 [v2:172.21.15.98:6802/120507228,v1:172.21.15.98:6803/120507228]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2023-07-14T19:24:24.437 INFO:teuthology.orchestra.run.smithi101.stderr:--> Zapping: /dev/vg_nvme/lv_4 2023-07-14T19:24:24.438 INFO:teuthology.orchestra.run.smithi101.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 conv=fsync 2023-07-14T19:24:24.438 INFO:teuthology.orchestra.run.smithi101.stderr: stderr: 10+0 records in 2023-07-14T19:24:24.438 INFO:teuthology.orchestra.run.smithi101.stderr:10+0 records out 2023-07-14T19:24:24.438 INFO:teuthology.orchestra.run.smithi101.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0109845 s, 955 MB/s 2023-07-14T19:24:24.438 INFO:teuthology.orchestra.run.smithi101.stderr: stderr: 2023-07-14T19:24:24.438 INFO:teuthology.orchestra.run.smithi101.stderr:--> Zapping successful for: 2023-07-14T19:24:24.532 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:24:24 smithi098 ceph-mon[127486]: from='osd.0 [v2:172.21.15.98:6802/120507228,v1:172.21.15.98:6803/120507228]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2023-07-14T19:24:24.532 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:24:24 smithi098 ceph-mon[127486]: osdmap e6: 1 total, 0 up, 1 in 2023-07-14T19:24:24.532 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:24:24 smithi098 ceph-mon[127486]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2023-07-14T19:24:24.532 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:24:24 smithi098 ceph-mon[127486]: from='osd.0 [v2:172.21.15.98:6802/120507228,v1:172.21.15.98:6803/120507228]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi098", "root=default"]}]: dispatch 2023-07-14T19:24:24.533 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:24:24 smithi098 ceph-mon[127486]: pgmap v37: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-14T19:24:24.533 INFO:journalctl@ceph.osd.0.smithi098.stdout:Jul 14 19:24:24 smithi098 ceph-9adea648-227b-11ee-9b33-001a4aab830c-osd-0[140489]: 2023-07-14T19:24:24.239+0000 7fdc59e22700 -1 osd.0 0 waiting for initial osdmap 2023-07-14T19:24:24.533 INFO:journalctl@ceph.osd.0.smithi098.stdout:Jul 14 19:24:24 smithi098 ceph-9adea648-227b-11ee-9b33-001a4aab830c-osd-0[140489]: 2023-07-14T19:24:24.246+0000 7fdc56419700 -1 osd.0 7 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-07-14T19:24:24.615 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:24:24 smithi101 ceph-mon[128807]: from='osd.0 [v2:172.21.15.98:6802/120507228,v1:172.21.15.98:6803/120507228]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2023-07-14T19:24:24.615 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:24:24 smithi101 ceph-mon[128807]: osdmap e6: 1 total, 0 up, 1 in 2023-07-14T19:24:24.616 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:24:24 smithi101 ceph-mon[128807]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2023-07-14T19:24:24.616 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:24:24 smithi101 ceph-mon[128807]: from='osd.0 [v2:172.21.15.98:6802/120507228,v1:172.21.15.98:6803/120507228]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi098", "root=default"]}]: dispatch 2023-07-14T19:24:24.616 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:24:24 smithi101 ceph-mon[128807]: pgmap v37: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-14T19:24:24.720 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:24:24 smithi139 ceph-mon[129293]: from='osd.0 [v2:172.21.15.98:6802/120507228,v1:172.21.15.98:6803/120507228]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2023-07-14T19:24:24.721 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:24:24 smithi139 ceph-mon[129293]: osdmap e6: 1 total, 0 up, 1 in 2023-07-14T19:24:24.721 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:24:24 smithi139 ceph-mon[129293]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2023-07-14T19:24:24.721 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:24:24 smithi139 ceph-mon[129293]: from='osd.0 [v2:172.21.15.98:6802/120507228,v1:172.21.15.98:6803/120507228]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi098", "root=default"]}]: dispatch 2023-07-14T19:24:24.721 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:24:24 smithi139 ceph-mon[129293]: pgmap v37: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-14T19:24:24.872 DEBUG:teuthology.orchestra.run.smithi101:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81c9387d2e7576da065747ec9c16d1c7d69016ec shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 9adea648-227b-11ee-9b33-001a4aab830c -- ceph orch daemon add osd smithi101:vg_nvme/lv_4 2023-07-14T19:24:25.419 INFO:teuthology.orchestra.run.smithi101.stderr:Inferring config /var/lib/ceph/9adea648-227b-11ee-9b33-001a4aab830c/mon.b/config 2023-07-14T19:24:25.531 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:24:25 smithi098 ceph-mon[127486]: from='osd.0 [v2:172.21.15.98:6802/120507228,v1:172.21.15.98:6803/120507228]' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi098", "root=default"]}]': finished 2023-07-14T19:24:25.531 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:24:25 smithi098 ceph-mon[127486]: osdmap e7: 1 total, 0 up, 1 in 2023-07-14T19:24:25.532 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:24:25 smithi098 ceph-mon[127486]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2023-07-14T19:24:25.532 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:24:25 smithi098 ceph-mon[127486]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2023-07-14T19:24:25.532 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:24:25 smithi098 ceph-mon[127486]: Detected new or changed devices on smithi098 2023-07-14T19:24:25.532 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:24:25 smithi098 ceph-mon[127486]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' 2023-07-14T19:24:25.532 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:24:25 smithi098 ceph-mon[127486]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' 2023-07-14T19:24:25.532 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:24:25 smithi098 ceph-mon[127486]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2023-07-14T19:24:25.533 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:24:25 smithi098 ceph-mon[127486]: Adjusting osd_memory_target on smithi098 to 17168M 2023-07-14T19:24:25.533 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:24:25 smithi098 ceph-mon[127486]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' 2023-07-14T19:24:25.533 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:24:25 smithi098 ceph-mon[127486]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T19:24:25.533 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:24:25 smithi098 ceph-mon[127486]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-14T19:24:25.533 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:24:25 smithi098 ceph-mon[127486]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' 2023-07-14T19:24:25.604 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:24:25 smithi101 ceph-mon[128807]: from='osd.0 [v2:172.21.15.98:6802/120507228,v1:172.21.15.98:6803/120507228]' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi098", "root=default"]}]': finished 2023-07-14T19:24:25.604 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:24:25 smithi101 ceph-mon[128807]: osdmap e7: 1 total, 0 up, 1 in 2023-07-14T19:24:25.605 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:24:25 smithi101 ceph-mon[128807]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2023-07-14T19:24:25.605 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:24:25 smithi101 ceph-mon[128807]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2023-07-14T19:24:25.605 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:24:25 smithi101 ceph-mon[128807]: Detected new or changed devices on smithi098 2023-07-14T19:24:25.605 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:24:25 smithi101 ceph-mon[128807]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' 2023-07-14T19:24:25.605 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:24:25 smithi101 ceph-mon[128807]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' 2023-07-14T19:24:25.605 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:24:25 smithi101 ceph-mon[128807]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2023-07-14T19:24:25.606 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:24:25 smithi101 ceph-mon[128807]: Adjusting osd_memory_target on smithi098 to 17168M 2023-07-14T19:24:25.606 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:24:25 smithi101 ceph-mon[128807]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' 2023-07-14T19:24:25.606 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:24:25 smithi101 ceph-mon[128807]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T19:24:25.606 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:24:25 smithi101 ceph-mon[128807]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-14T19:24:25.606 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:24:25 smithi101 ceph-mon[128807]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' 2023-07-14T19:24:25.720 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:24:25 smithi139 ceph-mon[129293]: from='osd.0 [v2:172.21.15.98:6802/120507228,v1:172.21.15.98:6803/120507228]' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi098", "root=default"]}]': finished 2023-07-14T19:24:25.721 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:24:25 smithi139 ceph-mon[129293]: osdmap e7: 1 total, 0 up, 1 in 2023-07-14T19:24:25.721 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:24:25 smithi139 ceph-mon[129293]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2023-07-14T19:24:25.721 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:24:25 smithi139 ceph-mon[129293]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2023-07-14T19:24:25.721 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:24:25 smithi139 ceph-mon[129293]: Detected new or changed devices on smithi098 2023-07-14T19:24:25.721 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:24:25 smithi139 ceph-mon[129293]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' 2023-07-14T19:24:25.721 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:24:25 smithi139 ceph-mon[129293]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' 2023-07-14T19:24:25.721 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:24:25 smithi139 ceph-mon[129293]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2023-07-14T19:24:25.722 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:24:25 smithi139 ceph-mon[129293]: Adjusting osd_memory_target on smithi098 to 17168M 2023-07-14T19:24:25.722 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:24:25 smithi139 ceph-mon[129293]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' 2023-07-14T19:24:25.722 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:24:25 smithi139 ceph-mon[129293]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T19:24:25.722 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:24:25 smithi139 ceph-mon[129293]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-14T19:24:25.722 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:24:25 smithi139 ceph-mon[129293]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' 2023-07-14T19:24:26.531 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:24:26 smithi098 ceph-mon[127486]: purged_snaps scrub starts 2023-07-14T19:24:26.532 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:24:26 smithi098 ceph-mon[127486]: purged_snaps scrub ok 2023-07-14T19:24:26.532 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:24:26 smithi098 ceph-mon[127486]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2023-07-14T19:24:26.532 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:24:26 smithi098 ceph-mon[127486]: osd.0 [v2:172.21.15.98:6802/120507228,v1:172.21.15.98:6803/120507228] boot 2023-07-14T19:24:26.532 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:24:26 smithi098 ceph-mon[127486]: osdmap e8: 1 total, 1 up, 1 in 2023-07-14T19:24:26.532 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:24:26 smithi098 ceph-mon[127486]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2023-07-14T19:24:26.532 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:24:26 smithi098 ceph-mon[127486]: pgmap v40: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-14T19:24:26.623 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-14T19:24:26.620+0000 7f186fae9700 1 -- 172.21.15.101:0/2125968169 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f18681012c0 msgr2=0x7f18681016d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:24:26.624 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-14T19:24:26.621+0000 7f186fae9700 1 --2- 172.21.15.101:0/2125968169 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f18681012c0 0x7f18681016d0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f1858009990 tx=0x7f1858009ca0 comp rx=0 tx=0).stop 2023-07-14T19:24:26.624 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-14T19:24:26.621+0000 7f186fae9700 1 -- 172.21.15.101:0/2125968169 shutdown_connections 2023-07-14T19:24:26.624 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-14T19:24:26.621+0000 7f186fae9700 1 --2- 172.21.15.101:0/2125968169 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f18680fe3f0 0x7f18681082b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:24:26.624 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-14T19:24:26.621+0000 7f186fae9700 1 --2- 172.21.15.101:0/2125968169 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f18681012c0 0x7f18681016d0 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:24:26.624 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-14T19:24:26.621+0000 7f186fae9700 1 --2- 172.21.15.101:0/2125968169 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1868101ca0 0x7f18680fdd60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:24:26.624 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-14T19:24:26.621+0000 7f186fae9700 1 -- 172.21.15.101:0/2125968169 >> 172.21.15.101:0/2125968169 conn(0x7f18680f9990 msgr2=0x7f18680fbdc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:24:26.625 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-14T19:24:26.621+0000 7f186fae9700 1 -- 172.21.15.101:0/2125968169 shutdown_connections 2023-07-14T19:24:26.625 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-14T19:24:26.621+0000 7f186fae9700 1 -- 172.21.15.101:0/2125968169 wait complete. 2023-07-14T19:24:26.625 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-14T19:24:26.621+0000 7f186fae9700 1 Processor -- start 2023-07-14T19:24:26.625 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-14T19:24:26.621+0000 7f186fae9700 1 -- start start 2023-07-14T19:24:26.625 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-14T19:24:26.621+0000 7f186fae9700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f18680fe3f0 0x7f18681959b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:24:26.625 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-14T19:24:26.621+0000 7f186fae9700 1 --2- >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f18681012c0 0x7f1868195ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:24:26.625 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-14T19:24:26.621+0000 7f186fae9700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1868101ca0 0x7f186819cf70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:24:26.626 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-14T19:24:26.622+0000 7f186fae9700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f186819d540 con 0x7f1868101ca0 2023-07-14T19:24:26.626 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-14T19:24:26.622+0000 7f186fae9700 1 -- --> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f186819d680 con 0x7f18681012c0 2023-07-14T19:24:26.626 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-14T19:24:26.622+0000 7f186fae9700 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f186819d810 con 0x7f18680fe3f0 2023-07-14T19:24:26.626 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-14T19:24:26.622+0000 7f186d084700 1 --2- >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f18681012c0 0x7f1868195ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:24:26.626 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-14T19:24:26.622+0000 7f186d084700 1 --2- >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f18681012c0 0x7f1868195ef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.101:3300/0 says I am v2:172.21.15.101:36020/0 (socket says 172.21.15.101:36020) 2023-07-14T19:24:26.626 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-14T19:24:26.622+0000 7f186d084700 1 -- 172.21.15.101:0/2207042557 learned_addr learned my addr 172.21.15.101:0/2207042557 (peer_addr_for_me v2:172.21.15.101:0/0) 2023-07-14T19:24:26.626 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-14T19:24:26.622+0000 7f186d885700 1 --2- 172.21.15.101:0/2207042557 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f18680fe3f0 0x7f18681959b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:24:26.627 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-14T19:24:26.622+0000 7f186e086700 1 --2- 172.21.15.101:0/2207042557 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1868101ca0 0x7f186819cf70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:24:26.627 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-14T19:24:26.622+0000 7f186d084700 1 -- 172.21.15.101:0/2207042557 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f18680fe3f0 msgr2=0x7f18681959b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:24:26.627 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-14T19:24:26.622+0000 7f186d084700 1 --2- 172.21.15.101:0/2207042557 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f18680fe3f0 0x7f18681959b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:24:26.627 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-14T19:24:26.622+0000 7f186d084700 1 -- 172.21.15.101:0/2207042557 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1868101ca0 msgr2=0x7f186819cf70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:24:26.627 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-14T19:24:26.622+0000 7f186d084700 1 --2- 172.21.15.101:0/2207042557 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1868101ca0 0x7f186819cf70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:24:26.627 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-14T19:24:26.622+0000 7f186d084700 1 -- 172.21.15.101:0/2207042557 --> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f18580095f0 con 0x7f18681012c0 2023-07-14T19:24:26.627 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-14T19:24:26.622+0000 7f186e086700 1 --2- 172.21.15.101:0/2207042557 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1868101ca0 0x7f186819cf70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-07-14T19:24:26.628 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-14T19:24:26.622+0000 7f186d084700 1 --2- 172.21.15.101:0/2207042557 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f18681012c0 0x7f1868195ef0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f186400c8f0 tx=0x7f186400cc00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:24:26.628 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-14T19:24:26.622+0000 7f185effd700 1 -- 172.21.15.101:0/2207042557 <== mon.2 v2:172.21.15.101:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1864007940 con 0x7f18681012c0 2023-07-14T19:24:26.628 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-14T19:24:26.622+0000 7f186fae9700 1 -- 172.21.15.101:0/2207042557 --> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f186819daf0 con 0x7f18681012c0 2023-07-14T19:24:26.628 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-14T19:24:26.622+0000 7f185effd700 1 -- 172.21.15.101:0/2207042557 <== mon.2 v2:172.21.15.101:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f186400f450 con 0x7f18681012c0 2023-07-14T19:24:26.628 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-14T19:24:26.622+0000 7f185effd700 1 -- 172.21.15.101:0/2207042557 <== mon.2 v2:172.21.15.101:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1864018660 con 0x7f18681012c0 2023-07-14T19:24:26.628 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-14T19:24:26.622+0000 7f186fae9700 1 -- 172.21.15.101:0/2207042557 --> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f186819dff0 con 0x7f18681012c0 2023-07-14T19:24:26.629 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-14T19:24:26.623+0000 7f186fae9700 1 -- 172.21.15.101:0/2207042557 --> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f186810b4c0 con 0x7f18681012c0 2023-07-14T19:24:26.629 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-14T19:24:26.624+0000 7f185effd700 1 -- 172.21.15.101:0/2207042557 <== mon.2 v2:172.21.15.101:3300/0 4 ==== mgrmap(e 15) v1 ==== 135659+0+0 (secure 0 0 0) 0x7f186400f5c0 con 0x7f18681012c0 2023-07-14T19:24:26.630 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-14T19:24:26.625+0000 7f185effd700 1 --2- 172.21.15.101:0/2207042557 >> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] conn(0x7f18540a10c0 0x7f18540a3570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:24:26.630 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-14T19:24:26.625+0000 7f185effd700 1 -- 172.21.15.101:0/2207042557 <== mon.2 v2:172.21.15.101:3300/0 5 ==== osd_map(9..9 src has 1..9) v4 ==== 1767+0+0 (secure 0 0 0) 0x7f18640cad30 con 0x7f18681012c0 2023-07-14T19:24:26.630 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-14T19:24:26.625+0000 7f186d885700 1 --2- 172.21.15.101:0/2207042557 >> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] conn(0x7f18540a10c0 0x7f18540a3570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:24:26.630 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-14T19:24:26.626+0000 7f186d885700 1 --2- 172.21.15.101:0/2207042557 >> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] conn(0x7f18540a10c0 0x7f18540a3570 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f1858006010 tx=0x7f1858005960 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:24:26.630 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-14T19:24:26.629+0000 7f185effd700 1 -- 172.21.15.101:0/2207042557 <== mon.2 v2:172.21.15.101:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f18640cf070 con 0x7f18681012c0 2023-07-14T19:24:26.691 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:24:26 smithi101 ceph-mon[128807]: purged_snaps scrub starts 2023-07-14T19:24:26.692 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:24:26 smithi101 ceph-mon[128807]: purged_snaps scrub ok 2023-07-14T19:24:26.692 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:24:26 smithi101 ceph-mon[128807]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2023-07-14T19:24:26.692 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:24:26 smithi101 ceph-mon[128807]: osd.0 [v2:172.21.15.98:6802/120507228,v1:172.21.15.98:6803/120507228] boot 2023-07-14T19:24:26.692 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:24:26 smithi101 ceph-mon[128807]: osdmap e8: 1 total, 1 up, 1 in 2023-07-14T19:24:26.692 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:24:26 smithi101 ceph-mon[128807]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2023-07-14T19:24:26.692 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:24:26 smithi101 ceph-mon[128807]: pgmap v40: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-14T19:24:26.720 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:24:26 smithi139 ceph-mon[129293]: purged_snaps scrub starts 2023-07-14T19:24:26.721 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:24:26 smithi139 ceph-mon[129293]: purged_snaps scrub ok 2023-07-14T19:24:26.721 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:24:26 smithi139 ceph-mon[129293]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2023-07-14T19:24:26.721 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:24:26 smithi139 ceph-mon[129293]: osd.0 [v2:172.21.15.98:6802/120507228,v1:172.21.15.98:6803/120507228] boot 2023-07-14T19:24:26.721 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:24:26 smithi139 ceph-mon[129293]: osdmap e8: 1 total, 1 up, 1 in 2023-07-14T19:24:26.721 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:24:26 smithi139 ceph-mon[129293]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2023-07-14T19:24:26.721 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:24:26 smithi139 ceph-mon[129293]: pgmap v40: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-14T19:24:26.821 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-14T19:24:26.820+0000 7f186fae9700 1 -- 172.21.15.101:0/2207042557 --> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi101:vg_nvme/lv_4", "target": ["mon-mgr", ""]}) v1 -- 0x7f1868060a80 con 0x7f18540a10c0 2023-07-14T19:24:27.531 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:24:27 smithi098 ceph-mon[127486]: osdmap e9: 1 total, 1 up, 1 in 2023-07-14T19:24:27.532 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:24:27 smithi098 ceph-mon[127486]: from='client.24149 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi101:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2023-07-14T19:24:27.532 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:24:27 smithi098 ceph-mon[127486]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-07-14T19:24:27.532 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:24:27 smithi098 ceph-mon[127486]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-07-14T19:24:27.532 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:24:27 smithi098 ceph-mon[127486]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T19:24:27.628 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:24:27 smithi101 ceph-mon[128807]: osdmap e9: 1 total, 1 up, 1 in 2023-07-14T19:24:27.628 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:24:27 smithi101 ceph-mon[128807]: from='client.24149 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi101:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2023-07-14T19:24:27.629 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:24:27 smithi101 ceph-mon[128807]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-07-14T19:24:27.629 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:24:27 smithi101 ceph-mon[128807]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-07-14T19:24:27.629 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:24:27 smithi101 ceph-mon[128807]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T19:24:27.720 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:24:27 smithi139 ceph-mon[129293]: osdmap e9: 1 total, 1 up, 1 in 2023-07-14T19:24:27.721 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:24:27 smithi139 ceph-mon[129293]: from='client.24149 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi101:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2023-07-14T19:24:27.721 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:24:27 smithi139 ceph-mon[129293]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-07-14T19:24:27.721 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:24:27 smithi139 ceph-mon[129293]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-07-14T19:24:27.721 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:24:27 smithi139 ceph-mon[129293]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T19:24:28.531 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:24:28 smithi098 ceph-mon[127486]: pgmap v42: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2023-07-14T19:24:28.691 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:24:28 smithi101 ceph-mon[128807]: pgmap v42: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2023-07-14T19:24:28.720 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:24:28 smithi139 ceph-mon[129293]: pgmap v42: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2023-07-14T19:24:29.531 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:24:29 smithi098 ceph-mon[127486]: from='client.? 172.21.15.101:0/1561196728' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "2b001add-1aaa-4496-a403-7ef9725b80a0"}]: dispatch 2023-07-14T19:24:29.531 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:24:29 smithi098 ceph-mon[127486]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "2b001add-1aaa-4496-a403-7ef9725b80a0"}]: dispatch 2023-07-14T19:24:29.532 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:24:29 smithi098 ceph-mon[127486]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "2b001add-1aaa-4496-a403-7ef9725b80a0"}]': finished 2023-07-14T19:24:29.532 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:24:29 smithi098 ceph-mon[127486]: osdmap e10: 2 total, 1 up, 2 in 2023-07-14T19:24:29.532 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:24:29 smithi098 ceph-mon[127486]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2023-07-14T19:24:29.692 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:24:29 smithi101 ceph-mon[128807]: from='client.? 172.21.15.101:0/1561196728' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "2b001add-1aaa-4496-a403-7ef9725b80a0"}]: dispatch 2023-07-14T19:24:29.692 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:24:29 smithi101 ceph-mon[128807]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "2b001add-1aaa-4496-a403-7ef9725b80a0"}]: dispatch 2023-07-14T19:24:29.692 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:24:29 smithi101 ceph-mon[128807]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "2b001add-1aaa-4496-a403-7ef9725b80a0"}]': finished 2023-07-14T19:24:29.692 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:24:29 smithi101 ceph-mon[128807]: osdmap e10: 2 total, 1 up, 2 in 2023-07-14T19:24:29.692 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:24:29 smithi101 ceph-mon[128807]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2023-07-14T19:24:29.720 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:24:29 smithi139 ceph-mon[129293]: from='client.? 172.21.15.101:0/1561196728' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "2b001add-1aaa-4496-a403-7ef9725b80a0"}]: dispatch 2023-07-14T19:24:29.721 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:24:29 smithi139 ceph-mon[129293]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "2b001add-1aaa-4496-a403-7ef9725b80a0"}]: dispatch 2023-07-14T19:24:29.721 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:24:29 smithi139 ceph-mon[129293]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "2b001add-1aaa-4496-a403-7ef9725b80a0"}]': finished 2023-07-14T19:24:29.721 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:24:29 smithi139 ceph-mon[129293]: osdmap e10: 2 total, 1 up, 2 in 2023-07-14T19:24:29.721 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:24:29 smithi139 ceph-mon[129293]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2023-07-14T19:24:30.531 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:24:30 smithi098 ceph-mon[127486]: pgmap v44: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2023-07-14T19:24:30.532 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:24:30 smithi098 ceph-mon[127486]: from='client.? 172.21.15.101:0/1465619544' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-07-14T19:24:30.691 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:24:30 smithi101 ceph-mon[128807]: pgmap v44: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2023-07-14T19:24:30.692 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:24:30 smithi101 ceph-mon[128807]: from='client.? 172.21.15.101:0/1465619544' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-07-14T19:24:30.720 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:24:30 smithi139 ceph-mon[129293]: pgmap v44: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2023-07-14T19:24:30.721 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:24:30 smithi139 ceph-mon[129293]: from='client.? 172.21.15.101:0/1465619544' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-07-14T19:24:32.531 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:24:32 smithi098 ceph-mon[127486]: pgmap v45: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2023-07-14T19:24:32.691 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:24:32 smithi101 ceph-mon[128807]: pgmap v45: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2023-07-14T19:24:32.720 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:24:32 smithi139 ceph-mon[129293]: pgmap v45: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2023-07-14T19:24:34.692 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:24:34 smithi101 ceph-mon[128807]: pgmap v46: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2023-07-14T19:24:34.971 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:24:34 smithi139 ceph-mon[129293]: pgmap v46: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2023-07-14T19:24:35.031 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:24:34 smithi098 ceph-mon[127486]: pgmap v46: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2023-07-14T19:24:36.893 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:24:36 smithi101 ceph-mon[128807]: pgmap v47: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2023-07-14T19:24:36.894 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:24:36 smithi101 ceph-mon[128807]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2023-07-14T19:24:36.894 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:24:36 smithi101 ceph-mon[128807]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T19:24:36.970 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:24:36 smithi139 ceph-mon[129293]: pgmap v47: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2023-07-14T19:24:36.971 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:24:36 smithi139 ceph-mon[129293]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2023-07-14T19:24:36.971 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:24:36 smithi139 ceph-mon[129293]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T19:24:37.031 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:24:36 smithi098 ceph-mon[127486]: pgmap v47: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2023-07-14T19:24:37.031 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:24:36 smithi098 ceph-mon[127486]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2023-07-14T19:24:37.032 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:24:36 smithi098 ceph-mon[127486]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T19:24:37.882 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:24:37 smithi101 ceph-mon[128807]: Deploying daemon osd.1 on smithi101 2023-07-14T19:24:37.970 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:24:37 smithi139 ceph-mon[129293]: Deploying daemon osd.1 on smithi101 2023-07-14T19:24:38.031 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:24:37 smithi098 ceph-mon[127486]: Deploying daemon osd.1 on smithi101 2023-07-14T19:24:38.835 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:24:38 smithi101 ceph-mon[128807]: pgmap v48: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2023-07-14T19:24:38.970 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:24:38 smithi139 ceph-mon[129293]: pgmap v48: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2023-07-14T19:24:39.031 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:24:38 smithi098 ceph-mon[127486]: pgmap v48: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2023-07-14T19:24:40.791 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:24:40 smithi101 ceph-mon[128807]: pgmap v49: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2023-07-14T19:24:40.970 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:24:40 smithi139 ceph-mon[129293]: pgmap v49: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2023-07-14T19:24:41.031 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:24:40 smithi098 ceph-mon[127486]: pgmap v49: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2023-07-14T19:24:41.941 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:24:41 smithi101 ceph-mon[128807]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' 2023-07-14T19:24:41.942 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:24:41 smithi101 ceph-mon[128807]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' 2023-07-14T19:24:41.942 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:24:41 smithi101 ceph-mon[128807]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-14T19:24:42.220 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:24:41 smithi139 ceph-mon[129293]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' 2023-07-14T19:24:42.221 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:24:41 smithi139 ceph-mon[129293]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' 2023-07-14T19:24:42.221 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:24:41 smithi139 ceph-mon[129293]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-14T19:24:42.281 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:24:41 smithi098 ceph-mon[127486]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' 2023-07-14T19:24:42.281 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:24:41 smithi098 ceph-mon[127486]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' 2023-07-14T19:24:42.282 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:24:41 smithi098 ceph-mon[127486]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-14T19:24:43.191 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:24:42 smithi101 ceph-mon[128807]: pgmap v50: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2023-07-14T19:24:43.220 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:24:42 smithi139 ceph-mon[129293]: pgmap v50: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2023-07-14T19:24:43.281 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:24:42 smithi098 ceph-mon[127486]: pgmap v50: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2023-07-14T19:24:43.633 INFO:teuthology.orchestra.run.smithi101.stdout:Created osd(s) 1 on host 'smithi101' 2023-07-14T19:24:43.634 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-14T19:24:43.630+0000 7f185effd700 1 -- 172.21.15.101:0/2207042557 <== mgr.14152 v2:172.21.15.98:6800/452042164 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7f1868060a80 con 0x7f18540a10c0 2023-07-14T19:24:43.635 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-14T19:24:43.633+0000 7f186fae9700 1 -- 172.21.15.101:0/2207042557 >> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] conn(0x7f18540a10c0 msgr2=0x7f18540a3570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:24:43.635 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-14T19:24:43.633+0000 7f186fae9700 1 --2- 172.21.15.101:0/2207042557 >> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] conn(0x7f18540a10c0 0x7f18540a3570 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f1858006010 tx=0x7f1858005960 comp rx=0 tx=0).stop 2023-07-14T19:24:43.635 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-14T19:24:43.633+0000 7f186fae9700 1 -- 172.21.15.101:0/2207042557 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f18681012c0 msgr2=0x7f1868195ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:24:43.635 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-14T19:24:43.633+0000 7f186fae9700 1 --2- 172.21.15.101:0/2207042557 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f18681012c0 0x7f1868195ef0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f186400c8f0 tx=0x7f186400cc00 comp rx=0 tx=0).stop 2023-07-14T19:24:43.635 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-14T19:24:43.633+0000 7f186fae9700 1 -- 172.21.15.101:0/2207042557 shutdown_connections 2023-07-14T19:24:43.636 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-14T19:24:43.633+0000 7f186fae9700 1 --2- 172.21.15.101:0/2207042557 >> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] conn(0x7f18540a10c0 0x7f18540a3570 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:24:43.636 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-14T19:24:43.633+0000 7f186fae9700 1 --2- 172.21.15.101:0/2207042557 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f18680fe3f0 0x7f18681959b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:24:43.636 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-14T19:24:43.633+0000 7f186fae9700 1 --2- 172.21.15.101:0/2207042557 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1868101ca0 0x7f186819cf70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:24:43.636 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-14T19:24:43.633+0000 7f186fae9700 1 --2- 172.21.15.101:0/2207042557 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f18681012c0 0x7f1868195ef0 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:24:43.636 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-14T19:24:43.633+0000 7f186fae9700 1 -- 172.21.15.101:0/2207042557 >> 172.21.15.101:0/2207042557 conn(0x7f18680f9990 msgr2=0x7f18680fbbf0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:24:43.636 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-14T19:24:43.633+0000 7f186fae9700 1 -- 172.21.15.101:0/2207042557 shutdown_connections 2023-07-14T19:24:43.637 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-14T19:24:43.633+0000 7f186fae9700 1 -- 172.21.15.101:0/2207042557 wait complete. 2023-07-14T19:24:44.191 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:24:43 smithi101 ceph-mon[128807]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' 2023-07-14T19:24:44.191 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:24:43 smithi101 ceph-mon[128807]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' 2023-07-14T19:24:44.192 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:24:43 smithi101 ceph-mon[128807]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T19:24:44.192 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:24:43 smithi101 ceph-mon[128807]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-14T19:24:44.192 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:24:43 smithi101 ceph-mon[128807]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' 2023-07-14T19:24:44.192 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:24:43 smithi101 ceph-mon[128807]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' 2023-07-14T19:24:44.192 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:24:43 smithi101 ceph-mon[128807]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' 2023-07-14T19:24:44.192 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:24:43 smithi101 ceph-mon[128807]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-14T19:24:44.208 DEBUG:teuthology.orchestra.run.smithi101:osd.1> sudo journalctl -f -n 0 -u ceph-9adea648-227b-11ee-9b33-001a4aab830c@osd.1.service 2023-07-14T19:24:44.211 INFO:tasks.cephadm:Deploying osd.2 on smithi139 with /dev/vg_nvme/lv_4... 2023-07-14T19:24:44.211 DEBUG:teuthology.orchestra.run.smithi139:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81c9387d2e7576da065747ec9c16d1c7d69016ec shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 9adea648-227b-11ee-9b33-001a4aab830c -- ceph-volume lvm zap /dev/vg_nvme/lv_4 2023-07-14T19:24:44.220 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:24:43 smithi139 ceph-mon[129293]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' 2023-07-14T19:24:44.220 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:24:43 smithi139 ceph-mon[129293]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' 2023-07-14T19:24:44.221 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:24:43 smithi139 ceph-mon[129293]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T19:24:44.221 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:24:43 smithi139 ceph-mon[129293]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-14T19:24:44.221 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:24:43 smithi139 ceph-mon[129293]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' 2023-07-14T19:24:44.221 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:24:43 smithi139 ceph-mon[129293]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' 2023-07-14T19:24:44.221 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:24:43 smithi139 ceph-mon[129293]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' 2023-07-14T19:24:44.222 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:24:43 smithi139 ceph-mon[129293]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-14T19:24:44.239 INFO:journalctl@ceph.osd.1.smithi101.stdout:-- Logs begin at Fri 2023-07-14 18:56:58 UTC. -- 2023-07-14T19:24:44.281 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:24:43 smithi098 ceph-mon[127486]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' 2023-07-14T19:24:44.282 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:24:43 smithi098 ceph-mon[127486]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' 2023-07-14T19:24:44.282 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:24:43 smithi098 ceph-mon[127486]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T19:24:44.282 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:24:43 smithi098 ceph-mon[127486]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-14T19:24:44.282 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:24:43 smithi098 ceph-mon[127486]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' 2023-07-14T19:24:44.282 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:24:43 smithi098 ceph-mon[127486]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' 2023-07-14T19:24:44.282 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:24:43 smithi098 ceph-mon[127486]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' 2023-07-14T19:24:44.282 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:24:43 smithi098 ceph-mon[127486]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-14T19:24:44.771 INFO:teuthology.orchestra.run.smithi139.stderr:Inferring config /var/lib/ceph/9adea648-227b-11ee-9b33-001a4aab830c/mon.c/config 2023-07-14T19:24:44.942 INFO:journalctl@ceph.osd.1.smithi101.stdout:Jul 14 19:24:44 smithi101 ceph-9adea648-227b-11ee-9b33-001a4aab830c-osd-1[133877]: 2023-07-14T19:24:44.754+0000 7fab8467f640 -1 osd.1 0 log_to_monitors true 2023-07-14T19:24:45.221 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:24:44 smithi139 ceph-mon[129293]: pgmap v51: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2023-07-14T19:24:45.221 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:24:44 smithi139 ceph-mon[129293]: from='osd.1 [v2:172.21.15.101:6800/2450584584,v1:172.21.15.101:6801/2450584584]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2023-07-14T19:24:45.221 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:24:44 smithi139 ceph-mon[129293]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2023-07-14T19:24:45.281 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:24:44 smithi098 ceph-mon[127486]: pgmap v51: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2023-07-14T19:24:45.281 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:24:44 smithi098 ceph-mon[127486]: from='osd.1 [v2:172.21.15.101:6800/2450584584,v1:172.21.15.101:6801/2450584584]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2023-07-14T19:24:45.282 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:24:44 smithi098 ceph-mon[127486]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2023-07-14T19:24:45.402 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:24:44 smithi101 ceph-mon[128807]: pgmap v51: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2023-07-14T19:24:45.402 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:24:44 smithi101 ceph-mon[128807]: from='osd.1 [v2:172.21.15.101:6800/2450584584,v1:172.21.15.101:6801/2450584584]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2023-07-14T19:24:45.402 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:24:44 smithi101 ceph-mon[128807]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2023-07-14T19:24:46.221 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:24:45 smithi139 ceph-mon[129293]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2023-07-14T19:24:46.221 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:24:45 smithi139 ceph-mon[129293]: osdmap e11: 2 total, 1 up, 2 in 2023-07-14T19:24:46.221 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:24:45 smithi139 ceph-mon[129293]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2023-07-14T19:24:46.221 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:24:45 smithi139 ceph-mon[129293]: from='osd.1 [v2:172.21.15.101:6800/2450584584,v1:172.21.15.101:6801/2450584584]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi101", "root=default"]}]: dispatch 2023-07-14T19:24:46.221 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:24:45 smithi139 ceph-mon[129293]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi101", "root=default"]}]: dispatch 2023-07-14T19:24:46.255 INFO:teuthology.orchestra.run.smithi139.stderr:--> Zapping: /dev/vg_nvme/lv_4 2023-07-14T19:24:46.255 INFO:teuthology.orchestra.run.smithi139.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 conv=fsync 2023-07-14T19:24:46.255 INFO:teuthology.orchestra.run.smithi139.stderr: stderr: 10+0 records in 2023-07-14T19:24:46.256 INFO:teuthology.orchestra.run.smithi139.stderr:10+0 records out 2023-07-14T19:24:46.256 INFO:teuthology.orchestra.run.smithi139.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0125123 s, 838 MB/s 2023-07-14T19:24:46.256 INFO:teuthology.orchestra.run.smithi139.stderr: stderr: 2023-07-14T19:24:46.256 INFO:teuthology.orchestra.run.smithi139.stderr:--> Zapping successful for: 2023-07-14T19:24:46.281 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:24:45 smithi098 ceph-mon[127486]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2023-07-14T19:24:46.282 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:24:45 smithi098 ceph-mon[127486]: osdmap e11: 2 total, 1 up, 2 in 2023-07-14T19:24:46.282 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:24:45 smithi098 ceph-mon[127486]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2023-07-14T19:24:46.282 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:24:45 smithi098 ceph-mon[127486]: from='osd.1 [v2:172.21.15.101:6800/2450584584,v1:172.21.15.101:6801/2450584584]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi101", "root=default"]}]: dispatch 2023-07-14T19:24:46.282 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:24:45 smithi098 ceph-mon[127486]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi101", "root=default"]}]: dispatch 2023-07-14T19:24:46.303 INFO:journalctl@ceph.osd.1.smithi101.stdout:Jul 14 19:24:45 smithi101 ceph-9adea648-227b-11ee-9b33-001a4aab830c-osd-1[133877]: 2023-07-14T19:24:45.959+0000 7fab794f6700 -1 osd.1 0 waiting for initial osdmap 2023-07-14T19:24:46.304 INFO:journalctl@ceph.osd.1.smithi101.stdout:Jul 14 19:24:45 smithi101 ceph-9adea648-227b-11ee-9b33-001a4aab830c-osd-1[133877]: 2023-07-14T19:24:45.962+0000 7fab742ea700 -1 osd.1 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-07-14T19:24:46.304 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:24:45 smithi101 ceph-mon[128807]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2023-07-14T19:24:46.304 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:24:45 smithi101 ceph-mon[128807]: osdmap e11: 2 total, 1 up, 2 in 2023-07-14T19:24:46.304 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:24:45 smithi101 ceph-mon[128807]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2023-07-14T19:24:46.305 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:24:45 smithi101 ceph-mon[128807]: from='osd.1 [v2:172.21.15.101:6800/2450584584,v1:172.21.15.101:6801/2450584584]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi101", "root=default"]}]: dispatch 2023-07-14T19:24:46.305 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:24:45 smithi101 ceph-mon[128807]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi101", "root=default"]}]: dispatch 2023-07-14T19:24:46.709 DEBUG:teuthology.orchestra.run.smithi139:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81c9387d2e7576da065747ec9c16d1c7d69016ec shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 9adea648-227b-11ee-9b33-001a4aab830c -- ceph orch daemon add osd smithi139:vg_nvme/lv_4 2023-07-14T19:24:47.220 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:24:46 smithi139 ceph-mon[129293]: pgmap v53: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2023-07-14T19:24:47.221 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:24:46 smithi139 ceph-mon[129293]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi101", "root=default"]}]': finished 2023-07-14T19:24:47.221 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:24:46 smithi139 ceph-mon[129293]: osdmap e12: 2 total, 1 up, 2 in 2023-07-14T19:24:47.221 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:24:46 smithi139 ceph-mon[129293]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2023-07-14T19:24:47.221 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:24:46 smithi139 ceph-mon[129293]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2023-07-14T19:24:47.221 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:24:46 smithi139 ceph-mon[129293]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' 2023-07-14T19:24:47.221 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:24:46 smithi139 ceph-mon[129293]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' 2023-07-14T19:24:47.222 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:24:46 smithi139 ceph-mon[129293]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2023-07-14T19:24:47.222 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:24:46 smithi139 ceph-mon[129293]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' 2023-07-14T19:24:47.222 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:24:46 smithi139 ceph-mon[129293]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T19:24:47.222 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:24:46 smithi139 ceph-mon[129293]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-14T19:24:47.222 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:24:46 smithi139 ceph-mon[129293]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' 2023-07-14T19:24:47.281 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:24:46 smithi098 ceph-mon[127486]: pgmap v53: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2023-07-14T19:24:47.281 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:24:46 smithi098 ceph-mon[127486]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi101", "root=default"]}]': finished 2023-07-14T19:24:47.282 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:24:46 smithi098 ceph-mon[127486]: osdmap e12: 2 total, 1 up, 2 in 2023-07-14T19:24:47.282 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:24:46 smithi098 ceph-mon[127486]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2023-07-14T19:24:47.282 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:24:46 smithi098 ceph-mon[127486]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2023-07-14T19:24:47.282 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:24:46 smithi098 ceph-mon[127486]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' 2023-07-14T19:24:47.282 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:24:46 smithi098 ceph-mon[127486]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' 2023-07-14T19:24:47.283 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:24:46 smithi098 ceph-mon[127486]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2023-07-14T19:24:47.283 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:24:46 smithi098 ceph-mon[127486]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' 2023-07-14T19:24:47.283 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:24:46 smithi098 ceph-mon[127486]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T19:24:47.283 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:24:46 smithi098 ceph-mon[127486]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-14T19:24:47.283 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:24:46 smithi098 ceph-mon[127486]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' 2023-07-14T19:24:47.297 INFO:teuthology.orchestra.run.smithi139.stderr:Inferring config /var/lib/ceph/9adea648-227b-11ee-9b33-001a4aab830c/mon.c/config 2023-07-14T19:24:47.441 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:24:46 smithi101 ceph-mon[128807]: pgmap v53: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2023-07-14T19:24:47.442 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:24:46 smithi101 ceph-mon[128807]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi101", "root=default"]}]': finished 2023-07-14T19:24:47.442 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:24:46 smithi101 ceph-mon[128807]: osdmap e12: 2 total, 1 up, 2 in 2023-07-14T19:24:47.442 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:24:46 smithi101 ceph-mon[128807]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2023-07-14T19:24:47.442 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:24:46 smithi101 ceph-mon[128807]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2023-07-14T19:24:47.442 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:24:46 smithi101 ceph-mon[128807]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' 2023-07-14T19:24:47.442 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:24:46 smithi101 ceph-mon[128807]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' 2023-07-14T19:24:47.443 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:24:46 smithi101 ceph-mon[128807]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2023-07-14T19:24:47.443 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:24:46 smithi101 ceph-mon[128807]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' 2023-07-14T19:24:47.443 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:24:46 smithi101 ceph-mon[128807]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T19:24:47.443 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:24:46 smithi101 ceph-mon[128807]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-14T19:24:47.444 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:24:46 smithi101 ceph-mon[128807]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' 2023-07-14T19:24:48.220 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:24:47 smithi139 ceph-mon[129293]: purged_snaps scrub starts 2023-07-14T19:24:48.221 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:24:47 smithi139 ceph-mon[129293]: purged_snaps scrub ok 2023-07-14T19:24:48.221 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:24:47 smithi139 ceph-mon[129293]: Detected new or changed devices on smithi101 2023-07-14T19:24:48.221 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:24:47 smithi139 ceph-mon[129293]: Adjusting osd_memory_target on smithi101 to 17168M 2023-07-14T19:24:48.221 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:24:47 smithi139 ceph-mon[129293]: osd.1 [v2:172.21.15.101:6800/2450584584,v1:172.21.15.101:6801/2450584584] boot 2023-07-14T19:24:48.221 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:24:47 smithi139 ceph-mon[129293]: osdmap e13: 2 total, 2 up, 2 in 2023-07-14T19:24:48.221 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:24:47 smithi139 ceph-mon[129293]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2023-07-14T19:24:48.222 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:24:47 smithi139 ceph-mon[129293]: pgmap v56: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2023-07-14T19:24:48.281 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:24:47 smithi098 ceph-mon[127486]: purged_snaps scrub starts 2023-07-14T19:24:48.282 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:24:47 smithi098 ceph-mon[127486]: purged_snaps scrub ok 2023-07-14T19:24:48.282 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:24:47 smithi098 ceph-mon[127486]: Detected new or changed devices on smithi101 2023-07-14T19:24:48.282 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:24:47 smithi098 ceph-mon[127486]: Adjusting osd_memory_target on smithi101 to 17168M 2023-07-14T19:24:48.283 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:24:47 smithi098 ceph-mon[127486]: osd.1 [v2:172.21.15.101:6800/2450584584,v1:172.21.15.101:6801/2450584584] boot 2023-07-14T19:24:48.283 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:24:47 smithi098 ceph-mon[127486]: osdmap e13: 2 total, 2 up, 2 in 2023-07-14T19:24:48.283 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:24:47 smithi098 ceph-mon[127486]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2023-07-14T19:24:48.284 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:24:47 smithi098 ceph-mon[127486]: pgmap v56: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2023-07-14T19:24:48.441 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:24:47 smithi101 ceph-mon[128807]: purged_snaps scrub starts 2023-07-14T19:24:48.442 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:24:47 smithi101 ceph-mon[128807]: purged_snaps scrub ok 2023-07-14T19:24:48.442 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:24:47 smithi101 ceph-mon[128807]: Detected new or changed devices on smithi101 2023-07-14T19:24:48.442 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:24:47 smithi101 ceph-mon[128807]: Adjusting osd_memory_target on smithi101 to 17168M 2023-07-14T19:24:48.443 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:24:47 smithi101 ceph-mon[128807]: osd.1 [v2:172.21.15.101:6800/2450584584,v1:172.21.15.101:6801/2450584584] boot 2023-07-14T19:24:48.443 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:24:47 smithi101 ceph-mon[128807]: osdmap e13: 2 total, 2 up, 2 in 2023-07-14T19:24:48.443 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:24:47 smithi101 ceph-mon[128807]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2023-07-14T19:24:48.443 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:24:47 smithi101 ceph-mon[128807]: pgmap v56: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2023-07-14T19:24:48.459 INFO:teuthology.orchestra.run.smithi139.stderr:2023-07-14T19:24:48.458+0000 7f0fb9db6700 1 -- 172.21.15.139:0/165376752 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f0fb40ffdb0 msgr2=0x7f0fb4100200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:24:48.459 INFO:teuthology.orchestra.run.smithi139.stderr:2023-07-14T19:24:48.458+0000 7f0fb9db6700 1 --2- 172.21.15.139:0/165376752 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f0fb40ffdb0 0x7f0fb4100200 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f0fa4009990 tx=0x7f0fa4009ca0 comp rx=0 tx=0).stop 2023-07-14T19:24:48.459 INFO:teuthology.orchestra.run.smithi139.stderr:2023-07-14T19:24:48.458+0000 7f0fb9db6700 1 -- 172.21.15.139:0/165376752 shutdown_connections 2023-07-14T19:24:48.459 INFO:teuthology.orchestra.run.smithi139.stderr:2023-07-14T19:24:48.458+0000 7f0fb9db6700 1 --2- 172.21.15.139:0/165376752 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f0fb4100740 0x7f0fb4107000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:24:48.459 INFO:teuthology.orchestra.run.smithi139.stderr:2023-07-14T19:24:48.458+0000 7f0fb9db6700 1 --2- 172.21.15.139:0/165376752 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0fb40febb0 0x7f0fb40fefc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:24:48.460 INFO:teuthology.orchestra.run.smithi139.stderr:2023-07-14T19:24:48.458+0000 7f0fb9db6700 1 --2- 172.21.15.139:0/165376752 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f0fb40ffdb0 0x7f0fb4100200 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:24:48.460 INFO:teuthology.orchestra.run.smithi139.stderr:2023-07-14T19:24:48.458+0000 7f0fb9db6700 1 -- 172.21.15.139:0/165376752 >> 172.21.15.139:0/165376752 conn(0x7f0fb40fa1e0 msgr2=0x7f0fb40fc630 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:24:48.460 INFO:teuthology.orchestra.run.smithi139.stderr:2023-07-14T19:24:48.458+0000 7f0fb9db6700 1 -- 172.21.15.139:0/165376752 shutdown_connections 2023-07-14T19:24:48.460 INFO:teuthology.orchestra.run.smithi139.stderr:2023-07-14T19:24:48.458+0000 7f0fb9db6700 1 -- 172.21.15.139:0/165376752 wait complete. 2023-07-14T19:24:48.464 INFO:teuthology.orchestra.run.smithi139.stderr:2023-07-14T19:24:48.459+0000 7f0fb9db6700 1 Processor -- start 2023-07-14T19:24:48.464 INFO:teuthology.orchestra.run.smithi139.stderr:2023-07-14T19:24:48.459+0000 7f0fb9db6700 1 -- start start 2023-07-14T19:24:48.465 INFO:teuthology.orchestra.run.smithi139.stderr:2023-07-14T19:24:48.459+0000 7f0fb9db6700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0fb40febb0 0x7f0fb4194860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:24:48.465 INFO:teuthology.orchestra.run.smithi139.stderr:2023-07-14T19:24:48.459+0000 7f0fb9db6700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f0fb40ffdb0 0x7f0fb4194da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:24:48.465 INFO:teuthology.orchestra.run.smithi139.stderr:2023-07-14T19:24:48.459+0000 7f0fb9db6700 1 --2- >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f0fb4100740 0x7f0fb419be20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:24:48.466 INFO:teuthology.orchestra.run.smithi139.stderr:2023-07-14T19:24:48.459+0000 7f0fb9db6700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0fb419c3f0 con 0x7f0fb40febb0 2023-07-14T19:24:48.466 INFO:teuthology.orchestra.run.smithi139.stderr:2023-07-14T19:24:48.459+0000 7f0fb9db6700 1 -- --> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0fb419c530 con 0x7f0fb4100740 2023-07-14T19:24:48.466 INFO:teuthology.orchestra.run.smithi139.stderr:2023-07-14T19:24:48.459+0000 7f0fb9db6700 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0fb419c6c0 con 0x7f0fb40ffdb0 2023-07-14T19:24:48.466 INFO:teuthology.orchestra.run.smithi139.stderr:2023-07-14T19:24:48.459+0000 7f0fb2ffd700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f0fb40ffdb0 0x7f0fb4194da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:24:48.467 INFO:teuthology.orchestra.run.smithi139.stderr:2023-07-14T19:24:48.459+0000 7f0fb2ffd700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f0fb40ffdb0 0x7f0fb4194da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.139:3300/0 says I am v2:172.21.15.139:37090/0 (socket says 172.21.15.139:37090) 2023-07-14T19:24:48.467 INFO:teuthology.orchestra.run.smithi139.stderr:2023-07-14T19:24:48.459+0000 7f0fb2ffd700 1 -- 172.21.15.139:0/2578057343 learned_addr learned my addr 172.21.15.139:0/2578057343 (peer_addr_for_me v2:172.21.15.139:0/0) 2023-07-14T19:24:48.467 INFO:teuthology.orchestra.run.smithi139.stderr:2023-07-14T19:24:48.459+0000 7f0fb37fe700 1 --2- 172.21.15.139:0/2578057343 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0fb40febb0 0x7f0fb4194860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:24:48.468 INFO:teuthology.orchestra.run.smithi139.stderr:2023-07-14T19:24:48.459+0000 7f0fb3fff700 1 --2- 172.21.15.139:0/2578057343 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f0fb4100740 0x7f0fb419be20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:24:48.468 INFO:teuthology.orchestra.run.smithi139.stderr:2023-07-14T19:24:48.459+0000 7f0fb2ffd700 1 -- 172.21.15.139:0/2578057343 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f0fb4100740 msgr2=0x7f0fb419be20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:24:48.468 INFO:teuthology.orchestra.run.smithi139.stderr:2023-07-14T19:24:48.459+0000 7f0fb2ffd700 1 --2- 172.21.15.139:0/2578057343 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f0fb4100740 0x7f0fb419be20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:24:48.469 INFO:teuthology.orchestra.run.smithi139.stderr:2023-07-14T19:24:48.459+0000 7f0fb2ffd700 1 -- 172.21.15.139:0/2578057343 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0fb40febb0 msgr2=0x7f0fb4194860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:24:48.469 INFO:teuthology.orchestra.run.smithi139.stderr:2023-07-14T19:24:48.459+0000 7f0fb2ffd700 1 --2- 172.21.15.139:0/2578057343 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0fb40febb0 0x7f0fb4194860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:24:48.470 INFO:teuthology.orchestra.run.smithi139.stderr:2023-07-14T19:24:48.459+0000 7f0fb2ffd700 1 -- 172.21.15.139:0/2578057343 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0fa40095f0 con 0x7f0fb40ffdb0 2023-07-14T19:24:48.470 INFO:teuthology.orchestra.run.smithi139.stderr:2023-07-14T19:24:48.459+0000 7f0fb37fe700 1 --2- 172.21.15.139:0/2578057343 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0fb40febb0 0x7f0fb4194860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-07-14T19:24:48.470 INFO:teuthology.orchestra.run.smithi139.stderr:2023-07-14T19:24:48.459+0000 7f0fb3fff700 1 --2- 172.21.15.139:0/2578057343 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f0fb4100740 0x7f0fb419be20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-07-14T19:24:48.470 INFO:teuthology.orchestra.run.smithi139.stderr:2023-07-14T19:24:48.460+0000 7f0fb2ffd700 1 --2- 172.21.15.139:0/2578057343 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f0fb40ffdb0 0x7f0fb4194da0 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f0fa4000c00 tx=0x7f0fa400f740 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:24:48.471 INFO:teuthology.orchestra.run.smithi139.stderr:2023-07-14T19:24:48.460+0000 7f0fb0ff9700 1 -- 172.21.15.139:0/2578057343 <== mon.1 v2:172.21.15.139:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0fa401d070 con 0x7f0fb40ffdb0 2023-07-14T19:24:48.471 INFO:teuthology.orchestra.run.smithi139.stderr:2023-07-14T19:24:48.460+0000 7f0fb0ff9700 1 -- 172.21.15.139:0/2578057343 <== mon.1 v2:172.21.15.139:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f0fa400fba0 con 0x7f0fb40ffdb0 2023-07-14T19:24:48.471 INFO:teuthology.orchestra.run.smithi139.stderr:2023-07-14T19:24:48.460+0000 7f0fb0ff9700 1 -- 172.21.15.139:0/2578057343 <== mon.1 v2:172.21.15.139:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0fa4017650 con 0x7f0fb40ffdb0 2023-07-14T19:24:48.472 INFO:teuthology.orchestra.run.smithi139.stderr:2023-07-14T19:24:48.460+0000 7f0fb9db6700 1 -- 172.21.15.139:0/2578057343 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0fb419c940 con 0x7f0fb40ffdb0 2023-07-14T19:24:48.472 INFO:teuthology.orchestra.run.smithi139.stderr:2023-07-14T19:24:48.460+0000 7f0fb9db6700 1 -- 172.21.15.139:0/2578057343 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0fb419cd50 con 0x7f0fb40ffdb0 2023-07-14T19:24:48.472 INFO:teuthology.orchestra.run.smithi139.stderr:2023-07-14T19:24:48.461+0000 7f0fb9db6700 1 -- 172.21.15.139:0/2578057343 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0fb4106160 con 0x7f0fb40ffdb0 2023-07-14T19:24:48.472 INFO:teuthology.orchestra.run.smithi139.stderr:2023-07-14T19:24:48.462+0000 7f0fb0ff9700 1 -- 172.21.15.139:0/2578057343 <== mon.1 v2:172.21.15.139:3300/0 4 ==== mgrmap(e 15) v1 ==== 135659+0+0 (secure 0 0 0) 0x7f0fa4022470 con 0x7f0fb40ffdb0 2023-07-14T19:24:48.473 INFO:teuthology.orchestra.run.smithi139.stderr:2023-07-14T19:24:48.462+0000 7f0fb0ff9700 1 --2- 172.21.15.139:0/2578057343 >> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] conn(0x7f0f9c0a11c0 0x7f0f9c0a3670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:24:48.473 INFO:teuthology.orchestra.run.smithi139.stderr:2023-07-14T19:24:48.463+0000 7f0fb0ff9700 1 -- 172.21.15.139:0/2578057343 <== mon.1 v2:172.21.15.139:3300/0 5 ==== osd_map(14..14 src has 1..14) v4 ==== 2309+0+0 (secure 0 0 0) 0x7f0fa40cc5a0 con 0x7f0fb40ffdb0 2023-07-14T19:24:48.473 INFO:teuthology.orchestra.run.smithi139.stderr:2023-07-14T19:24:48.463+0000 7f0fb37fe700 1 --2- 172.21.15.139:0/2578057343 >> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] conn(0x7f0f9c0a11c0 0x7f0f9c0a3670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:24:48.474 INFO:teuthology.orchestra.run.smithi139.stderr:2023-07-14T19:24:48.463+0000 7f0fb37fe700 1 --2- 172.21.15.139:0/2578057343 >> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] conn(0x7f0f9c0a11c0 0x7f0f9c0a3670 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f0fa8005950 tx=0x7f0fa80058e0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:24:48.474 INFO:teuthology.orchestra.run.smithi139.stderr:2023-07-14T19:24:48.467+0000 7f0fb0ff9700 1 -- 172.21.15.139:0/2578057343 <== mon.1 v2:172.21.15.139:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f0fa4097650 con 0x7f0fb40ffdb0 2023-07-14T19:24:48.661 INFO:teuthology.orchestra.run.smithi139.stderr:2023-07-14T19:24:48.661+0000 7f0fb9db6700 1 -- 172.21.15.139:0/2578057343 --> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi139:vg_nvme/lv_4", "target": ["mon-mgr", ""]}) v1 -- 0x7f0fb4060a80 con 0x7f0f9c0a11c0 2023-07-14T19:24:49.281 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:24:48 smithi098 ceph-mon[127486]: osdmap e14: 2 total, 2 up, 2 in 2023-07-14T19:24:49.282 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:24:48 smithi098 ceph-mon[127486]: from='client.24157 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi139:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2023-07-14T19:24:49.282 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:24:48 smithi098 ceph-mon[127486]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-07-14T19:24:49.282 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:24:48 smithi098 ceph-mon[127486]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-07-14T19:24:49.282 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:24:48 smithi098 ceph-mon[127486]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T19:24:49.441 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:24:48 smithi101 ceph-mon[128807]: osdmap e14: 2 total, 2 up, 2 in 2023-07-14T19:24:49.474 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:24:48 smithi101 ceph-mon[128807]: from='client.24157 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi139:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2023-07-14T19:24:49.475 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:24:48 smithi101 ceph-mon[128807]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-07-14T19:24:49.475 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:24:48 smithi101 ceph-mon[128807]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-07-14T19:24:49.475 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:24:48 smithi101 ceph-mon[128807]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T19:24:49.476 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:24:48 smithi139 ceph-mon[129293]: osdmap e14: 2 total, 2 up, 2 in 2023-07-14T19:24:49.476 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:24:48 smithi139 ceph-mon[129293]: from='client.24157 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi139:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2023-07-14T19:24:49.476 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:24:48 smithi139 ceph-mon[129293]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-07-14T19:24:49.476 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:24:48 smithi139 ceph-mon[129293]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-07-14T19:24:49.476 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:24:48 smithi139 ceph-mon[129293]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T19:24:50.220 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:24:49 smithi139 ceph-mon[129293]: pgmap v58: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2023-07-14T19:24:50.281 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:24:49 smithi098 ceph-mon[127486]: pgmap v58: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2023-07-14T19:24:50.441 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:24:49 smithi101 ceph-mon[128807]: pgmap v58: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2023-07-14T19:24:51.442 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:24:51 smithi101 ceph-mon[128807]: from='client.? 172.21.15.139:0/744790170' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "48e6d499-3ca9-424b-b0fb-3ba5f33b9fc9"}]: dispatch 2023-07-14T19:24:51.442 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:24:51 smithi101 ceph-mon[128807]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "48e6d499-3ca9-424b-b0fb-3ba5f33b9fc9"}]: dispatch 2023-07-14T19:24:51.442 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:24:51 smithi101 ceph-mon[128807]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "48e6d499-3ca9-424b-b0fb-3ba5f33b9fc9"}]': finished 2023-07-14T19:24:51.442 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:24:51 smithi101 ceph-mon[128807]: osdmap e15: 3 total, 2 up, 3 in 2023-07-14T19:24:51.442 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:24:51 smithi101 ceph-mon[128807]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2023-07-14T19:24:51.470 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:24:51 smithi139 ceph-mon[129293]: from='client.? 172.21.15.139:0/744790170' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "48e6d499-3ca9-424b-b0fb-3ba5f33b9fc9"}]: dispatch 2023-07-14T19:24:51.471 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:24:51 smithi139 ceph-mon[129293]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "48e6d499-3ca9-424b-b0fb-3ba5f33b9fc9"}]: dispatch 2023-07-14T19:24:51.471 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:24:51 smithi139 ceph-mon[129293]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "48e6d499-3ca9-424b-b0fb-3ba5f33b9fc9"}]': finished 2023-07-14T19:24:51.471 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:24:51 smithi139 ceph-mon[129293]: osdmap e15: 3 total, 2 up, 3 in 2023-07-14T19:24:51.471 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:24:51 smithi139 ceph-mon[129293]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2023-07-14T19:24:51.531 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:24:51 smithi098 ceph-mon[127486]: from='client.? 172.21.15.139:0/744790170' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "48e6d499-3ca9-424b-b0fb-3ba5f33b9fc9"}]: dispatch 2023-07-14T19:24:51.532 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:24:51 smithi098 ceph-mon[127486]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "48e6d499-3ca9-424b-b0fb-3ba5f33b9fc9"}]: dispatch 2023-07-14T19:24:51.532 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:24:51 smithi098 ceph-mon[127486]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "48e6d499-3ca9-424b-b0fb-3ba5f33b9fc9"}]': finished 2023-07-14T19:24:51.532 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:24:51 smithi098 ceph-mon[127486]: osdmap e15: 3 total, 2 up, 3 in 2023-07-14T19:24:51.533 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:24:51 smithi098 ceph-mon[127486]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2023-07-14T19:24:52.531 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:24:52 smithi098 ceph-mon[127486]: pgmap v60: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2023-07-14T19:24:52.531 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:24:52 smithi098 ceph-mon[127486]: from='client.? 172.21.15.139:0/1469182671' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-07-14T19:24:52.694 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:24:52 smithi101 ceph-mon[128807]: pgmap v60: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2023-07-14T19:24:52.695 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:24:52 smithi101 ceph-mon[128807]: from='client.? 172.21.15.139:0/1469182671' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-07-14T19:24:52.720 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:24:52 smithi139 ceph-mon[129293]: pgmap v60: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2023-07-14T19:24:52.721 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:24:52 smithi139 ceph-mon[129293]: from='client.? 172.21.15.139:0/1469182671' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-07-14T19:24:54.927 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:24:54 smithi139 ceph-mon[129293]: pgmap v61: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2023-07-14T19:24:54.941 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:24:54 smithi101 ceph-mon[128807]: pgmap v61: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2023-07-14T19:24:55.031 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:24:54 smithi098 ceph-mon[127486]: pgmap v61: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2023-07-14T19:24:56.648 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:24:56 smithi139 ceph-mon[129293]: pgmap v62: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2023-07-14T19:24:56.941 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:24:56 smithi101 ceph-mon[128807]: pgmap v62: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2023-07-14T19:24:57.031 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:24:56 smithi098 ceph-mon[127486]: pgmap v62: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2023-07-14T19:24:58.941 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:24:58 smithi101 ceph-mon[128807]: pgmap v63: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2023-07-14T19:24:58.942 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:24:58 smithi101 ceph-mon[128807]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2023-07-14T19:24:58.942 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:24:58 smithi101 ceph-mon[128807]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T19:24:58.942 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:24:58 smithi101 ceph-mon[128807]: Deploying daemon osd.2 on smithi139 2023-07-14T19:24:58.971 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:24:58 smithi139 ceph-mon[129293]: pgmap v63: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2023-07-14T19:24:58.972 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:24:58 smithi139 ceph-mon[129293]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2023-07-14T19:24:58.972 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:24:58 smithi139 ceph-mon[129293]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T19:24:58.972 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:24:58 smithi139 ceph-mon[129293]: Deploying daemon osd.2 on smithi139 2023-07-14T19:24:59.032 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:24:58 smithi098 ceph-mon[127486]: pgmap v63: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2023-07-14T19:24:59.032 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:24:58 smithi098 ceph-mon[127486]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2023-07-14T19:24:59.032 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:24:58 smithi098 ceph-mon[127486]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T19:24:59.032 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:24:58 smithi098 ceph-mon[127486]: Deploying daemon osd.2 on smithi139 2023-07-14T19:25:00.941 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:25:00 smithi101 ceph-mon[128807]: pgmap v64: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2023-07-14T19:25:00.971 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:25:00 smithi139 ceph-mon[129293]: pgmap v64: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2023-07-14T19:25:01.031 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:25:00 smithi098 ceph-mon[127486]: pgmap v64: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2023-07-14T19:25:03.019 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:25:02 smithi139 ceph-mon[129293]: pgmap v65: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2023-07-14T19:25:03.020 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:25:02 smithi139 ceph-mon[129293]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' 2023-07-14T19:25:03.020 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:25:02 smithi139 ceph-mon[129293]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' 2023-07-14T19:25:03.020 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:25:02 smithi139 ceph-mon[129293]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-14T19:25:03.192 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:25:02 smithi101 ceph-mon[128807]: pgmap v65: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2023-07-14T19:25:03.192 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:25:02 smithi101 ceph-mon[128807]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' 2023-07-14T19:25:03.192 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:25:02 smithi101 ceph-mon[128807]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' 2023-07-14T19:25:03.192 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:25:02 smithi101 ceph-mon[128807]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-14T19:25:03.281 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:25:02 smithi098 ceph-mon[127486]: pgmap v65: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2023-07-14T19:25:03.281 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:25:02 smithi098 ceph-mon[127486]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' 2023-07-14T19:25:03.281 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:25:02 smithi098 ceph-mon[127486]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' 2023-07-14T19:25:03.282 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:25:02 smithi098 ceph-mon[127486]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-14T19:25:04.518 INFO:teuthology.orchestra.run.smithi139.stderr:2023-07-14T19:25:04.517+0000 7f0fb0ff9700 1 -- 172.21.15.139:0/2578057343 <== mgr.14152 v2:172.21.15.98:6800/452042164 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7f0fb4060a80 con 0x7f0f9c0a11c0 2023-07-14T19:25:04.519 INFO:teuthology.orchestra.run.smithi139.stdout:Created osd(s) 2 on host 'smithi139' 2023-07-14T19:25:04.521 INFO:teuthology.orchestra.run.smithi139.stderr:2023-07-14T19:25:04.519+0000 7f0fb9db6700 1 -- 172.21.15.139:0/2578057343 >> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] conn(0x7f0f9c0a11c0 msgr2=0x7f0f9c0a3670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:25:04.521 INFO:teuthology.orchestra.run.smithi139.stderr:2023-07-14T19:25:04.519+0000 7f0fb9db6700 1 --2- 172.21.15.139:0/2578057343 >> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] conn(0x7f0f9c0a11c0 0x7f0f9c0a3670 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f0fa8005950 tx=0x7f0fa80058e0 comp rx=0 tx=0).stop 2023-07-14T19:25:04.521 INFO:teuthology.orchestra.run.smithi139.stderr:2023-07-14T19:25:04.519+0000 7f0fb9db6700 1 -- 172.21.15.139:0/2578057343 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f0fb40ffdb0 msgr2=0x7f0fb4194da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:25:04.521 INFO:teuthology.orchestra.run.smithi139.stderr:2023-07-14T19:25:04.519+0000 7f0fb9db6700 1 --2- 172.21.15.139:0/2578057343 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f0fb40ffdb0 0x7f0fb4194da0 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f0fa4000c00 tx=0x7f0fa400f740 comp rx=0 tx=0).stop 2023-07-14T19:25:04.521 INFO:teuthology.orchestra.run.smithi139.stderr:2023-07-14T19:25:04.519+0000 7f0fb9db6700 1 -- 172.21.15.139:0/2578057343 shutdown_connections 2023-07-14T19:25:04.522 INFO:teuthology.orchestra.run.smithi139.stderr:2023-07-14T19:25:04.519+0000 7f0fb9db6700 1 --2- 172.21.15.139:0/2578057343 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f0fb4100740 0x7f0fb419be20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:25:04.522 INFO:teuthology.orchestra.run.smithi139.stderr:2023-07-14T19:25:04.519+0000 7f0fb9db6700 1 --2- 172.21.15.139:0/2578057343 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0fb40febb0 0x7f0fb4194860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:25:04.522 INFO:teuthology.orchestra.run.smithi139.stderr:2023-07-14T19:25:04.519+0000 7f0fb9db6700 1 --2- 172.21.15.139:0/2578057343 >> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] conn(0x7f0f9c0a11c0 0x7f0f9c0a3670 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:25:04.522 INFO:teuthology.orchestra.run.smithi139.stderr:2023-07-14T19:25:04.519+0000 7f0fb9db6700 1 --2- 172.21.15.139:0/2578057343 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f0fb40ffdb0 0x7f0fb4194da0 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:25:04.522 INFO:teuthology.orchestra.run.smithi139.stderr:2023-07-14T19:25:04.519+0000 7f0fb9db6700 1 -- 172.21.15.139:0/2578057343 >> 172.21.15.139:0/2578057343 conn(0x7f0fb40fa1e0 msgr2=0x7f0fb41032a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:25:04.522 INFO:teuthology.orchestra.run.smithi139.stderr:2023-07-14T19:25:04.519+0000 7f0fb9db6700 1 -- 172.21.15.139:0/2578057343 shutdown_connections 2023-07-14T19:25:04.522 INFO:teuthology.orchestra.run.smithi139.stderr:2023-07-14T19:25:04.519+0000 7f0fb9db6700 1 -- 172.21.15.139:0/2578057343 wait complete. 2023-07-14T19:25:04.979 DEBUG:teuthology.orchestra.run.smithi139:osd.2> sudo journalctl -f -n 0 -u ceph-9adea648-227b-11ee-9b33-001a4aab830c@osd.2.service 2023-07-14T19:25:04.981 INFO:tasks.cephadm:Deploying osd.3 on smithi098 with /dev/vg_nvme/lv_3... 2023-07-14T19:25:04.982 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81c9387d2e7576da065747ec9c16d1c7d69016ec shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 9adea648-227b-11ee-9b33-001a4aab830c -- ceph-volume lvm zap /dev/vg_nvme/lv_3 2023-07-14T19:25:05.011 INFO:journalctl@ceph.osd.2.smithi139.stdout:-- Logs begin at Fri 2023-07-14 18:56:10 UTC. -- 2023-07-14T19:25:05.191 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:25:04 smithi101 ceph-mon[128807]: pgmap v66: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2023-07-14T19:25:05.192 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:25:04 smithi101 ceph-mon[128807]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' 2023-07-14T19:25:05.192 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:25:04 smithi101 ceph-mon[128807]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' 2023-07-14T19:25:05.192 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:25:04 smithi101 ceph-mon[128807]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T19:25:05.192 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:25:04 smithi101 ceph-mon[128807]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-14T19:25:05.192 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:25:04 smithi101 ceph-mon[128807]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' 2023-07-14T19:25:05.193 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:25:04 smithi101 ceph-mon[128807]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' 2023-07-14T19:25:05.193 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:25:04 smithi101 ceph-mon[128807]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' 2023-07-14T19:25:05.193 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:25:04 smithi101 ceph-mon[128807]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-14T19:25:05.220 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:25:04 smithi139 ceph-mon[129293]: pgmap v66: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2023-07-14T19:25:05.221 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:25:04 smithi139 ceph-mon[129293]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' 2023-07-14T19:25:05.221 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:25:04 smithi139 ceph-mon[129293]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' 2023-07-14T19:25:05.221 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:25:04 smithi139 ceph-mon[129293]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T19:25:05.221 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:25:04 smithi139 ceph-mon[129293]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-14T19:25:05.222 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:25:04 smithi139 ceph-mon[129293]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' 2023-07-14T19:25:05.222 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:25:04 smithi139 ceph-mon[129293]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' 2023-07-14T19:25:05.222 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:25:04 smithi139 ceph-mon[129293]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' 2023-07-14T19:25:05.222 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:25:04 smithi139 ceph-mon[129293]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-14T19:25:05.281 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:25:04 smithi098 ceph-mon[127486]: pgmap v66: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2023-07-14T19:25:05.281 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:25:04 smithi098 ceph-mon[127486]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' 2023-07-14T19:25:05.282 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:25:04 smithi098 ceph-mon[127486]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' 2023-07-14T19:25:05.282 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:25:04 smithi098 ceph-mon[127486]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T19:25:05.282 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:25:04 smithi098 ceph-mon[127486]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-14T19:25:05.282 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:25:04 smithi098 ceph-mon[127486]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' 2023-07-14T19:25:05.282 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:25:04 smithi098 ceph-mon[127486]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' 2023-07-14T19:25:05.282 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:25:04 smithi098 ceph-mon[127486]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' 2023-07-14T19:25:05.282 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:25:04 smithi098 ceph-mon[127486]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-14T19:25:05.546 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/9adea648-227b-11ee-9b33-001a4aab830c/mon.a/config 2023-07-14T19:25:05.970 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:25:05 smithi139 ceph-mon[129293]: from='osd.2 [v2:172.21.15.139:6800/945038995,v1:172.21.15.139:6801/945038995]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2023-07-14T19:25:05.971 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:25:05 smithi139 ceph-mon[129293]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2023-07-14T19:25:05.971 INFO:journalctl@ceph.osd.2.smithi139.stdout:Jul 14 19:25:05 smithi139 ceph-9adea648-227b-11ee-9b33-001a4aab830c-osd-2[135088]: 2023-07-14T19:25:05.883+0000 7fb2ed31d640 -1 osd.2 0 log_to_monitors true 2023-07-14T19:25:06.031 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:25:05 smithi098 ceph-mon[127486]: from='osd.2 [v2:172.21.15.139:6800/945038995,v1:172.21.15.139:6801/945038995]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2023-07-14T19:25:06.032 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:25:05 smithi098 ceph-mon[127486]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2023-07-14T19:25:06.191 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:25:05 smithi101 ceph-mon[128807]: from='osd.2 [v2:172.21.15.139:6800/945038995,v1:172.21.15.139:6801/945038995]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2023-07-14T19:25:06.192 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:25:05 smithi101 ceph-mon[128807]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2023-07-14T19:25:07.127 INFO:teuthology.orchestra.run.smithi098.stderr:--> Zapping: /dev/vg_nvme/lv_3 2023-07-14T19:25:07.127 INFO:teuthology.orchestra.run.smithi098.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 conv=fsync 2023-07-14T19:25:07.127 INFO:teuthology.orchestra.run.smithi098.stderr: stderr: 10+0 records in 2023-07-14T19:25:07.128 INFO:teuthology.orchestra.run.smithi098.stderr:10+0 records out 2023-07-14T19:25:07.128 INFO:teuthology.orchestra.run.smithi098.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0109967 s, 954 MB/s 2023-07-14T19:25:07.128 INFO:teuthology.orchestra.run.smithi098.stderr:--> Zapping successful for: 2023-07-14T19:25:07.185 INFO:journalctl@ceph.osd.2.smithi139.stdout:Jul 14 19:25:06 smithi139 ceph-9adea648-227b-11ee-9b33-001a4aab830c-osd-2[135088]: 2023-07-14T19:25:06.944+0000 7fb2e2194700 -1 osd.2 0 waiting for initial osdmap 2023-07-14T19:25:07.185 INFO:journalctl@ceph.osd.2.smithi139.stdout:Jul 14 19:25:06 smithi139 ceph-9adea648-227b-11ee-9b33-001a4aab830c-osd-2[135088]: 2023-07-14T19:25:06.950+0000 7fb2ddf8a700 -1 osd.2 17 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-07-14T19:25:07.186 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:25:06 smithi139 ceph-mon[129293]: pgmap v67: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2023-07-14T19:25:07.186 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:25:06 smithi139 ceph-mon[129293]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2023-07-14T19:25:07.186 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:25:06 smithi139 ceph-mon[129293]: osdmap e16: 3 total, 2 up, 3 in 2023-07-14T19:25:07.186 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:25:06 smithi139 ceph-mon[129293]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2023-07-14T19:25:07.186 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:25:06 smithi139 ceph-mon[129293]: from='osd.2 [v2:172.21.15.139:6800/945038995,v1:172.21.15.139:6801/945038995]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi139", "root=default"]}]: dispatch 2023-07-14T19:25:07.186 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:25:06 smithi139 ceph-mon[129293]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi139", "root=default"]}]: dispatch 2023-07-14T19:25:07.192 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:25:06 smithi101 ceph-mon[128807]: pgmap v67: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2023-07-14T19:25:07.192 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:25:06 smithi101 ceph-mon[128807]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2023-07-14T19:25:07.192 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:25:06 smithi101 ceph-mon[128807]: osdmap e16: 3 total, 2 up, 3 in 2023-07-14T19:25:07.193 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:25:06 smithi101 ceph-mon[128807]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2023-07-14T19:25:07.193 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:25:06 smithi101 ceph-mon[128807]: from='osd.2 [v2:172.21.15.139:6800/945038995,v1:172.21.15.139:6801/945038995]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi139", "root=default"]}]: dispatch 2023-07-14T19:25:07.193 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:25:06 smithi101 ceph-mon[128807]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi139", "root=default"]}]: dispatch 2023-07-14T19:25:07.269 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:25:06 smithi098 ceph-mon[127486]: pgmap v67: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2023-07-14T19:25:07.269 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:25:06 smithi098 ceph-mon[127486]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2023-07-14T19:25:07.269 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:25:06 smithi098 ceph-mon[127486]: osdmap e16: 3 total, 2 up, 3 in 2023-07-14T19:25:07.270 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:25:06 smithi098 ceph-mon[127486]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2023-07-14T19:25:07.270 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:25:06 smithi098 ceph-mon[127486]: from='osd.2 [v2:172.21.15.139:6800/945038995,v1:172.21.15.139:6801/945038995]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi139", "root=default"]}]: dispatch 2023-07-14T19:25:07.270 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:25:06 smithi098 ceph-mon[127486]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi139", "root=default"]}]: dispatch 2023-07-14T19:25:07.498 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81c9387d2e7576da065747ec9c16d1c7d69016ec shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 9adea648-227b-11ee-9b33-001a4aab830c -- ceph orch daemon add osd smithi098:vg_nvme/lv_3 2023-07-14T19:25:08.053 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/9adea648-227b-11ee-9b33-001a4aab830c/mon.a/config 2023-07-14T19:25:08.220 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:25:07 smithi139 ceph-mon[129293]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi139", "root=default"]}]': finished 2023-07-14T19:25:08.221 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:25:07 smithi139 ceph-mon[129293]: osdmap e17: 3 total, 2 up, 3 in 2023-07-14T19:25:08.221 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:25:07 smithi139 ceph-mon[129293]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2023-07-14T19:25:08.221 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:25:07 smithi139 ceph-mon[129293]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2023-07-14T19:25:08.221 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:25:07 smithi139 ceph-mon[129293]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' 2023-07-14T19:25:08.221 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:25:07 smithi139 ceph-mon[129293]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' 2023-07-14T19:25:08.221 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:25:07 smithi139 ceph-mon[129293]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2023-07-14T19:25:08.222 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:25:07 smithi139 ceph-mon[129293]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' 2023-07-14T19:25:08.222 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:25:07 smithi139 ceph-mon[129293]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T19:25:08.222 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:25:07 smithi139 ceph-mon[129293]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-14T19:25:08.222 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:25:07 smithi139 ceph-mon[129293]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' 2023-07-14T19:25:08.268 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:25:07 smithi098 ceph-mon[127486]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi139", "root=default"]}]': finished 2023-07-14T19:25:08.268 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:25:07 smithi098 ceph-mon[127486]: osdmap e17: 3 total, 2 up, 3 in 2023-07-14T19:25:08.268 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:25:07 smithi098 ceph-mon[127486]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2023-07-14T19:25:08.268 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:25:07 smithi098 ceph-mon[127486]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2023-07-14T19:25:08.269 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:25:07 smithi098 ceph-mon[127486]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' 2023-07-14T19:25:08.269 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:25:07 smithi098 ceph-mon[127486]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' 2023-07-14T19:25:08.269 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:25:07 smithi098 ceph-mon[127486]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2023-07-14T19:25:08.269 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:25:07 smithi098 ceph-mon[127486]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' 2023-07-14T19:25:08.269 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:25:07 smithi098 ceph-mon[127486]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T19:25:08.269 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:25:07 smithi098 ceph-mon[127486]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-14T19:25:08.269 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:25:07 smithi098 ceph-mon[127486]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' 2023-07-14T19:25:08.442 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:25:07 smithi101 ceph-mon[128807]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi139", "root=default"]}]': finished 2023-07-14T19:25:08.443 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:25:07 smithi101 ceph-mon[128807]: osdmap e17: 3 total, 2 up, 3 in 2023-07-14T19:25:08.443 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:25:07 smithi101 ceph-mon[128807]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2023-07-14T19:25:08.443 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:25:07 smithi101 ceph-mon[128807]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2023-07-14T19:25:08.443 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:25:07 smithi101 ceph-mon[128807]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' 2023-07-14T19:25:08.443 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:25:07 smithi101 ceph-mon[128807]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' 2023-07-14T19:25:08.443 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:25:07 smithi101 ceph-mon[128807]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2023-07-14T19:25:08.443 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:25:07 smithi101 ceph-mon[128807]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' 2023-07-14T19:25:08.444 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:25:07 smithi101 ceph-mon[128807]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T19:25:08.444 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:25:07 smithi101 ceph-mon[128807]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-14T19:25:08.444 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:25:07 smithi101 ceph-mon[128807]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' 2023-07-14T19:25:09.101 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:25:08 smithi098 ceph-mon[127486]: purged_snaps scrub starts 2023-07-14T19:25:09.101 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:25:08 smithi098 ceph-mon[127486]: purged_snaps scrub ok 2023-07-14T19:25:09.101 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:25:08 smithi098 ceph-mon[127486]: Detected new or changed devices on smithi139 2023-07-14T19:25:09.101 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:25:08 smithi098 ceph-mon[127486]: Adjusting osd_memory_target on smithi139 to 17168M 2023-07-14T19:25:09.101 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:25:08 smithi098 ceph-mon[127486]: pgmap v70: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2023-07-14T19:25:09.101 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:25:08 smithi098 ceph-mon[127486]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2023-07-14T19:25:09.102 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:25:08 smithi098 ceph-mon[127486]: osd.2 [v2:172.21.15.139:6800/945038995,v1:172.21.15.139:6801/945038995] boot 2023-07-14T19:25:09.102 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:25:08 smithi098 ceph-mon[127486]: osdmap e18: 3 total, 3 up, 3 in 2023-07-14T19:25:09.102 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:25:08 smithi098 ceph-mon[127486]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2023-07-14T19:25:09.220 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:25:08 smithi139 ceph-mon[129293]: purged_snaps scrub starts 2023-07-14T19:25:09.221 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:25:08 smithi139 ceph-mon[129293]: purged_snaps scrub ok 2023-07-14T19:25:09.221 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:25:08 smithi139 ceph-mon[129293]: Detected new or changed devices on smithi139 2023-07-14T19:25:09.221 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:25:08 smithi139 ceph-mon[129293]: Adjusting osd_memory_target on smithi139 to 17168M 2023-07-14T19:25:09.221 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:25:08 smithi139 ceph-mon[129293]: pgmap v70: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2023-07-14T19:25:09.221 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:25:08 smithi139 ceph-mon[129293]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2023-07-14T19:25:09.221 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:25:08 smithi139 ceph-mon[129293]: osd.2 [v2:172.21.15.139:6800/945038995,v1:172.21.15.139:6801/945038995] boot 2023-07-14T19:25:09.222 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:25:08 smithi139 ceph-mon[129293]: osdmap e18: 3 total, 3 up, 3 in 2023-07-14T19:25:09.222 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:25:08 smithi139 ceph-mon[129293]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2023-07-14T19:25:09.244 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:25:09.242+0000 7f573ba9f700 1 -- 172.21.15.98:0/41674718 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5734100740 msgr2=0x7f5734107000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:25:09.244 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:25:09.242+0000 7f573ba9f700 1 --2- 172.21.15.98:0/41674718 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5734100740 0x7f5734107000 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f572c00b0a0 tx=0x7f572c00b3b0 comp rx=0 tx=0).stop 2023-07-14T19:25:09.244 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:25:09.242+0000 7f573ba9f700 1 -- 172.21.15.98:0/41674718 shutdown_connections 2023-07-14T19:25:09.244 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:25:09.242+0000 7f573ba9f700 1 --2- 172.21.15.98:0/41674718 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f57340febb0 0x7f57340fefc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:25:09.245 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:25:09.242+0000 7f573ba9f700 1 --2- 172.21.15.98:0/41674718 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5734100740 0x7f5734107000 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:25:09.245 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:25:09.242+0000 7f573ba9f700 1 --2- 172.21.15.98:0/41674718 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f57340ffdb0 0x7f5734100200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:25:09.245 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:25:09.242+0000 7f573ba9f700 1 -- 172.21.15.98:0/41674718 >> 172.21.15.98:0/41674718 conn(0x7f57340fa1e0 msgr2=0x7f57340fc630 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:25:09.245 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:25:09.242+0000 7f573ba9f700 1 -- 172.21.15.98:0/41674718 shutdown_connections 2023-07-14T19:25:09.245 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:25:09.242+0000 7f573ba9f700 1 -- 172.21.15.98:0/41674718 wait complete. 2023-07-14T19:25:09.247 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:25:09.243+0000 7f573ba9f700 1 Processor -- start 2023-07-14T19:25:09.247 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:25:09.243+0000 7f573ba9f700 1 -- start start 2023-07-14T19:25:09.247 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:25:09.243+0000 7f573ba9f700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f57340febb0 0x7f5734102b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:25:09.247 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:25:09.243+0000 7f573ba9f700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f57340ffdb0 0x7f5734101190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:25:09.248 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:25:09.243+0000 7f573ba9f700 1 --2- >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f5734100740 0x7f57341016d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:25:09.248 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:25:09.243+0000 7f573ba9f700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5734101e50 con 0x7f57340ffdb0 2023-07-14T19:25:09.248 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:25:09.243+0000 7f573ba9f700 1 -- --> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5734101fc0 con 0x7f5734100740 2023-07-14T19:25:09.248 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:25:09.243+0000 7f573ba9f700 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f573419fb10 con 0x7f57340febb0 2023-07-14T19:25:09.248 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:25:09.243+0000 7f573903a700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f57340ffdb0 0x7f5734101190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:25:09.248 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:25:09.243+0000 7f573903a700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f57340ffdb0 0x7f5734101190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:44828/0 (socket says 172.21.15.98:44828) 2023-07-14T19:25:09.248 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:25:09.243+0000 7f573903a700 1 -- 172.21.15.98:0/3401369431 learned_addr learned my addr 172.21.15.98:0/3401369431 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-07-14T19:25:09.249 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:25:09.243+0000 7f573983b700 1 --2- 172.21.15.98:0/3401369431 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f57340febb0 0x7f5734102b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:25:09.249 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:25:09.243+0000 7f573a03c700 1 --2- 172.21.15.98:0/3401369431 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f5734100740 0x7f57341016d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:25:09.249 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:25:09.243+0000 7f573903a700 1 -- 172.21.15.98:0/3401369431 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f57340febb0 msgr2=0x7f5734102b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:25:09.249 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:25:09.243+0000 7f573903a700 1 --2- 172.21.15.98:0/3401369431 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f57340febb0 0x7f5734102b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:25:09.249 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:25:09.243+0000 7f573903a700 1 -- 172.21.15.98:0/3401369431 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f5734100740 msgr2=0x7f57341016d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:25:09.249 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:25:09.243+0000 7f573903a700 1 --2- 172.21.15.98:0/3401369431 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f5734100740 0x7f57341016d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:25:09.249 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:25:09.243+0000 7f573903a700 1 -- 172.21.15.98:0/3401369431 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f572c009d00 con 0x7f57340ffdb0 2023-07-14T19:25:09.249 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:25:09.243+0000 7f573983b700 1 --2- 172.21.15.98:0/3401369431 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f57340febb0 0x7f5734102b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-07-14T19:25:09.250 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:25:09.244+0000 7f573903a700 1 --2- 172.21.15.98:0/3401369431 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f57340ffdb0 0x7f5734101190 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f572400e9a0 tx=0x7f572400ed60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:25:09.250 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:25:09.244+0000 7f572affd700 1 -- 172.21.15.98:0/3401369431 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5724009790 con 0x7f57340ffdb0 2023-07-14T19:25:09.250 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:25:09.244+0000 7f573ba9f700 1 -- 172.21.15.98:0/3401369431 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f573419fdf0 con 0x7f57340ffdb0 2023-07-14T19:25:09.250 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:25:09.244+0000 7f572affd700 1 -- 172.21.15.98:0/3401369431 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f572400cbd0 con 0x7f57340ffdb0 2023-07-14T19:25:09.250 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:25:09.244+0000 7f572affd700 1 -- 172.21.15.98:0/3401369431 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5724018680 con 0x7f57340ffdb0 2023-07-14T19:25:09.250 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:25:09.244+0000 7f573ba9f700 1 -- 172.21.15.98:0/3401369431 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f57341a0260 con 0x7f57340ffdb0 2023-07-14T19:25:09.250 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:25:09.245+0000 7f573ba9f700 1 -- 172.21.15.98:0/3401369431 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f573410a3f0 con 0x7f57340ffdb0 2023-07-14T19:25:09.251 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:25:09.246+0000 7f572affd700 1 -- 172.21.15.98:0/3401369431 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 15) v1 ==== 135659+0+0 (secure 0 0 0) 0x7f572400cd40 con 0x7f57340ffdb0 2023-07-14T19:25:09.251 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:25:09.246+0000 7f572affd700 1 --2- 172.21.15.98:0/3401369431 >> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] conn(0x7f57200a10c0 0x7f57200a3570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:25:09.251 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:25:09.246+0000 7f573983b700 1 --2- 172.21.15.98:0/3401369431 >> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] conn(0x7f57200a10c0 0x7f57200a3570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:25:09.252 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:25:09.246+0000 7f572affd700 1 -- 172.21.15.98:0/3401369431 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(19..19 src has 1..19) v4 ==== 2851+0+0 (secure 0 0 0) 0x7f5724014070 con 0x7f57340ffdb0 2023-07-14T19:25:09.252 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:25:09.247+0000 7f573983b700 1 --2- 172.21.15.98:0/3401369431 >> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] conn(0x7f57200a10c0 0x7f57200a3570 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f57340ffc10 tx=0x7f5730009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:25:09.252 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:25:09.250+0000 7f572affd700 1 -- 172.21.15.98:0/3401369431 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f5724051780 con 0x7f57340ffdb0 2023-07-14T19:25:09.442 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:25:08 smithi101 ceph-mon[128807]: purged_snaps scrub starts 2023-07-14T19:25:09.442 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:25:08 smithi101 ceph-mon[128807]: purged_snaps scrub ok 2023-07-14T19:25:09.442 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:25:08 smithi101 ceph-mon[128807]: Detected new or changed devices on smithi139 2023-07-14T19:25:09.443 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:25:08 smithi101 ceph-mon[128807]: Adjusting osd_memory_target on smithi139 to 17168M 2023-07-14T19:25:09.443 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:25:08 smithi101 ceph-mon[128807]: pgmap v70: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2023-07-14T19:25:09.443 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:25:08 smithi101 ceph-mon[128807]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2023-07-14T19:25:09.443 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:25:08 smithi101 ceph-mon[128807]: osd.2 [v2:172.21.15.139:6800/945038995,v1:172.21.15.139:6801/945038995] boot 2023-07-14T19:25:09.443 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:25:08 smithi101 ceph-mon[128807]: osdmap e18: 3 total, 3 up, 3 in 2023-07-14T19:25:09.444 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:25:08 smithi101 ceph-mon[128807]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2023-07-14T19:25:09.446 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:25:09.444+0000 7f573ba9f700 1 -- 172.21.15.98:0/3401369431 --> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi098:vg_nvme/lv_3", "target": ["mon-mgr", ""]}) v1 -- 0x7f57340fb150 con 0x7f57200a10c0 2023-07-14T19:25:10.531 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:25:10 smithi098 ceph-mon[127486]: osdmap e19: 3 total, 3 up, 3 in 2023-07-14T19:25:10.531 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:25:10 smithi098 ceph-mon[127486]: from='client.14292 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi098:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2023-07-14T19:25:10.532 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:25:10 smithi098 ceph-mon[127486]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-07-14T19:25:10.532 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:25:10 smithi098 ceph-mon[127486]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-07-14T19:25:10.532 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:25:10 smithi098 ceph-mon[127486]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T19:25:10.532 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:25:10 smithi098 ceph-mon[127486]: pgmap v73: 0 pgs: ; 0 B data, 80 MiB used, 268 GiB / 268 GiB avail 2023-07-14T19:25:10.532 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:25:10 smithi098 ceph-mon[127486]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]: dispatch 2023-07-14T19:25:10.691 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:25:10 smithi101 ceph-mon[128807]: osdmap e19: 3 total, 3 up, 3 in 2023-07-14T19:25:10.692 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:25:10 smithi101 ceph-mon[128807]: from='client.14292 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi098:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2023-07-14T19:25:10.692 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:25:10 smithi101 ceph-mon[128807]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-07-14T19:25:10.692 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:25:10 smithi101 ceph-mon[128807]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-07-14T19:25:10.692 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:25:10 smithi101 ceph-mon[128807]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T19:25:10.692 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:25:10 smithi101 ceph-mon[128807]: pgmap v73: 0 pgs: ; 0 B data, 80 MiB used, 268 GiB / 268 GiB avail 2023-07-14T19:25:10.692 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:25:10 smithi101 ceph-mon[128807]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]: dispatch 2023-07-14T19:25:10.720 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:25:10 smithi139 ceph-mon[129293]: osdmap e19: 3 total, 3 up, 3 in 2023-07-14T19:25:10.721 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:25:10 smithi139 ceph-mon[129293]: from='client.14292 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi098:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2023-07-14T19:25:10.721 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:25:10 smithi139 ceph-mon[129293]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-07-14T19:25:10.721 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:25:10 smithi139 ceph-mon[129293]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-07-14T19:25:10.721 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:25:10 smithi139 ceph-mon[129293]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T19:25:10.721 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:25:10 smithi139 ceph-mon[129293]: pgmap v73: 0 pgs: ; 0 B data, 80 MiB used, 268 GiB / 268 GiB avail 2023-07-14T19:25:10.721 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:25:10 smithi139 ceph-mon[129293]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]: dispatch 2023-07-14T19:25:11.531 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:25:11 smithi098 ceph-mon[127486]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2023-07-14T19:25:11.531 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:25:11 smithi098 ceph-mon[127486]: osdmap e20: 3 total, 3 up, 3 in 2023-07-14T19:25:11.531 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:25:11 smithi098 ceph-mon[127486]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2023-07-14T19:25:11.691 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:25:11 smithi101 ceph-mon[128807]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2023-07-14T19:25:11.692 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:25:11 smithi101 ceph-mon[128807]: osdmap e20: 3 total, 3 up, 3 in 2023-07-14T19:25:11.692 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:25:11 smithi101 ceph-mon[128807]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2023-07-14T19:25:11.720 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:25:11 smithi139 ceph-mon[129293]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2023-07-14T19:25:11.721 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:25:11 smithi139 ceph-mon[129293]: osdmap e20: 3 total, 3 up, 3 in 2023-07-14T19:25:11.721 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:25:11 smithi139 ceph-mon[129293]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2023-07-14T19:25:11.861 INFO:journalctl@ceph.osd.0.smithi098.stdout:Jul 14 19:25:11 smithi098 sudo[143542]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/smartctl -x --json=o /dev/nvme0n1 2023-07-14T19:25:11.861 INFO:journalctl@ceph.osd.0.smithi098.stdout:Jul 14 19:25:11 smithi098 sudo[143542]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2023-07-14T19:25:11.862 INFO:journalctl@ceph.osd.0.smithi098.stdout:Jul 14 19:25:11 smithi098 sudo[143542]: pam_unix(sudo:session): session opened for user root by (uid=0) 2023-07-14T19:25:11.862 INFO:journalctl@ceph.osd.0.smithi098.stdout:Jul 14 19:25:11 smithi098 sudo[143542]: pam_unix(sudo:session): session closed for user root 2023-07-14T19:25:12.252 INFO:journalctl@ceph.osd.0.smithi098.stdout:Jul 14 19:25:11 smithi098 sudo[143551]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2023-07-14T19:25:12.252 INFO:journalctl@ceph.osd.0.smithi098.stdout:Jul 14 19:25:11 smithi098 sudo[143551]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2023-07-14T19:25:12.252 INFO:journalctl@ceph.osd.0.smithi098.stdout:Jul 14 19:25:11 smithi098 sudo[143551]: pam_unix(sudo:session): session opened for user root by (uid=0) 2023-07-14T19:25:12.252 INFO:journalctl@ceph.osd.0.smithi098.stdout:Jul 14 19:25:11 smithi098 sudo[143551]: pam_unix(sudo:session): session closed for user root 2023-07-14T19:25:12.386 INFO:journalctl@ceph.osd.1.smithi101.stdout:Jul 14 19:25:12 smithi101 sudo[136204]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/smartctl -x --json=o /dev/nvme0n1 2023-07-14T19:25:12.386 INFO:journalctl@ceph.osd.1.smithi101.stdout:Jul 14 19:25:12 smithi101 sudo[136204]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2023-07-14T19:25:12.386 INFO:journalctl@ceph.osd.1.smithi101.stdout:Jul 14 19:25:12 smithi101 sudo[136204]: pam_unix(sudo:session): session opened for user root by (uid=0) 2023-07-14T19:25:12.387 INFO:journalctl@ceph.osd.1.smithi101.stdout:Jul 14 19:25:12 smithi101 sudo[136204]: pam_unix(sudo:session): session closed for user root 2023-07-14T19:25:12.387 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:25:12 smithi101 ceph-mon[128807]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2023-07-14T19:25:12.387 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:25:12 smithi101 ceph-mon[128807]: osdmap e21: 3 total, 3 up, 3 in 2023-07-14T19:25:12.388 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:25:12 smithi101 ceph-mon[128807]: pgmap v76: 1 pgs: 1 unknown; 0 B data, 80 MiB used, 268 GiB / 268 GiB avail 2023-07-14T19:25:12.388 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:25:12 smithi101 ceph-mon[128807]: from='client.? 172.21.15.98:0/2933235586' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "5530b265-31a5-447b-a1ea-17ebae9cfddf"}]: dispatch 2023-07-14T19:25:12.388 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:25:12 smithi101 ceph-mon[128807]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "5530b265-31a5-447b-a1ea-17ebae9cfddf"}]: dispatch 2023-07-14T19:25:12.389 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:25:12 smithi101 ceph-mon[128807]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "5530b265-31a5-447b-a1ea-17ebae9cfddf"}]': finished 2023-07-14T19:25:12.389 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:25:12 smithi101 ceph-mon[128807]: osdmap e22: 4 total, 3 up, 4 in 2023-07-14T19:25:12.389 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:25:12 smithi101 ceph-mon[128807]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2023-07-14T19:25:12.389 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:25:12 smithi101 ceph-mon[128807]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' 2023-07-14T19:25:12.531 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:25:12 smithi098 ceph-mon[127486]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2023-07-14T19:25:12.531 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:25:12 smithi098 ceph-mon[127486]: osdmap e21: 3 total, 3 up, 3 in 2023-07-14T19:25:12.531 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:25:12 smithi098 ceph-mon[127486]: pgmap v76: 1 pgs: 1 unknown; 0 B data, 80 MiB used, 268 GiB / 268 GiB avail 2023-07-14T19:25:12.532 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:25:12 smithi098 ceph-mon[127486]: from='client.? 172.21.15.98:0/2933235586' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "5530b265-31a5-447b-a1ea-17ebae9cfddf"}]: dispatch 2023-07-14T19:25:12.532 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:25:12 smithi098 ceph-mon[127486]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "5530b265-31a5-447b-a1ea-17ebae9cfddf"}]: dispatch 2023-07-14T19:25:12.532 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:25:12 smithi098 ceph-mon[127486]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "5530b265-31a5-447b-a1ea-17ebae9cfddf"}]': finished 2023-07-14T19:25:12.532 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:25:12 smithi098 ceph-mon[127486]: osdmap e22: 4 total, 3 up, 4 in 2023-07-14T19:25:12.532 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:25:12 smithi098 ceph-mon[127486]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2023-07-14T19:25:12.532 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:25:12 smithi098 ceph-mon[127486]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' 2023-07-14T19:25:12.656 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:25:12 smithi139 ceph-mon[129293]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2023-07-14T19:25:12.656 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:25:12 smithi139 ceph-mon[129293]: osdmap e21: 3 total, 3 up, 3 in 2023-07-14T19:25:12.656 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:25:12 smithi139 ceph-mon[129293]: pgmap v76: 1 pgs: 1 unknown; 0 B data, 80 MiB used, 268 GiB / 268 GiB avail 2023-07-14T19:25:12.656 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:25:12 smithi139 ceph-mon[129293]: from='client.? 172.21.15.98:0/2933235586' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "5530b265-31a5-447b-a1ea-17ebae9cfddf"}]: dispatch 2023-07-14T19:25:12.657 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:25:12 smithi139 ceph-mon[129293]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "5530b265-31a5-447b-a1ea-17ebae9cfddf"}]: dispatch 2023-07-14T19:25:12.657 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:25:12 smithi139 ceph-mon[129293]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "5530b265-31a5-447b-a1ea-17ebae9cfddf"}]': finished 2023-07-14T19:25:12.657 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:25:12 smithi139 ceph-mon[129293]: osdmap e22: 4 total, 3 up, 4 in 2023-07-14T19:25:12.657 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:25:12 smithi139 ceph-mon[129293]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2023-07-14T19:25:12.657 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:25:12 smithi139 ceph-mon[129293]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' 2023-07-14T19:25:12.691 INFO:journalctl@ceph.osd.1.smithi101.stdout:Jul 14 19:25:12 smithi101 sudo[136207]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2023-07-14T19:25:12.692 INFO:journalctl@ceph.osd.1.smithi101.stdout:Jul 14 19:25:12 smithi101 sudo[136207]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2023-07-14T19:25:12.692 INFO:journalctl@ceph.osd.1.smithi101.stdout:Jul 14 19:25:12 smithi101 sudo[136207]: pam_unix(sudo:session): session opened for user root by (uid=0) 2023-07-14T19:25:12.692 INFO:journalctl@ceph.osd.1.smithi101.stdout:Jul 14 19:25:12 smithi101 sudo[136207]: pam_unix(sudo:session): session closed for user root 2023-07-14T19:25:12.900 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:25:12.898+0000 7f572affd700 1 -- 172.21.15.98:0/3401369431 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mgrmap(e 16) v1 ==== 135715+0+0 (secure 0 0 0) 0x7f5724096730 con 0x7f57340ffdb0 2023-07-14T19:25:12.908 INFO:journalctl@ceph.osd.2.smithi139.stdout:Jul 14 19:25:12 smithi139 sudo[137345]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/smartctl -x --json=o /dev/nvme0n1 2023-07-14T19:25:12.908 INFO:journalctl@ceph.osd.2.smithi139.stdout:Jul 14 19:25:12 smithi139 sudo[137345]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2023-07-14T19:25:12.908 INFO:journalctl@ceph.osd.2.smithi139.stdout:Jul 14 19:25:12 smithi139 sudo[137345]: pam_unix(sudo:session): session opened for user root by (uid=0) 2023-07-14T19:25:12.909 INFO:journalctl@ceph.osd.2.smithi139.stdout:Jul 14 19:25:12 smithi139 sudo[137345]: pam_unix(sudo:session): session closed for user root 2023-07-14T19:25:13.221 INFO:journalctl@ceph.osd.2.smithi139.stdout:Jul 14 19:25:12 smithi139 sudo[137348]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2023-07-14T19:25:13.221 INFO:journalctl@ceph.osd.2.smithi139.stdout:Jul 14 19:25:12 smithi139 sudo[137348]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2023-07-14T19:25:13.221 INFO:journalctl@ceph.osd.2.smithi139.stdout:Jul 14 19:25:12 smithi139 sudo[137348]: pam_unix(sudo:session): session opened for user root by (uid=0) 2023-07-14T19:25:13.222 INFO:journalctl@ceph.osd.2.smithi139.stdout:Jul 14 19:25:12 smithi139 sudo[137348]: pam_unix(sudo:session): session closed for user root 2023-07-14T19:25:13.446 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:25:13 smithi098 sudo[143636]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/smartctl -x --json=o /dev/nvme0n1 2023-07-14T19:25:13.446 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:25:13 smithi098 sudo[143636]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2023-07-14T19:25:13.447 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:25:13 smithi098 sudo[143636]: pam_unix(sudo:session): session opened for user root by (uid=0) 2023-07-14T19:25:13.447 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:25:13 smithi098 sudo[143636]: pam_unix(sudo:session): session closed for user root 2023-07-14T19:25:13.705 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:25:13 smithi139 ceph-mon[129293]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' 2023-07-14T19:25:13.706 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:25:13 smithi139 ceph-mon[129293]: from='client.? 172.21.15.98:0/1631764594' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-07-14T19:25:13.706 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:25:13 smithi139 ceph-mon[129293]: mgrmap e16: x(active, since 2m), standbys: z, y 2023-07-14T19:25:13.706 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:25:13 smithi139 ceph-mon[129293]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' 2023-07-14T19:25:13.706 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:25:13 smithi139 ceph-mon[129293]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2023-07-14T19:25:13.706 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:25:13 smithi139 sudo[137351]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/smartctl -x --json=o /dev/nvme0n1 2023-07-14T19:25:13.706 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:25:13 smithi139 sudo[137351]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2023-07-14T19:25:13.706 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:25:13 smithi139 sudo[137351]: pam_unix(sudo:session): session opened for user root by (uid=0) 2023-07-14T19:25:13.707 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:25:13 smithi139 sudo[137351]: pam_unix(sudo:session): session closed for user root 2023-07-14T19:25:13.781 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:25:13 smithi098 sudo[143825]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2023-07-14T19:25:13.781 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:25:13 smithi098 sudo[143825]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2023-07-14T19:25:13.782 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:25:13 smithi098 sudo[143825]: pam_unix(sudo:session): session opened for user root by (uid=0) 2023-07-14T19:25:13.782 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:25:13 smithi098 sudo[143825]: pam_unix(sudo:session): session closed for user root 2023-07-14T19:25:13.782 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:25:13 smithi098 ceph-mon[127486]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' 2023-07-14T19:25:13.782 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:25:13 smithi098 ceph-mon[127486]: from='client.? 172.21.15.98:0/1631764594' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-07-14T19:25:13.782 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:25:13 smithi098 ceph-mon[127486]: mgrmap e16: x(active, since 2m), standbys: z, y 2023-07-14T19:25:13.782 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:25:13 smithi098 ceph-mon[127486]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' 2023-07-14T19:25:13.782 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:25:13 smithi098 ceph-mon[127486]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2023-07-14T19:25:13.941 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:25:13 smithi101 ceph-mon[128807]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' 2023-07-14T19:25:13.942 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:25:13 smithi101 ceph-mon[128807]: from='client.? 172.21.15.98:0/1631764594' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-07-14T19:25:13.942 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:25:13 smithi101 ceph-mon[128807]: mgrmap e16: x(active, since 2m), standbys: z, y 2023-07-14T19:25:13.942 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:25:13 smithi101 ceph-mon[128807]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' 2023-07-14T19:25:13.942 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:25:13 smithi101 ceph-mon[128807]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2023-07-14T19:25:13.970 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:25:13 smithi139 sudo[137354]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2023-07-14T19:25:13.971 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:25:13 smithi139 sudo[137354]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2023-07-14T19:25:13.971 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:25:13 smithi139 sudo[137354]: pam_unix(sudo:session): session opened for user root by (uid=0) 2023-07-14T19:25:13.971 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:25:13 smithi139 sudo[137354]: pam_unix(sudo:session): session closed for user root 2023-07-14T19:25:14.435 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:25:14 smithi101 sudo[136210]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/smartctl -x --json=o /dev/nvme0n1 2023-07-14T19:25:14.435 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:25:14 smithi101 sudo[136210]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2023-07-14T19:25:14.435 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:25:14 smithi101 sudo[136210]: pam_unix(sudo:session): session opened for user root by (uid=0) 2023-07-14T19:25:14.435 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:25:14 smithi101 sudo[136210]: pam_unix(sudo:session): session closed for user root 2023-07-14T19:25:14.691 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:25:14 smithi101 sudo[136213]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2023-07-14T19:25:14.692 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:25:14 smithi101 sudo[136213]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2023-07-14T19:25:14.692 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:25:14 smithi101 sudo[136213]: pam_unix(sudo:session): session opened for user root by (uid=0) 2023-07-14T19:25:14.692 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:25:14 smithi101 sudo[136213]: pam_unix(sudo:session): session closed for user root 2023-07-14T19:25:14.692 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:25:14 smithi101 ceph-mon[128807]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2023-07-14T19:25:14.692 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:25:14 smithi101 ceph-mon[128807]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2023-07-14T19:25:14.692 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:25:14 smithi101 ceph-mon[128807]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-07-14T19:25:14.692 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:25:14 smithi101 ceph-mon[128807]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-07-14T19:25:14.693 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:25:14 smithi101 ceph-mon[128807]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2023-07-14T19:25:14.693 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:25:14 smithi101 ceph-mon[128807]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2023-07-14T19:25:14.693 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:25:14 smithi101 ceph-mon[128807]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-07-14T19:25:14.693 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:25:14 smithi101 ceph-mon[128807]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-07-14T19:25:14.693 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:25:14 smithi101 ceph-mon[128807]: pgmap v78: 1 pgs: 1 active+clean; 577 KiB data, 81 MiB used, 268 GiB / 268 GiB avail 2023-07-14T19:25:14.693 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:25:14 smithi101 ceph-mon[128807]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2023-07-14T19:25:14.693 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:25:14 smithi101 ceph-mon[128807]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2023-07-14T19:25:14.694 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:25:14 smithi101 ceph-mon[128807]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2023-07-14T19:25:14.694 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:25:14 smithi101 ceph-mon[128807]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-07-14T19:25:14.694 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:25:14 smithi101 ceph-mon[128807]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-07-14T19:25:14.694 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:25:14 smithi101 ceph-mon[128807]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2023-07-14T19:25:14.720 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:25:14 smithi139 ceph-mon[129293]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2023-07-14T19:25:14.721 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:25:14 smithi139 ceph-mon[129293]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2023-07-14T19:25:14.721 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:25:14 smithi139 ceph-mon[129293]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-07-14T19:25:14.721 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:25:14 smithi139 ceph-mon[129293]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-07-14T19:25:14.721 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:25:14 smithi139 ceph-mon[129293]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2023-07-14T19:25:14.721 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:25:14 smithi139 ceph-mon[129293]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2023-07-14T19:25:14.721 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:25:14 smithi139 ceph-mon[129293]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-07-14T19:25:14.721 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:25:14 smithi139 ceph-mon[129293]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-07-14T19:25:14.722 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:25:14 smithi139 ceph-mon[129293]: pgmap v78: 1 pgs: 1 active+clean; 577 KiB data, 81 MiB used, 268 GiB / 268 GiB avail 2023-07-14T19:25:14.722 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:25:14 smithi139 ceph-mon[129293]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2023-07-14T19:25:14.722 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:25:14 smithi139 ceph-mon[129293]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2023-07-14T19:25:14.722 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:25:14 smithi139 ceph-mon[129293]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2023-07-14T19:25:14.722 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:25:14 smithi139 ceph-mon[129293]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-07-14T19:25:14.723 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:25:14 smithi139 ceph-mon[129293]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-07-14T19:25:14.723 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:25:14 smithi139 ceph-mon[129293]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2023-07-14T19:25:14.781 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:25:14 smithi098 ceph-mon[127486]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2023-07-14T19:25:14.781 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:25:14 smithi098 ceph-mon[127486]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2023-07-14T19:25:14.782 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:25:14 smithi098 ceph-mon[127486]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-07-14T19:25:14.782 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:25:14 smithi098 ceph-mon[127486]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-07-14T19:25:14.782 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:25:14 smithi098 ceph-mon[127486]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2023-07-14T19:25:14.782 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:25:14 smithi098 ceph-mon[127486]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2023-07-14T19:25:14.782 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:25:14 smithi098 ceph-mon[127486]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-07-14T19:25:14.782 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:25:14 smithi098 ceph-mon[127486]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-07-14T19:25:14.782 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:25:14 smithi098 ceph-mon[127486]: pgmap v78: 1 pgs: 1 active+clean; 577 KiB data, 81 MiB used, 268 GiB / 268 GiB avail 2023-07-14T19:25:14.783 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:25:14 smithi098 ceph-mon[127486]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2023-07-14T19:25:14.783 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:25:14 smithi098 ceph-mon[127486]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2023-07-14T19:25:14.783 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:25:14 smithi098 ceph-mon[127486]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2023-07-14T19:25:14.783 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:25:14 smithi098 ceph-mon[127486]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-07-14T19:25:14.783 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:25:14 smithi098 ceph-mon[127486]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-07-14T19:25:14.783 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:25:14 smithi098 ceph-mon[127486]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2023-07-14T19:25:16.882 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:25:16 smithi098 ceph-mon[127486]: pgmap v79: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2023-07-14T19:25:16.941 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:25:16 smithi101 ceph-mon[128807]: pgmap v79: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2023-07-14T19:25:16.970 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:25:16 smithi139 ceph-mon[129293]: pgmap v79: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2023-07-14T19:25:18.781 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:25:18 smithi098 ceph-mon[127486]: pgmap v80: 1 pgs: 1 active+clean; 577 KiB data, 83 MiB used, 268 GiB / 268 GiB avail 2023-07-14T19:25:18.782 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:25:18 smithi098 ceph-mon[127486]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2023-07-14T19:25:18.782 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:25:18 smithi098 ceph-mon[127486]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T19:25:18.941 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:25:18 smithi101 ceph-mon[128807]: pgmap v80: 1 pgs: 1 active+clean; 577 KiB data, 83 MiB used, 268 GiB / 268 GiB avail 2023-07-14T19:25:18.942 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:25:18 smithi101 ceph-mon[128807]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2023-07-14T19:25:18.942 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:25:18 smithi101 ceph-mon[128807]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T19:25:18.971 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:25:18 smithi139 ceph-mon[129293]: pgmap v80: 1 pgs: 1 active+clean; 577 KiB data, 83 MiB used, 268 GiB / 268 GiB avail 2023-07-14T19:25:18.971 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:25:18 smithi139 ceph-mon[129293]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2023-07-14T19:25:18.971 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:25:18 smithi139 ceph-mon[129293]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T19:25:19.704 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:25:19 smithi098 ceph-mon[127486]: Deploying daemon osd.3 on smithi098 2023-07-14T19:25:19.941 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:25:19 smithi101 ceph-mon[128807]: Deploying daemon osd.3 on smithi098 2023-07-14T19:25:19.970 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:25:19 smithi139 ceph-mon[129293]: Deploying daemon osd.3 on smithi098 2023-07-14T19:25:20.774 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:25:20 smithi098 ceph-mon[127486]: pgmap v81: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail; 1.7 KiB/s rd, 44 KiB/s wr, 4 op/s 2023-07-14T19:25:20.941 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:25:20 smithi101 ceph-mon[128807]: pgmap v81: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail; 1.7 KiB/s rd, 44 KiB/s wr, 4 op/s 2023-07-14T19:25:20.970 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:25:20 smithi139 ceph-mon[129293]: pgmap v81: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail; 1.7 KiB/s rd, 44 KiB/s wr, 4 op/s 2023-07-14T19:25:22.532 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:25:22 smithi098 ceph-mon[127486]: pgmap v82: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail; 1.2 KiB/s rd, 33 KiB/s wr, 3 op/s 2023-07-14T19:25:22.691 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:25:22 smithi101 ceph-mon[128807]: pgmap v82: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail; 1.2 KiB/s rd, 33 KiB/s wr, 3 op/s 2023-07-14T19:25:22.720 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:25:22 smithi139 ceph-mon[129293]: pgmap v82: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail; 1.2 KiB/s rd, 33 KiB/s wr, 3 op/s 2023-07-14T19:25:23.491 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:25:23 smithi098 ceph-mon[127486]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-14T19:25:23.491 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:25:23 smithi098 ceph-mon[127486]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' 2023-07-14T19:25:23.491 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:25:23 smithi098 ceph-mon[127486]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' 2023-07-14T19:25:23.691 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:25:23 smithi101 ceph-mon[128807]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-14T19:25:23.692 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:25:23 smithi101 ceph-mon[128807]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' 2023-07-14T19:25:23.692 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:25:23 smithi101 ceph-mon[128807]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' 2023-07-14T19:25:23.720 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:25:23 smithi139 ceph-mon[129293]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-14T19:25:23.721 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:25:23 smithi139 ceph-mon[129293]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' 2023-07-14T19:25:23.721 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:25:23 smithi139 ceph-mon[129293]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' 2023-07-14T19:25:24.463 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:25:24 smithi098 ceph-mon[127486]: pgmap v83: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail; 1023 B/s rd, 26 KiB/s wr, 2 op/s 2023-07-14T19:25:24.691 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:25:24 smithi101 ceph-mon[128807]: pgmap v83: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail; 1023 B/s rd, 26 KiB/s wr, 2 op/s 2023-07-14T19:25:24.720 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:25:24 smithi139 ceph-mon[129293]: pgmap v83: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail; 1023 B/s rd, 26 KiB/s wr, 2 op/s 2023-07-14T19:25:25.902 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:25:25.900+0000 7f572affd700 1 -- 172.21.15.98:0/3401369431 <== mgr.14152 v2:172.21.15.98:6800/452042164 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7f57340fb150 con 0x7f57200a10c0 2023-07-14T19:25:25.903 INFO:teuthology.orchestra.run.smithi098.stdout:Created osd(s) 3 on host 'smithi098' 2023-07-14T19:25:25.904 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:25:25.902+0000 7f573ba9f700 1 -- 172.21.15.98:0/3401369431 >> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] conn(0x7f57200a10c0 msgr2=0x7f57200a3570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:25:25.904 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:25:25.902+0000 7f573ba9f700 1 --2- 172.21.15.98:0/3401369431 >> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] conn(0x7f57200a10c0 0x7f57200a3570 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f57340ffc10 tx=0x7f5730009260 comp rx=0 tx=0).stop 2023-07-14T19:25:25.904 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:25:25.902+0000 7f573ba9f700 1 -- 172.21.15.98:0/3401369431 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f57340ffdb0 msgr2=0x7f5734101190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:25:25.905 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:25:25.902+0000 7f573ba9f700 1 --2- 172.21.15.98:0/3401369431 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f57340ffdb0 0x7f5734101190 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f572400e9a0 tx=0x7f572400ed60 comp rx=0 tx=0).stop 2023-07-14T19:25:25.905 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:25:25.902+0000 7f573ba9f700 1 -- 172.21.15.98:0/3401369431 shutdown_connections 2023-07-14T19:25:25.905 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:25:25.902+0000 7f573ba9f700 1 --2- 172.21.15.98:0/3401369431 >> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] conn(0x7f57200a10c0 0x7f57200a3570 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:25:25.905 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:25:25.902+0000 7f573ba9f700 1 --2- 172.21.15.98:0/3401369431 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f57340febb0 0x7f5734102b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:25:25.905 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:25:25.902+0000 7f573ba9f700 1 --2- 172.21.15.98:0/3401369431 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f5734100740 0x7f57341016d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:25:25.905 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:25:25.902+0000 7f573ba9f700 1 --2- 172.21.15.98:0/3401369431 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f57340ffdb0 0x7f5734101190 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:25:25.905 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:25:25.902+0000 7f573ba9f700 1 -- 172.21.15.98:0/3401369431 >> 172.21.15.98:0/3401369431 conn(0x7f57340fa1e0 msgr2=0x7f57340faa40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:25:25.906 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:25:25.903+0000 7f573ba9f700 1 -- 172.21.15.98:0/3401369431 shutdown_connections 2023-07-14T19:25:25.906 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:25:25.903+0000 7f573ba9f700 1 -- 172.21.15.98:0/3401369431 wait complete. 2023-07-14T19:25:26.345 DEBUG:teuthology.orchestra.run.smithi098:osd.3> sudo journalctl -f -n 0 -u ceph-9adea648-227b-11ee-9b33-001a4aab830c@osd.3.service 2023-07-14T19:25:26.348 INFO:tasks.cephadm:Deploying osd.4 on smithi101 with /dev/vg_nvme/lv_3... 2023-07-14T19:25:26.349 DEBUG:teuthology.orchestra.run.smithi101:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81c9387d2e7576da065747ec9c16d1c7d69016ec shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 9adea648-227b-11ee-9b33-001a4aab830c -- ceph-volume lvm zap /dev/vg_nvme/lv_3 2023-07-14T19:25:26.380 INFO:journalctl@ceph.osd.3.smithi098.stdout:-- Logs begin at Fri 2023-07-14 18:57:10 UTC. -- 2023-07-14T19:25:26.531 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:25:26 smithi098 ceph-mon[127486]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' 2023-07-14T19:25:26.532 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:25:26 smithi098 ceph-mon[127486]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' 2023-07-14T19:25:26.532 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:25:26 smithi098 ceph-mon[127486]: pgmap v84: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail; 853 B/s rd, 22 KiB/s wr, 2 op/s 2023-07-14T19:25:26.532 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:25:26 smithi098 ceph-mon[127486]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T19:25:26.533 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:25:26 smithi098 ceph-mon[127486]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-14T19:25:26.533 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:25:26 smithi098 ceph-mon[127486]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' 2023-07-14T19:25:26.533 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:25:26 smithi098 ceph-mon[127486]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' 2023-07-14T19:25:26.533 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:25:26 smithi098 ceph-mon[127486]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' 2023-07-14T19:25:26.534 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:25:26 smithi098 ceph-mon[127486]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-14T19:25:26.691 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:25:26 smithi101 ceph-mon[128807]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' 2023-07-14T19:25:26.692 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:25:26 smithi101 ceph-mon[128807]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' 2023-07-14T19:25:26.692 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:25:26 smithi101 ceph-mon[128807]: pgmap v84: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail; 853 B/s rd, 22 KiB/s wr, 2 op/s 2023-07-14T19:25:26.692 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:25:26 smithi101 ceph-mon[128807]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T19:25:26.692 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:25:26 smithi101 ceph-mon[128807]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-14T19:25:26.692 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:25:26 smithi101 ceph-mon[128807]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' 2023-07-14T19:25:26.692 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:25:26 smithi101 ceph-mon[128807]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' 2023-07-14T19:25:26.692 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:25:26 smithi101 ceph-mon[128807]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' 2023-07-14T19:25:26.693 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:25:26 smithi101 ceph-mon[128807]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-14T19:25:26.720 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:25:26 smithi139 ceph-mon[129293]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' 2023-07-14T19:25:26.721 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:25:26 smithi139 ceph-mon[129293]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' 2023-07-14T19:25:26.721 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:25:26 smithi139 ceph-mon[129293]: pgmap v84: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail; 853 B/s rd, 22 KiB/s wr, 2 op/s 2023-07-14T19:25:26.721 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:25:26 smithi139 ceph-mon[129293]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T19:25:26.721 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:25:26 smithi139 ceph-mon[129293]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-14T19:25:26.721 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:25:26 smithi139 ceph-mon[129293]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' 2023-07-14T19:25:26.721 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:25:26 smithi139 ceph-mon[129293]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' 2023-07-14T19:25:26.722 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:25:26 smithi139 ceph-mon[129293]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' 2023-07-14T19:25:26.722 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:25:26 smithi139 ceph-mon[129293]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-14T19:25:26.911 INFO:teuthology.orchestra.run.smithi101.stderr:Inferring config /var/lib/ceph/9adea648-227b-11ee-9b33-001a4aab830c/mon.b/config 2023-07-14T19:25:27.133 INFO:journalctl@ceph.osd.3.smithi098.stdout:Jul 14 19:25:27 smithi098 ceph-9adea648-227b-11ee-9b33-001a4aab830c-osd-3[145819]: 2023-07-14T19:25:27.001+0000 7f6b243f7640 -1 osd.3 0 log_to_monitors true 2023-07-14T19:25:27.441 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:25:27 smithi098 ceph-mon[127486]: from='osd.3 [v2:172.21.15.98:6810/3095249665,v1:172.21.15.98:6811/3095249665]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2023-07-14T19:25:27.645 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:25:27 smithi101 ceph-mon[128807]: from='osd.3 [v2:172.21.15.98:6810/3095249665,v1:172.21.15.98:6811/3095249665]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2023-07-14T19:25:27.720 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:25:27 smithi139 ceph-mon[129293]: from='osd.3 [v2:172.21.15.98:6810/3095249665,v1:172.21.15.98:6811/3095249665]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2023-07-14T19:25:28.410 INFO:teuthology.orchestra.run.smithi101.stderr:--> Zapping: /dev/vg_nvme/lv_3 2023-07-14T19:25:28.411 INFO:teuthology.orchestra.run.smithi101.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 conv=fsync 2023-07-14T19:25:28.411 INFO:teuthology.orchestra.run.smithi101.stderr: stderr: 10+0 records in 2023-07-14T19:25:28.411 INFO:teuthology.orchestra.run.smithi101.stderr:10+0 records out 2023-07-14T19:25:28.411 INFO:teuthology.orchestra.run.smithi101.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0109471 s, 958 MB/s 2023-07-14T19:25:28.411 INFO:teuthology.orchestra.run.smithi101.stderr:--> Zapping successful for: 2023-07-14T19:25:28.512 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:25:28 smithi098 ceph-mon[127486]: from='osd.3 [v2:172.21.15.98:6810/3095249665,v1:172.21.15.98:6811/3095249665]' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2023-07-14T19:25:28.512 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:25:28 smithi098 ceph-mon[127486]: osdmap e23: 4 total, 3 up, 4 in 2023-07-14T19:25:28.512 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:25:28 smithi098 ceph-mon[127486]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2023-07-14T19:25:28.513 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:25:28 smithi098 ceph-mon[127486]: from='osd.3 [v2:172.21.15.98:6810/3095249665,v1:172.21.15.98:6811/3095249665]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi098", "root=default"]}]: dispatch 2023-07-14T19:25:28.513 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:25:28 smithi098 ceph-mon[127486]: pgmap v86: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail; 1023 B/s rd, 26 KiB/s wr, 2 op/s 2023-07-14T19:25:28.513 INFO:journalctl@ceph.osd.3.smithi098.stdout:Jul 14 19:25:28 smithi098 ceph-9adea648-227b-11ee-9b33-001a4aab830c-osd-3[145819]: 2023-07-14T19:25:28.376+0000 7f6b1926e700 -1 osd.3 0 waiting for initial osdmap 2023-07-14T19:25:28.513 INFO:journalctl@ceph.osd.3.smithi098.stdout:Jul 14 19:25:28 smithi098 ceph-9adea648-227b-11ee-9b33-001a4aab830c-osd-3[145819]: 2023-07-14T19:25:28.381+0000 7f6b1185d700 -1 osd.3 24 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-07-14T19:25:28.691 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:25:28 smithi101 ceph-mon[128807]: from='osd.3 [v2:172.21.15.98:6810/3095249665,v1:172.21.15.98:6811/3095249665]' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2023-07-14T19:25:28.692 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:25:28 smithi101 ceph-mon[128807]: osdmap e23: 4 total, 3 up, 4 in 2023-07-14T19:25:28.692 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:25:28 smithi101 ceph-mon[128807]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2023-07-14T19:25:28.692 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:25:28 smithi101 ceph-mon[128807]: from='osd.3 [v2:172.21.15.98:6810/3095249665,v1:172.21.15.98:6811/3095249665]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi098", "root=default"]}]: dispatch 2023-07-14T19:25:28.692 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:25:28 smithi101 ceph-mon[128807]: pgmap v86: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail; 1023 B/s rd, 26 KiB/s wr, 2 op/s 2023-07-14T19:25:28.720 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:25:28 smithi139 ceph-mon[129293]: from='osd.3 [v2:172.21.15.98:6810/3095249665,v1:172.21.15.98:6811/3095249665]' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2023-07-14T19:25:28.721 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:25:28 smithi139 ceph-mon[129293]: osdmap e23: 4 total, 3 up, 4 in 2023-07-14T19:25:28.721 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:25:28 smithi139 ceph-mon[129293]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2023-07-14T19:25:28.721 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:25:28 smithi139 ceph-mon[129293]: from='osd.3 [v2:172.21.15.98:6810/3095249665,v1:172.21.15.98:6811/3095249665]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi098", "root=default"]}]: dispatch 2023-07-14T19:25:28.721 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:25:28 smithi139 ceph-mon[129293]: pgmap v86: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail; 1023 B/s rd, 26 KiB/s wr, 2 op/s 2023-07-14T19:25:28.864 DEBUG:teuthology.orchestra.run.smithi101:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81c9387d2e7576da065747ec9c16d1c7d69016ec shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 9adea648-227b-11ee-9b33-001a4aab830c -- ceph orch daemon add osd smithi101:vg_nvme/lv_3 2023-07-14T19:25:29.412 INFO:teuthology.orchestra.run.smithi101.stderr:Inferring config /var/lib/ceph/9adea648-227b-11ee-9b33-001a4aab830c/mon.b/config 2023-07-14T19:25:29.637 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:25:29 smithi101 ceph-mon[128807]: from='osd.3 [v2:172.21.15.98:6810/3095249665,v1:172.21.15.98:6811/3095249665]' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi098", "root=default"]}]': finished 2023-07-14T19:25:29.637 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:25:29 smithi101 ceph-mon[128807]: osdmap e24: 4 total, 3 up, 4 in 2023-07-14T19:25:29.637 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:25:29 smithi101 ceph-mon[128807]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2023-07-14T19:25:29.637 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:25:29 smithi101 ceph-mon[128807]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2023-07-14T19:25:29.638 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:25:29 smithi101 ceph-mon[128807]: Detected new or changed devices on smithi098 2023-07-14T19:25:29.638 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:25:29 smithi101 ceph-mon[128807]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' 2023-07-14T19:25:29.638 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:25:29 smithi101 ceph-mon[128807]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' 2023-07-14T19:25:29.638 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:25:29 smithi101 ceph-mon[128807]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2023-07-14T19:25:29.638 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:25:29 smithi101 ceph-mon[128807]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2023-07-14T19:25:29.638 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:25:29 smithi101 ceph-mon[128807]: Adjusting osd_memory_target on smithi098 to 8584M 2023-07-14T19:25:29.639 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:25:29 smithi101 ceph-mon[128807]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' 2023-07-14T19:25:29.639 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:25:29 smithi101 ceph-mon[128807]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T19:25:29.639 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:25:29 smithi101 ceph-mon[128807]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-14T19:25:29.639 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:25:29 smithi101 ceph-mon[128807]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' 2023-07-14T19:25:29.720 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:25:29 smithi139 ceph-mon[129293]: from='osd.3 [v2:172.21.15.98:6810/3095249665,v1:172.21.15.98:6811/3095249665]' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi098", "root=default"]}]': finished 2023-07-14T19:25:29.721 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:25:29 smithi139 ceph-mon[129293]: osdmap e24: 4 total, 3 up, 4 in 2023-07-14T19:25:29.721 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:25:29 smithi139 ceph-mon[129293]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2023-07-14T19:25:29.721 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:25:29 smithi139 ceph-mon[129293]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2023-07-14T19:25:29.721 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:25:29 smithi139 ceph-mon[129293]: Detected new or changed devices on smithi098 2023-07-14T19:25:29.722 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:25:29 smithi139 ceph-mon[129293]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' 2023-07-14T19:25:29.722 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:25:29 smithi139 ceph-mon[129293]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' 2023-07-14T19:25:29.722 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:25:29 smithi139 ceph-mon[129293]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2023-07-14T19:25:29.722 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:25:29 smithi139 ceph-mon[129293]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2023-07-14T19:25:29.722 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:25:29 smithi139 ceph-mon[129293]: Adjusting osd_memory_target on smithi098 to 8584M 2023-07-14T19:25:29.722 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:25:29 smithi139 ceph-mon[129293]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' 2023-07-14T19:25:29.722 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:25:29 smithi139 ceph-mon[129293]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T19:25:29.723 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:25:29 smithi139 ceph-mon[129293]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-14T19:25:29.723 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:25:29 smithi139 ceph-mon[129293]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' 2023-07-14T19:25:29.781 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:25:29 smithi098 ceph-mon[127486]: from='osd.3 [v2:172.21.15.98:6810/3095249665,v1:172.21.15.98:6811/3095249665]' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi098", "root=default"]}]': finished 2023-07-14T19:25:29.781 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:25:29 smithi098 ceph-mon[127486]: osdmap e24: 4 total, 3 up, 4 in 2023-07-14T19:25:29.782 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:25:29 smithi098 ceph-mon[127486]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2023-07-14T19:25:29.782 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:25:29 smithi098 ceph-mon[127486]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2023-07-14T19:25:29.782 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:25:29 smithi098 ceph-mon[127486]: Detected new or changed devices on smithi098 2023-07-14T19:25:29.782 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:25:29 smithi098 ceph-mon[127486]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' 2023-07-14T19:25:29.782 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:25:29 smithi098 ceph-mon[127486]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' 2023-07-14T19:25:29.782 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:25:29 smithi098 ceph-mon[127486]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2023-07-14T19:25:29.782 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:25:29 smithi098 ceph-mon[127486]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2023-07-14T19:25:29.783 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:25:29 smithi098 ceph-mon[127486]: Adjusting osd_memory_target on smithi098 to 8584M 2023-07-14T19:25:29.783 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:25:29 smithi098 ceph-mon[127486]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' 2023-07-14T19:25:29.783 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:25:29 smithi098 ceph-mon[127486]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T19:25:29.783 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:25:29 smithi098 ceph-mon[127486]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-14T19:25:29.783 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:25:29 smithi098 ceph-mon[127486]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' 2023-07-14T19:25:30.587 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-14T19:25:30.582+0000 7f4ba6bcf700 1 -- 172.21.15.101:0/3253419395 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f4ba0100740 msgr2=0x7f4ba0107000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:25:30.587 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-14T19:25:30.582+0000 7f4ba6bcf700 1 --2- 172.21.15.101:0/3253419395 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f4ba0100740 0x7f4ba0107000 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f4b9800b0a0 tx=0x7f4b9800b3b0 comp rx=0 tx=0).stop 2023-07-14T19:25:30.588 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-14T19:25:30.582+0000 7f4ba6bcf700 1 -- 172.21.15.101:0/3253419395 shutdown_connections 2023-07-14T19:25:30.588 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-14T19:25:30.582+0000 7f4ba6bcf700 1 --2- 172.21.15.101:0/3253419395 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f4ba00febb0 0x7f4ba00fefc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:25:30.588 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-14T19:25:30.582+0000 7f4ba6bcf700 1 --2- 172.21.15.101:0/3253419395 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f4ba0100740 0x7f4ba0107000 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:25:30.588 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-14T19:25:30.582+0000 7f4ba6bcf700 1 --2- 172.21.15.101:0/3253419395 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4ba00ffdb0 0x7f4ba0100200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:25:30.589 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-14T19:25:30.582+0000 7f4ba6bcf700 1 -- 172.21.15.101:0/3253419395 >> 172.21.15.101:0/3253419395 conn(0x7f4ba00fa1e0 msgr2=0x7f4ba00fc630 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:25:30.589 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-14T19:25:30.583+0000 7f4ba6bcf700 1 -- 172.21.15.101:0/3253419395 shutdown_connections 2023-07-14T19:25:30.589 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-14T19:25:30.583+0000 7f4ba6bcf700 1 -- 172.21.15.101:0/3253419395 wait complete. 2023-07-14T19:25:30.589 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-14T19:25:30.583+0000 7f4ba6bcf700 1 Processor -- start 2023-07-14T19:25:30.590 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-14T19:25:30.583+0000 7f4ba6bcf700 1 -- start start 2023-07-14T19:25:30.590 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-14T19:25:30.583+0000 7f4ba6bcf700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4ba00febb0 0x7f4ba0068c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:25:30.590 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-14T19:25:30.583+0000 7f4ba6bcf700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f4ba00ffdb0 0x7f4ba019a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:25:30.591 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-14T19:25:30.583+0000 7f4ba6bcf700 1 --2- >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f4ba0100740 0x7f4ba019ce90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:25:30.591 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-14T19:25:30.583+0000 7f4ba6bcf700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4ba019d630 con 0x7f4ba00febb0 2023-07-14T19:25:30.591 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-14T19:25:30.583+0000 7f4ba6bcf700 1 -- --> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4ba019d7a0 con 0x7f4ba0100740 2023-07-14T19:25:30.591 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-14T19:25:30.583+0000 7f4ba6bcf700 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4ba019d960 con 0x7f4ba00ffdb0 2023-07-14T19:25:30.592 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-14T19:25:30.583+0000 7f4ba516c700 1 --2- >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f4ba0100740 0x7f4ba019ce90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:25:30.592 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-14T19:25:30.583+0000 7f4ba516c700 1 --2- >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f4ba0100740 0x7f4ba019ce90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.101:3300/0 says I am v2:172.21.15.101:36096/0 (socket says 172.21.15.101:36096) 2023-07-14T19:25:30.592 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-14T19:25:30.583+0000 7f4ba516c700 1 -- 172.21.15.101:0/1371150822 learned_addr learned my addr 172.21.15.101:0/1371150822 (peer_addr_for_me v2:172.21.15.101:0/0) 2023-07-14T19:25:30.592 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-14T19:25:30.583+0000 7f4ba496b700 1 --2- 172.21.15.101:0/1371150822 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4ba00febb0 0x7f4ba0068c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:25:30.593 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-14T19:25:30.584+0000 7f4b9ffff700 1 --2- 172.21.15.101:0/1371150822 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f4ba00ffdb0 0x7f4ba019a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:25:30.593 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-14T19:25:30.584+0000 7f4ba516c700 1 -- 172.21.15.101:0/1371150822 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f4ba00ffdb0 msgr2=0x7f4ba019a840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:25:30.593 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-14T19:25:30.584+0000 7f4ba516c700 1 --2- 172.21.15.101:0/1371150822 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f4ba00ffdb0 0x7f4ba019a840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:25:30.593 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-14T19:25:30.584+0000 7f4ba516c700 1 -- 172.21.15.101:0/1371150822 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4ba00febb0 msgr2=0x7f4ba0068c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:25:30.593 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-14T19:25:30.584+0000 7f4ba516c700 1 --2- 172.21.15.101:0/1371150822 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4ba00febb0 0x7f4ba0068c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:25:30.594 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-14T19:25:30.584+0000 7f4ba516c700 1 -- 172.21.15.101:0/1371150822 --> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4b98009d00 con 0x7f4ba0100740 2023-07-14T19:25:30.594 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-14T19:25:30.584+0000 7f4ba516c700 1 --2- 172.21.15.101:0/1371150822 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f4ba0100740 0x7f4ba019ce90 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f4b9800b070 tx=0x7f4b98004a50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:25:30.594 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-14T19:25:30.584+0000 7f4b9dffb700 1 -- 172.21.15.101:0/1371150822 <== mon.2 v2:172.21.15.101:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4b9800e040 con 0x7f4ba0100740 2023-07-14T19:25:30.594 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-14T19:25:30.584+0000 7f4b9dffb700 1 -- 172.21.15.101:0/1371150822 <== mon.2 v2:172.21.15.101:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f4b98004dd0 con 0x7f4ba0100740 2023-07-14T19:25:30.595 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-14T19:25:30.584+0000 7f4b9dffb700 1 -- 172.21.15.101:0/1371150822 <== mon.2 v2:172.21.15.101:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4b9801b6b0 con 0x7f4ba0100740 2023-07-14T19:25:30.595 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-14T19:25:30.584+0000 7f4ba6bcf700 1 -- 172.21.15.101:0/1371150822 --> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4ba019dbe0 con 0x7f4ba0100740 2023-07-14T19:25:30.595 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-14T19:25:30.584+0000 7f4ba6bcf700 1 -- 172.21.15.101:0/1371150822 --> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4ba019e050 con 0x7f4ba0100740 2023-07-14T19:25:30.595 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-14T19:25:30.585+0000 7f4ba6bcf700 1 -- 172.21.15.101:0/1371150822 --> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4ba0106150 con 0x7f4ba0100740 2023-07-14T19:25:30.597 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-14T19:25:30.586+0000 7f4b9dffb700 1 -- 172.21.15.101:0/1371150822 <== mon.2 v2:172.21.15.101:3300/0 4 ==== mgrmap(e 16) v1 ==== 135715+0+0 (secure 0 0 0) 0x7f4b98003a00 con 0x7f4ba0100740 2023-07-14T19:25:30.597 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-14T19:25:30.587+0000 7f4b9dffb700 1 --2- 172.21.15.101:0/1371150822 >> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] conn(0x7f4b900a11d0 0x7f4b900a3680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:25:30.597 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-14T19:25:30.587+0000 7f4b9dffb700 1 -- 172.21.15.101:0/1371150822 <== mon.2 v2:172.21.15.101:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 3698+0+0 (secure 0 0 0) 0x7f4b980ccb00 con 0x7f4ba0100740 2023-07-14T19:25:30.598 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-14T19:25:30.587+0000 7f4ba496b700 1 --2- 172.21.15.101:0/1371150822 >> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] conn(0x7f4b900a11d0 0x7f4b900a3680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:25:30.598 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-14T19:25:30.588+0000 7f4ba496b700 1 --2- 172.21.15.101:0/1371150822 >> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] conn(0x7f4b900a11d0 0x7f4b900a3680 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f4ba00ffc10 tx=0x7f4b9400a380 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:25:30.598 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-14T19:25:30.591+0000 7f4b9dffb700 1 -- 172.21.15.101:0/1371150822 <== mon.2 v2:172.21.15.101:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f4b980974e0 con 0x7f4ba0100740 2023-07-14T19:25:30.691 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:25:30 smithi101 ceph-mon[128807]: purged_snaps scrub starts 2023-07-14T19:25:30.691 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:25:30 smithi101 ceph-mon[128807]: purged_snaps scrub ok 2023-07-14T19:25:30.692 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:25:30 smithi101 ceph-mon[128807]: osd.3 [v2:172.21.15.98:6810/3095249665,v1:172.21.15.98:6811/3095249665] boot 2023-07-14T19:25:30.692 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:25:30 smithi101 ceph-mon[128807]: osdmap e25: 4 total, 4 up, 4 in 2023-07-14T19:25:30.692 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:25:30 smithi101 ceph-mon[128807]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2023-07-14T19:25:30.692 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:25:30 smithi101 ceph-mon[128807]: pgmap v89: 1 pgs: 1 active+clean; 577 KiB data, 109 MiB used, 358 GiB / 358 GiB avail 2023-07-14T19:25:30.720 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:25:30 smithi139 ceph-mon[129293]: purged_snaps scrub starts 2023-07-14T19:25:30.721 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:25:30 smithi139 ceph-mon[129293]: purged_snaps scrub ok 2023-07-14T19:25:30.721 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:25:30 smithi139 ceph-mon[129293]: osd.3 [v2:172.21.15.98:6810/3095249665,v1:172.21.15.98:6811/3095249665] boot 2023-07-14T19:25:30.721 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:25:30 smithi139 ceph-mon[129293]: osdmap e25: 4 total, 4 up, 4 in 2023-07-14T19:25:30.721 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:25:30 smithi139 ceph-mon[129293]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2023-07-14T19:25:30.721 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:25:30 smithi139 ceph-mon[129293]: pgmap v89: 1 pgs: 1 active+clean; 577 KiB data, 109 MiB used, 358 GiB / 358 GiB avail 2023-07-14T19:25:30.781 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:25:30 smithi098 ceph-mon[127486]: purged_snaps scrub starts 2023-07-14T19:25:30.781 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:25:30 smithi098 ceph-mon[127486]: purged_snaps scrub ok 2023-07-14T19:25:30.781 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:25:30 smithi098 ceph-mon[127486]: osd.3 [v2:172.21.15.98:6810/3095249665,v1:172.21.15.98:6811/3095249665] boot 2023-07-14T19:25:30.782 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:25:30 smithi098 ceph-mon[127486]: osdmap e25: 4 total, 4 up, 4 in 2023-07-14T19:25:30.782 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:25:30 smithi098 ceph-mon[127486]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2023-07-14T19:25:30.782 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:25:30 smithi098 ceph-mon[127486]: pgmap v89: 1 pgs: 1 active+clean; 577 KiB data, 109 MiB used, 358 GiB / 358 GiB avail 2023-07-14T19:25:30.787 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-14T19:25:30.785+0000 7f4ba6bcf700 1 -- 172.21.15.101:0/1371150822 --> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi101:vg_nvme/lv_3", "target": ["mon-mgr", ""]}) v1 -- 0x7f4ba0060a80 con 0x7f4b900a11d0 2023-07-14T19:25:31.641 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:25:31 smithi101 ceph-mon[128807]: osdmap e26: 4 total, 4 up, 4 in 2023-07-14T19:25:31.641 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:25:31 smithi101 ceph-mon[128807]: from='client.24215 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi101:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2023-07-14T19:25:31.641 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:25:31 smithi101 ceph-mon[128807]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-07-14T19:25:31.641 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:25:31 smithi101 ceph-mon[128807]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-07-14T19:25:31.642 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:25:31 smithi101 ceph-mon[128807]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T19:25:31.642 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:25:31 smithi101 ceph-mon[128807]: osdmap e27: 4 total, 4 up, 4 in 2023-07-14T19:25:31.720 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:25:31 smithi139 ceph-mon[129293]: osdmap e26: 4 total, 4 up, 4 in 2023-07-14T19:25:31.721 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:25:31 smithi139 ceph-mon[129293]: from='client.24215 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi101:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2023-07-14T19:25:31.721 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:25:31 smithi139 ceph-mon[129293]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-07-14T19:25:31.721 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:25:31 smithi139 ceph-mon[129293]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-07-14T19:25:31.721 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:25:31 smithi139 ceph-mon[129293]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T19:25:31.721 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:25:31 smithi139 ceph-mon[129293]: osdmap e27: 4 total, 4 up, 4 in 2023-07-14T19:25:31.781 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:25:31 smithi098 ceph-mon[127486]: osdmap e26: 4 total, 4 up, 4 in 2023-07-14T19:25:31.782 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:25:31 smithi098 ceph-mon[127486]: from='client.24215 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi101:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2023-07-14T19:25:31.782 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:25:31 smithi098 ceph-mon[127486]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-07-14T19:25:31.782 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:25:31 smithi098 ceph-mon[127486]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-07-14T19:25:31.783 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:25:31 smithi098 ceph-mon[127486]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T19:25:31.783 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:25:31 smithi098 ceph-mon[127486]: osdmap e27: 4 total, 4 up, 4 in 2023-07-14T19:25:32.691 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:25:32 smithi101 ceph-mon[128807]: pgmap v92: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2023-07-14T19:25:32.720 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:25:32 smithi139 ceph-mon[129293]: pgmap v92: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2023-07-14T19:25:32.781 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:25:32 smithi098 ceph-mon[127486]: pgmap v92: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2023-07-14T19:25:33.691 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:25:33 smithi101 ceph-mon[128807]: osdmap e28: 4 total, 4 up, 4 in 2023-07-14T19:25:33.691 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:25:33 smithi101 ceph-mon[128807]: from='client.? 172.21.15.101:0/757214237' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "615d4642-a997-4f95-9e38-9fbdaa831400"}]: dispatch 2023-07-14T19:25:33.692 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:25:33 smithi101 ceph-mon[128807]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "615d4642-a997-4f95-9e38-9fbdaa831400"}]: dispatch 2023-07-14T19:25:33.692 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:25:33 smithi101 ceph-mon[128807]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "615d4642-a997-4f95-9e38-9fbdaa831400"}]': finished 2023-07-14T19:25:33.692 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:25:33 smithi101 ceph-mon[128807]: osdmap e29: 5 total, 4 up, 5 in 2023-07-14T19:25:33.692 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:25:33 smithi101 ceph-mon[128807]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2023-07-14T19:25:33.720 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:25:33 smithi139 ceph-mon[129293]: osdmap e28: 4 total, 4 up, 4 in 2023-07-14T19:25:33.721 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:25:33 smithi139 ceph-mon[129293]: from='client.? 172.21.15.101:0/757214237' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "615d4642-a997-4f95-9e38-9fbdaa831400"}]: dispatch 2023-07-14T19:25:33.721 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:25:33 smithi139 ceph-mon[129293]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "615d4642-a997-4f95-9e38-9fbdaa831400"}]: dispatch 2023-07-14T19:25:33.721 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:25:33 smithi139 ceph-mon[129293]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "615d4642-a997-4f95-9e38-9fbdaa831400"}]': finished 2023-07-14T19:25:33.721 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:25:33 smithi139 ceph-mon[129293]: osdmap e29: 5 total, 4 up, 5 in 2023-07-14T19:25:33.721 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:25:33 smithi139 ceph-mon[129293]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2023-07-14T19:25:33.781 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:25:33 smithi098 ceph-mon[127486]: osdmap e28: 4 total, 4 up, 4 in 2023-07-14T19:25:33.782 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:25:33 smithi098 ceph-mon[127486]: from='client.? 172.21.15.101:0/757214237' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "615d4642-a997-4f95-9e38-9fbdaa831400"}]: dispatch 2023-07-14T19:25:33.782 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:25:33 smithi098 ceph-mon[127486]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "615d4642-a997-4f95-9e38-9fbdaa831400"}]: dispatch 2023-07-14T19:25:33.782 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:25:33 smithi098 ceph-mon[127486]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "615d4642-a997-4f95-9e38-9fbdaa831400"}]': finished 2023-07-14T19:25:33.783 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:25:33 smithi098 ceph-mon[127486]: osdmap e29: 5 total, 4 up, 5 in 2023-07-14T19:25:33.783 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:25:33 smithi098 ceph-mon[127486]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2023-07-14T19:25:34.691 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:25:34 smithi101 ceph-mon[128807]: pgmap v95: 1 pgs: 1 peering; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail; 144 KiB/s, 0 objects/s recovering 2023-07-14T19:25:34.692 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:25:34 smithi101 ceph-mon[128807]: from='client.? 172.21.15.101:0/3193765560' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-07-14T19:25:34.720 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:25:34 smithi139 ceph-mon[129293]: pgmap v95: 1 pgs: 1 peering; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail; 144 KiB/s, 0 objects/s recovering 2023-07-14T19:25:34.721 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:25:34 smithi139 ceph-mon[129293]: from='client.? 172.21.15.101:0/3193765560' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-07-14T19:25:34.781 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:25:34 smithi098 ceph-mon[127486]: pgmap v95: 1 pgs: 1 peering; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail; 144 KiB/s, 0 objects/s recovering 2023-07-14T19:25:34.782 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:25:34 smithi098 ceph-mon[127486]: from='client.? 172.21.15.101:0/3193765560' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-07-14T19:25:36.941 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:25:36 smithi101 ceph-mon[128807]: pgmap v96: 1 pgs: 1 peering; 577 KiB data, 109 MiB used, 358 GiB / 358 GiB avail; 111 KiB/s, 0 objects/s recovering 2023-07-14T19:25:36.970 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:25:36 smithi139 ceph-mon[129293]: pgmap v96: 1 pgs: 1 peering; 577 KiB data, 109 MiB used, 358 GiB / 358 GiB avail; 111 KiB/s, 0 objects/s recovering 2023-07-14T19:25:37.031 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:25:36 smithi098 ceph-mon[127486]: pgmap v96: 1 pgs: 1 peering; 577 KiB data, 109 MiB used, 358 GiB / 358 GiB avail; 111 KiB/s, 0 objects/s recovering 2023-07-14T19:25:38.941 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:25:38 smithi101 ceph-mon[128807]: pgmap v97: 1 pgs: 1 peering; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail; 93 KiB/s, 0 objects/s recovering 2023-07-14T19:25:38.970 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:25:38 smithi139 ceph-mon[129293]: pgmap v97: 1 pgs: 1 peering; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail; 93 KiB/s, 0 objects/s recovering 2023-07-14T19:25:39.031 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:25:38 smithi098 ceph-mon[127486]: pgmap v97: 1 pgs: 1 peering; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail; 93 KiB/s, 0 objects/s recovering 2023-07-14T19:25:40.879 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:25:40 smithi101 ceph-mon[128807]: pgmap v98: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail; 72 KiB/s, 0 objects/s recovering 2023-07-14T19:25:40.880 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:25:40 smithi101 ceph-mon[128807]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2023-07-14T19:25:40.880 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:25:40 smithi101 ceph-mon[128807]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T19:25:40.970 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:25:40 smithi139 ceph-mon[129293]: pgmap v98: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail; 72 KiB/s, 0 objects/s recovering 2023-07-14T19:25:40.971 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:25:40 smithi139 ceph-mon[129293]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2023-07-14T19:25:40.971 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:25:40 smithi139 ceph-mon[129293]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T19:25:41.031 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:25:40 smithi098 ceph-mon[127486]: pgmap v98: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail; 72 KiB/s, 0 objects/s recovering 2023-07-14T19:25:41.031 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:25:40 smithi098 ceph-mon[127486]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2023-07-14T19:25:41.032 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:25:40 smithi098 ceph-mon[127486]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T19:25:41.890 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:25:41 smithi101 ceph-mon[128807]: Deploying daemon osd.4 on smithi101 2023-07-14T19:25:41.970 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:25:41 smithi139 ceph-mon[129293]: Deploying daemon osd.4 on smithi101 2023-07-14T19:25:42.031 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:25:41 smithi098 ceph-mon[127486]: Deploying daemon osd.4 on smithi101 2023-07-14T19:25:42.691 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:25:42 smithi101 ceph-mon[128807]: pgmap v99: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail; 63 KiB/s, 0 objects/s recovering 2023-07-14T19:25:42.970 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:25:42 smithi139 ceph-mon[129293]: pgmap v99: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail; 63 KiB/s, 0 objects/s recovering 2023-07-14T19:25:43.031 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:25:42 smithi098 ceph-mon[127486]: pgmap v99: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail; 63 KiB/s, 0 objects/s recovering 2023-07-14T19:25:44.910 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:25:44 smithi101 ceph-mon[128807]: pgmap v100: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2023-07-14T19:25:44.911 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:25:44 smithi101 ceph-mon[128807]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' 2023-07-14T19:25:44.911 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:25:44 smithi101 ceph-mon[128807]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' 2023-07-14T19:25:44.911 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:25:44 smithi101 ceph-mon[128807]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-14T19:25:44.970 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:25:44 smithi139 ceph-mon[129293]: pgmap v100: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2023-07-14T19:25:44.970 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:25:44 smithi139 ceph-mon[129293]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' 2023-07-14T19:25:44.971 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:25:44 smithi139 ceph-mon[129293]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' 2023-07-14T19:25:44.971 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:25:44 smithi139 ceph-mon[129293]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-14T19:25:45.031 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:25:44 smithi098 ceph-mon[127486]: pgmap v100: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2023-07-14T19:25:45.031 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:25:44 smithi098 ceph-mon[127486]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' 2023-07-14T19:25:45.032 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:25:44 smithi098 ceph-mon[127486]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' 2023-07-14T19:25:45.032 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:25:44 smithi098 ceph-mon[127486]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-14T19:25:46.895 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:25:46 smithi101 ceph-mon[128807]: pgmap v101: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2023-07-14T19:25:46.970 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:25:46 smithi139 ceph-mon[129293]: pgmap v101: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2023-07-14T19:25:47.031 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:25:46 smithi098 ceph-mon[127486]: pgmap v101: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2023-07-14T19:25:47.195 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-14T19:25:47.194+0000 7f4b9dffb700 1 -- 172.21.15.101:0/1371150822 <== mgr.14152 v2:172.21.15.98:6800/452042164 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7f4ba0060a80 con 0x7f4b900a11d0 2023-07-14T19:25:47.195 INFO:teuthology.orchestra.run.smithi101.stdout:Created osd(s) 4 on host 'smithi101' 2023-07-14T19:25:47.197 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-14T19:25:47.196+0000 7f4ba6bcf700 1 -- 172.21.15.101:0/1371150822 >> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] conn(0x7f4b900a11d0 msgr2=0x7f4b900a3680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:25:47.197 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-14T19:25:47.196+0000 7f4ba6bcf700 1 --2- 172.21.15.101:0/1371150822 >> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] conn(0x7f4b900a11d0 0x7f4b900a3680 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f4ba00ffc10 tx=0x7f4b9400a380 comp rx=0 tx=0).stop 2023-07-14T19:25:47.198 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-14T19:25:47.196+0000 7f4ba6bcf700 1 -- 172.21.15.101:0/1371150822 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f4ba0100740 msgr2=0x7f4ba019ce90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:25:47.198 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-14T19:25:47.196+0000 7f4ba6bcf700 1 --2- 172.21.15.101:0/1371150822 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f4ba0100740 0x7f4ba019ce90 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f4b9800b070 tx=0x7f4b98004a50 comp rx=0 tx=0).stop 2023-07-14T19:25:47.198 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-14T19:25:47.196+0000 7f4ba6bcf700 1 -- 172.21.15.101:0/1371150822 shutdown_connections 2023-07-14T19:25:47.198 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-14T19:25:47.196+0000 7f4ba6bcf700 1 --2- 172.21.15.101:0/1371150822 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f4ba0100740 0x7f4ba019ce90 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:25:47.198 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-14T19:25:47.196+0000 7f4ba6bcf700 1 --2- 172.21.15.101:0/1371150822 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4ba00febb0 0x7f4ba0068c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:25:47.198 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-14T19:25:47.196+0000 7f4ba6bcf700 1 --2- 172.21.15.101:0/1371150822 >> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] conn(0x7f4b900a11d0 0x7f4b900a3680 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:25:47.198 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-14T19:25:47.196+0000 7f4ba6bcf700 1 --2- 172.21.15.101:0/1371150822 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f4ba00ffdb0 0x7f4ba019a840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:25:47.199 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-14T19:25:47.196+0000 7f4ba6bcf700 1 -- 172.21.15.101:0/1371150822 >> 172.21.15.101:0/1371150822 conn(0x7f4ba00fa1e0 msgr2=0x7f4ba0103290 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:25:47.199 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-14T19:25:47.197+0000 7f4ba6bcf700 1 -- 172.21.15.101:0/1371150822 shutdown_connections 2023-07-14T19:25:47.199 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-14T19:25:47.197+0000 7f4ba6bcf700 1 -- 172.21.15.101:0/1371150822 wait complete. 2023-07-14T19:25:47.698 DEBUG:teuthology.orchestra.run.smithi101:osd.4> sudo journalctl -f -n 0 -u ceph-9adea648-227b-11ee-9b33-001a4aab830c@osd.4.service 2023-07-14T19:25:47.701 INFO:tasks.cephadm:Deploying osd.5 on smithi139 with /dev/vg_nvme/lv_3... 2023-07-14T19:25:47.702 DEBUG:teuthology.orchestra.run.smithi139:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81c9387d2e7576da065747ec9c16d1c7d69016ec shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 9adea648-227b-11ee-9b33-001a4aab830c -- ceph-volume lvm zap /dev/vg_nvme/lv_3 2023-07-14T19:25:47.735 INFO:journalctl@ceph.osd.4.smithi101.stdout:-- Logs begin at Fri 2023-07-14 18:56:58 UTC. -- 2023-07-14T19:25:47.941 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:25:47 smithi101 ceph-mon[128807]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' 2023-07-14T19:25:47.941 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:25:47 smithi101 ceph-mon[128807]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' 2023-07-14T19:25:47.942 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:25:47 smithi101 ceph-mon[128807]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T19:25:47.942 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:25:47 smithi101 ceph-mon[128807]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-14T19:25:47.942 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:25:47 smithi101 ceph-mon[128807]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' 2023-07-14T19:25:47.942 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:25:47 smithi101 ceph-mon[128807]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' 2023-07-14T19:25:47.942 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:25:47 smithi101 ceph-mon[128807]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' 2023-07-14T19:25:47.942 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:25:47 smithi101 ceph-mon[128807]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-14T19:25:48.031 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:25:47 smithi098 ceph-mon[127486]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' 2023-07-14T19:25:48.031 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:25:47 smithi098 ceph-mon[127486]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' 2023-07-14T19:25:48.031 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:25:47 smithi098 ceph-mon[127486]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T19:25:48.032 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:25:47 smithi098 ceph-mon[127486]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-14T19:25:48.032 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:25:47 smithi098 ceph-mon[127486]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' 2023-07-14T19:25:48.032 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:25:47 smithi098 ceph-mon[127486]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' 2023-07-14T19:25:48.032 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:25:47 smithi098 ceph-mon[127486]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' 2023-07-14T19:25:48.032 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:25:47 smithi098 ceph-mon[127486]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-14T19:25:48.220 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:25:47 smithi139 ceph-mon[129293]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' 2023-07-14T19:25:48.220 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:25:47 smithi139 ceph-mon[129293]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' 2023-07-14T19:25:48.221 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:25:47 smithi139 ceph-mon[129293]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T19:25:48.221 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:25:47 smithi139 ceph-mon[129293]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-14T19:25:48.221 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:25:47 smithi139 ceph-mon[129293]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' 2023-07-14T19:25:48.221 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:25:47 smithi139 ceph-mon[129293]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' 2023-07-14T19:25:48.221 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:25:47 smithi139 ceph-mon[129293]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' 2023-07-14T19:25:48.222 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:25:47 smithi139 ceph-mon[129293]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-14T19:25:48.266 INFO:teuthology.orchestra.run.smithi139.stderr:Inferring config /var/lib/ceph/9adea648-227b-11ee-9b33-001a4aab830c/mon.c/config 2023-07-14T19:25:48.691 INFO:journalctl@ceph.osd.4.smithi101.stdout:Jul 14 19:25:48 smithi101 ceph-9adea648-227b-11ee-9b33-001a4aab830c-osd-4[139397]: 2023-07-14T19:25:48.440+0000 7f3f10fa3640 -1 osd.4 0 log_to_monitors true 2023-07-14T19:25:48.747 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:25:48 smithi139 ceph-mon[129293]: pgmap v102: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2023-07-14T19:25:48.747 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:25:48 smithi139 ceph-mon[129293]: from='osd.4 [v2:172.21.15.101:6808/2844761406,v1:172.21.15.101:6809/2844761406]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2023-07-14T19:25:48.747 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:25:48 smithi139 ceph-mon[129293]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2023-07-14T19:25:49.011 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:25:48 smithi101 ceph-mon[128807]: pgmap v102: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2023-07-14T19:25:49.011 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:25:48 smithi101 ceph-mon[128807]: from='osd.4 [v2:172.21.15.101:6808/2844761406,v1:172.21.15.101:6809/2844761406]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2023-07-14T19:25:49.012 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:25:48 smithi101 ceph-mon[128807]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2023-07-14T19:25:49.031 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:25:48 smithi098 ceph-mon[127486]: pgmap v102: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2023-07-14T19:25:49.031 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:25:48 smithi098 ceph-mon[127486]: from='osd.4 [v2:172.21.15.101:6808/2844761406,v1:172.21.15.101:6809/2844761406]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2023-07-14T19:25:49.032 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:25:48 smithi098 ceph-mon[127486]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2023-07-14T19:25:49.749 INFO:teuthology.orchestra.run.smithi139.stderr:--> Zapping: /dev/vg_nvme/lv_3 2023-07-14T19:25:49.749 INFO:teuthology.orchestra.run.smithi139.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 conv=fsync 2023-07-14T19:25:49.749 INFO:teuthology.orchestra.run.smithi139.stderr: stderr: 10+0 records in 2023-07-14T19:25:49.750 INFO:teuthology.orchestra.run.smithi139.stderr:10+0 records out 2023-07-14T19:25:49.750 INFO:teuthology.orchestra.run.smithi139.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0109975 s, 953 MB/s 2023-07-14T19:25:49.750 INFO:teuthology.orchestra.run.smithi139.stderr:--> Zapping successful for: 2023-07-14T19:25:50.107 DEBUG:teuthology.orchestra.run.smithi139:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81c9387d2e7576da065747ec9c16d1c7d69016ec shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 9adea648-227b-11ee-9b33-001a4aab830c -- ceph orch daemon add osd smithi139:vg_nvme/lv_3 2023-07-14T19:25:50.442 INFO:journalctl@ceph.osd.4.smithi101.stdout:Jul 14 19:25:50 smithi101 ceph-9adea648-227b-11ee-9b33-001a4aab830c-osd-4[139397]: 2023-07-14T19:25:50.170+0000 7f3f05e1a700 -1 osd.4 0 waiting for initial osdmap 2023-07-14T19:25:50.442 INFO:journalctl@ceph.osd.4.smithi101.stdout:Jul 14 19:25:50 smithi101 ceph-9adea648-227b-11ee-9b33-001a4aab830c-osd-4[139397]: 2023-07-14T19:25:50.176+0000 7f3efe409700 -1 osd.4 31 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-07-14T19:25:50.442 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:25:50 smithi101 ceph-mon[128807]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2023-07-14T19:25:50.443 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:25:50 smithi101 ceph-mon[128807]: osdmap e30: 5 total, 4 up, 5 in 2023-07-14T19:25:50.443 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:25:50 smithi101 ceph-mon[128807]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2023-07-14T19:25:50.443 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:25:50 smithi101 ceph-mon[128807]: from='osd.4 [v2:172.21.15.101:6808/2844761406,v1:172.21.15.101:6809/2844761406]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi101", "root=default"]}]: dispatch 2023-07-14T19:25:50.443 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:25:50 smithi101 ceph-mon[128807]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi101", "root=default"]}]: dispatch 2023-07-14T19:25:50.443 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:25:50 smithi101 ceph-mon[128807]: pgmap v104: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2023-07-14T19:25:50.443 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:25:50 smithi101 ceph-mon[128807]: Detected new or changed devices on smithi101 2023-07-14T19:25:50.443 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:25:50 smithi101 ceph-mon[128807]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' 2023-07-14T19:25:50.444 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:25:50 smithi101 ceph-mon[128807]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' 2023-07-14T19:25:50.444 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:25:50 smithi101 ceph-mon[128807]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2023-07-14T19:25:50.444 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:25:50 smithi101 ceph-mon[128807]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2023-07-14T19:25:50.444 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:25:50 smithi101 ceph-mon[128807]: Adjusting osd_memory_target on smithi101 to 8584M 2023-07-14T19:25:50.444 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:25:50 smithi101 ceph-mon[128807]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' 2023-07-14T19:25:50.444 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:25:50 smithi101 ceph-mon[128807]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T19:25:50.445 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:25:50 smithi101 ceph-mon[128807]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-14T19:25:50.445 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:25:50 smithi101 ceph-mon[128807]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' 2023-07-14T19:25:50.470 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:25:50 smithi139 ceph-mon[129293]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2023-07-14T19:25:50.471 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:25:50 smithi139 ceph-mon[129293]: osdmap e30: 5 total, 4 up, 5 in 2023-07-14T19:25:50.471 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:25:50 smithi139 ceph-mon[129293]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2023-07-14T19:25:50.471 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:25:50 smithi139 ceph-mon[129293]: from='osd.4 [v2:172.21.15.101:6808/2844761406,v1:172.21.15.101:6809/2844761406]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi101", "root=default"]}]: dispatch 2023-07-14T19:25:50.471 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:25:50 smithi139 ceph-mon[129293]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi101", "root=default"]}]: dispatch 2023-07-14T19:25:50.472 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:25:50 smithi139 ceph-mon[129293]: pgmap v104: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2023-07-14T19:25:50.472 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:25:50 smithi139 ceph-mon[129293]: Detected new or changed devices on smithi101 2023-07-14T19:25:50.472 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:25:50 smithi139 ceph-mon[129293]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' 2023-07-14T19:25:50.472 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:25:50 smithi139 ceph-mon[129293]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' 2023-07-14T19:25:50.472 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:25:50 smithi139 ceph-mon[129293]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2023-07-14T19:25:50.472 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:25:50 smithi139 ceph-mon[129293]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2023-07-14T19:25:50.472 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:25:50 smithi139 ceph-mon[129293]: Adjusting osd_memory_target on smithi101 to 8584M 2023-07-14T19:25:50.473 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:25:50 smithi139 ceph-mon[129293]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' 2023-07-14T19:25:50.473 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:25:50 smithi139 ceph-mon[129293]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T19:25:50.473 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:25:50 smithi139 ceph-mon[129293]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-14T19:25:50.473 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:25:50 smithi139 ceph-mon[129293]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' 2023-07-14T19:25:50.531 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:25:50 smithi098 ceph-mon[127486]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2023-07-14T19:25:50.532 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:25:50 smithi098 ceph-mon[127486]: osdmap e30: 5 total, 4 up, 5 in 2023-07-14T19:25:50.532 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:25:50 smithi098 ceph-mon[127486]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2023-07-14T19:25:50.532 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:25:50 smithi098 ceph-mon[127486]: from='osd.4 [v2:172.21.15.101:6808/2844761406,v1:172.21.15.101:6809/2844761406]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi101", "root=default"]}]: dispatch 2023-07-14T19:25:50.533 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:25:50 smithi098 ceph-mon[127486]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi101", "root=default"]}]: dispatch 2023-07-14T19:25:50.533 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:25:50 smithi098 ceph-mon[127486]: pgmap v104: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2023-07-14T19:25:50.533 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:25:50 smithi098 ceph-mon[127486]: Detected new or changed devices on smithi101 2023-07-14T19:25:50.533 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:25:50 smithi098 ceph-mon[127486]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' 2023-07-14T19:25:50.533 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:25:50 smithi098 ceph-mon[127486]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' 2023-07-14T19:25:50.533 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:25:50 smithi098 ceph-mon[127486]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2023-07-14T19:25:50.533 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:25:50 smithi098 ceph-mon[127486]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2023-07-14T19:25:50.533 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:25:50 smithi098 ceph-mon[127486]: Adjusting osd_memory_target on smithi101 to 8584M 2023-07-14T19:25:50.534 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:25:50 smithi098 ceph-mon[127486]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' 2023-07-14T19:25:50.534 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:25:50 smithi098 ceph-mon[127486]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T19:25:50.534 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:25:50 smithi098 ceph-mon[127486]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-14T19:25:50.534 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:25:50 smithi098 ceph-mon[127486]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' 2023-07-14T19:25:50.659 INFO:teuthology.orchestra.run.smithi139.stderr:Inferring config /var/lib/ceph/9adea648-227b-11ee-9b33-001a4aab830c/mon.c/config 2023-07-14T19:25:51.441 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:25:51 smithi101 ceph-mon[128807]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi101", "root=default"]}]': finished 2023-07-14T19:25:51.441 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:25:51 smithi101 ceph-mon[128807]: osdmap e31: 5 total, 4 up, 5 in 2023-07-14T19:25:51.442 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:25:51 smithi101 ceph-mon[128807]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2023-07-14T19:25:51.442 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:25:51 smithi101 ceph-mon[128807]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2023-07-14T19:25:51.471 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:25:51 smithi139 ceph-mon[129293]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi101", "root=default"]}]': finished 2023-07-14T19:25:51.471 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:25:51 smithi139 ceph-mon[129293]: osdmap e31: 5 total, 4 up, 5 in 2023-07-14T19:25:51.471 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:25:51 smithi139 ceph-mon[129293]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2023-07-14T19:25:51.471 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:25:51 smithi139 ceph-mon[129293]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2023-07-14T19:25:51.531 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:25:51 smithi098 ceph-mon[127486]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi101", "root=default"]}]': finished 2023-07-14T19:25:51.532 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:25:51 smithi098 ceph-mon[127486]: osdmap e31: 5 total, 4 up, 5 in 2023-07-14T19:25:51.532 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:25:51 smithi098 ceph-mon[127486]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2023-07-14T19:25:51.532 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:25:51 smithi098 ceph-mon[127486]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2023-07-14T19:25:51.994 INFO:teuthology.orchestra.run.smithi139.stderr:2023-07-14T19:25:51.992+0000 7f8346cef700 1 -- 172.21.15.139:0/580033198 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f83400ff7c0 msgr2=0x7f8340107070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:25:51.994 INFO:teuthology.orchestra.run.smithi139.stderr:2023-07-14T19:25:51.992+0000 7f8346cef700 1 --2- 172.21.15.139:0/580033198 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f83400ff7c0 0x7f8340107070 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f833800b0a0 tx=0x7f833800b3b0 comp rx=0 tx=0).stop 2023-07-14T19:25:51.994 INFO:teuthology.orchestra.run.smithi139.stderr:2023-07-14T19:25:51.992+0000 7f8346cef700 1 -- 172.21.15.139:0/580033198 shutdown_connections 2023-07-14T19:25:51.994 INFO:teuthology.orchestra.run.smithi139.stderr:2023-07-14T19:25:51.992+0000 7f8346cef700 1 --2- 172.21.15.139:0/580033198 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f83400ff7c0 0x7f8340107070 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:25:51.994 INFO:teuthology.orchestra.run.smithi139.stderr:2023-07-14T19:25:51.992+0000 7f8346cef700 1 --2- 172.21.15.139:0/580033198 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8340100240 0x7f8340100650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:25:51.995 INFO:teuthology.orchestra.run.smithi139.stderr:2023-07-14T19:25:51.992+0000 7f8346cef700 1 --2- 172.21.15.139:0/580033198 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f83400fee10 0x7f83400ff280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:25:51.995 INFO:teuthology.orchestra.run.smithi139.stderr:2023-07-14T19:25:51.992+0000 7f8346cef700 1 -- 172.21.15.139:0/580033198 >> 172.21.15.139:0/580033198 conn(0x7f83400f9990 msgr2=0x7f83400fbdc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:25:51.995 INFO:teuthology.orchestra.run.smithi139.stderr:2023-07-14T19:25:51.992+0000 7f8346cef700 1 -- 172.21.15.139:0/580033198 shutdown_connections 2023-07-14T19:25:51.995 INFO:teuthology.orchestra.run.smithi139.stderr:2023-07-14T19:25:51.992+0000 7f8346cef700 1 -- 172.21.15.139:0/580033198 wait complete. 2023-07-14T19:25:51.995 INFO:teuthology.orchestra.run.smithi139.stderr:2023-07-14T19:25:51.992+0000 7f8346cef700 1 Processor -- start 2023-07-14T19:25:51.995 INFO:teuthology.orchestra.run.smithi139.stderr:2023-07-14T19:25:51.992+0000 7f8346cef700 1 -- start start 2023-07-14T19:25:51.998 INFO:teuthology.orchestra.run.smithi139.stderr:2023-07-14T19:25:51.993+0000 7f8346cef700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f83400fee10 0x7f8340194980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:25:51.998 INFO:teuthology.orchestra.run.smithi139.stderr:2023-07-14T19:25:51.993+0000 7f8346cef700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f83400ff7c0 0x7f8340194ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:25:51.998 INFO:teuthology.orchestra.run.smithi139.stderr:2023-07-14T19:25:51.993+0000 7f8346cef700 1 --2- >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f8340100240 0x7f834019bf40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:25:51.998 INFO:teuthology.orchestra.run.smithi139.stderr:2023-07-14T19:25:51.993+0000 7f8346cef700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f834019c510 con 0x7f83400ff7c0 2023-07-14T19:25:51.998 INFO:teuthology.orchestra.run.smithi139.stderr:2023-07-14T19:25:51.993+0000 7f8346cef700 1 -- --> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f834019c650 con 0x7f8340100240 2023-07-14T19:25:51.999 INFO:teuthology.orchestra.run.smithi139.stderr:2023-07-14T19:25:51.993+0000 7f8346cef700 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f834019c7e0 con 0x7f83400fee10 2023-07-14T19:25:51.999 INFO:teuthology.orchestra.run.smithi139.stderr:2023-07-14T19:25:51.993+0000 7f8344a8b700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f83400fee10 0x7f8340194980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:25:51.999 INFO:teuthology.orchestra.run.smithi139.stderr:2023-07-14T19:25:51.993+0000 7f8344a8b700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f83400fee10 0x7f8340194980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.139:3300/0 says I am v2:172.21.15.139:37172/0 (socket says 172.21.15.139:37172) 2023-07-14T19:25:51.999 INFO:teuthology.orchestra.run.smithi139.stderr:2023-07-14T19:25:51.993+0000 7f8344a8b700 1 -- 172.21.15.139:0/1413382702 learned_addr learned my addr 172.21.15.139:0/1413382702 (peer_addr_for_me v2:172.21.15.139:0/0) 2023-07-14T19:25:51.999 INFO:teuthology.orchestra.run.smithi139.stderr:2023-07-14T19:25:51.993+0000 7f833ffff700 1 --2- 172.21.15.139:0/1413382702 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f83400ff7c0 0x7f8340194ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:25:51.999 INFO:teuthology.orchestra.run.smithi139.stderr:2023-07-14T19:25:51.993+0000 7f834528c700 1 --2- 172.21.15.139:0/1413382702 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f8340100240 0x7f834019bf40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:25:51.999 INFO:teuthology.orchestra.run.smithi139.stderr:2023-07-14T19:25:51.993+0000 7f8344a8b700 1 -- 172.21.15.139:0/1413382702 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f8340100240 msgr2=0x7f834019bf40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:25:51.999 INFO:teuthology.orchestra.run.smithi139.stderr:2023-07-14T19:25:51.993+0000 7f8344a8b700 1 --2- 172.21.15.139:0/1413382702 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f8340100240 0x7f834019bf40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:25:52.000 INFO:teuthology.orchestra.run.smithi139.stderr:2023-07-14T19:25:51.993+0000 7f8344a8b700 1 -- 172.21.15.139:0/1413382702 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f83400ff7c0 msgr2=0x7f8340194ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:25:52.000 INFO:teuthology.orchestra.run.smithi139.stderr:2023-07-14T19:25:51.993+0000 7f8344a8b700 1 --2- 172.21.15.139:0/1413382702 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f83400ff7c0 0x7f8340194ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:25:52.000 INFO:teuthology.orchestra.run.smithi139.stderr:2023-07-14T19:25:51.993+0000 7f8344a8b700 1 -- 172.21.15.139:0/1413382702 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8338009d00 con 0x7f83400fee10 2023-07-14T19:25:52.000 INFO:teuthology.orchestra.run.smithi139.stderr:2023-07-14T19:25:51.993+0000 7f834528c700 1 --2- 172.21.15.139:0/1413382702 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f8340100240 0x7f834019bf40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-07-14T19:25:52.000 INFO:teuthology.orchestra.run.smithi139.stderr:2023-07-14T19:25:51.993+0000 7f8344a8b700 1 --2- 172.21.15.139:0/1413382702 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f83400fee10 0x7f8340194980 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f833400eae0 tx=0x7f833400edf0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:25:52.000 INFO:teuthology.orchestra.run.smithi139.stderr:2023-07-14T19:25:51.993+0000 7f833dffb700 1 -- 172.21.15.139:0/1413382702 <== mon.1 v2:172.21.15.139:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8334013070 con 0x7f83400fee10 2023-07-14T19:25:52.000 INFO:teuthology.orchestra.run.smithi139.stderr:2023-07-14T19:25:51.993+0000 7f8346cef700 1 -- 172.21.15.139:0/1413382702 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f83400fdd60 con 0x7f83400fee10 2023-07-14T19:25:52.001 INFO:teuthology.orchestra.run.smithi139.stderr:2023-07-14T19:25:51.993+0000 7f833dffb700 1 -- 172.21.15.139:0/1413382702 <== mon.1 v2:172.21.15.139:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f8334017410 con 0x7f83400fee10 2023-07-14T19:25:52.001 INFO:teuthology.orchestra.run.smithi139.stderr:2023-07-14T19:25:51.993+0000 7f833dffb700 1 -- 172.21.15.139:0/1413382702 <== mon.1 v2:172.21.15.139:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8334009790 con 0x7f83400fee10 2023-07-14T19:25:52.001 INFO:teuthology.orchestra.run.smithi139.stderr:2023-07-14T19:25:51.994+0000 7f8346cef700 1 -- 172.21.15.139:0/1413382702 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f83400fe2b0 con 0x7f83400fee10 2023-07-14T19:25:52.001 INFO:teuthology.orchestra.run.smithi139.stderr:2023-07-14T19:25:51.994+0000 7f8346cef700 1 -- 172.21.15.139:0/1413382702 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f834010a380 con 0x7f83400fee10 2023-07-14T19:25:52.001 INFO:teuthology.orchestra.run.smithi139.stderr:2023-07-14T19:25:51.995+0000 7f833dffb700 1 -- 172.21.15.139:0/1413382702 <== mon.1 v2:172.21.15.139:3300/0 4 ==== mgrmap(e 16) v1 ==== 135715+0+0 (secure 0 0 0) 0x7f8334017880 con 0x7f83400fee10 2023-07-14T19:25:52.001 INFO:teuthology.orchestra.run.smithi139.stderr:2023-07-14T19:25:51.996+0000 7f833dffb700 1 --2- 172.21.15.139:0/1413382702 >> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] conn(0x7f83300a1150 0x7f83300a3600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:25:52.001 INFO:teuthology.orchestra.run.smithi139.stderr:2023-07-14T19:25:51.996+0000 7f833dffb700 1 -- 172.21.15.139:0/1413382702 <== mon.1 v2:172.21.15.139:3300/0 5 ==== osd_map(32..32 src has 1..32) v4 ==== 4101+0+0 (secure 0 0 0) 0x7f83340cceb0 con 0x7f83400fee10 2023-07-14T19:25:52.002 INFO:teuthology.orchestra.run.smithi139.stderr:2023-07-14T19:25:51.996+0000 7f833ffff700 1 --2- 172.21.15.139:0/1413382702 >> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] conn(0x7f83300a1150 0x7f83300a3600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:25:52.002 INFO:teuthology.orchestra.run.smithi139.stderr:2023-07-14T19:25:51.997+0000 7f833ffff700 1 --2- 172.21.15.139:0/1413382702 >> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] conn(0x7f83300a1150 0x7f83300a3600 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f832c005950 tx=0x7f832c00b410 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:25:52.002 INFO:teuthology.orchestra.run.smithi139.stderr:2023-07-14T19:25:52.000+0000 7f833dffb700 1 -- 172.21.15.139:0/1413382702 <== mon.1 v2:172.21.15.139:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f8334097700 con 0x7f83400fee10 2023-07-14T19:25:52.198 INFO:teuthology.orchestra.run.smithi139.stderr:2023-07-14T19:25:52.196+0000 7f8346cef700 1 -- 172.21.15.139:0/1413382702 --> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi139:vg_nvme/lv_3", "target": ["mon-mgr", ""]}) v1 -- 0x7f8340060a80 con 0x7f83300a1150 2023-07-14T19:25:52.441 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:25:52 smithi101 ceph-mon[128807]: purged_snaps scrub starts 2023-07-14T19:25:52.442 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:25:52 smithi101 ceph-mon[128807]: purged_snaps scrub ok 2023-07-14T19:25:52.442 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:25:52 smithi101 ceph-mon[128807]: osd.4 [v2:172.21.15.101:6808/2844761406,v1:172.21.15.101:6809/2844761406] boot 2023-07-14T19:25:52.442 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:25:52 smithi101 ceph-mon[128807]: osdmap e32: 5 total, 5 up, 5 in 2023-07-14T19:25:52.442 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:25:52 smithi101 ceph-mon[128807]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2023-07-14T19:25:52.442 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:25:52 smithi101 ceph-mon[128807]: pgmap v107: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2023-07-14T19:25:52.470 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:25:52 smithi139 ceph-mon[129293]: purged_snaps scrub starts 2023-07-14T19:25:52.471 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:25:52 smithi139 ceph-mon[129293]: purged_snaps scrub ok 2023-07-14T19:25:52.471 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:25:52 smithi139 ceph-mon[129293]: osd.4 [v2:172.21.15.101:6808/2844761406,v1:172.21.15.101:6809/2844761406] boot 2023-07-14T19:25:52.471 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:25:52 smithi139 ceph-mon[129293]: osdmap e32: 5 total, 5 up, 5 in 2023-07-14T19:25:52.471 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:25:52 smithi139 ceph-mon[129293]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2023-07-14T19:25:52.471 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:25:52 smithi139 ceph-mon[129293]: pgmap v107: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2023-07-14T19:25:52.531 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:25:52 smithi098 ceph-mon[127486]: purged_snaps scrub starts 2023-07-14T19:25:52.532 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:25:52 smithi098 ceph-mon[127486]: purged_snaps scrub ok 2023-07-14T19:25:52.532 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:25:52 smithi098 ceph-mon[127486]: osd.4 [v2:172.21.15.101:6808/2844761406,v1:172.21.15.101:6809/2844761406] boot 2023-07-14T19:25:52.532 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:25:52 smithi098 ceph-mon[127486]: osdmap e32: 5 total, 5 up, 5 in 2023-07-14T19:25:52.532 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:25:52 smithi098 ceph-mon[127486]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2023-07-14T19:25:52.532 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:25:52 smithi098 ceph-mon[127486]: pgmap v107: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2023-07-14T19:25:53.276 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:25:53 smithi139 ceph-mon[129293]: from='client.24232 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi139:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2023-07-14T19:25:53.277 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:25:53 smithi139 ceph-mon[129293]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-07-14T19:25:53.277 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:25:53 smithi139 ceph-mon[129293]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-07-14T19:25:53.277 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:25:53 smithi139 ceph-mon[129293]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T19:25:53.277 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:25:53 smithi139 ceph-mon[129293]: osdmap e33: 5 total, 5 up, 5 in 2023-07-14T19:25:53.441 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:25:53 smithi101 ceph-mon[128807]: from='client.24232 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi139:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2023-07-14T19:25:53.441 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:25:53 smithi101 ceph-mon[128807]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-07-14T19:25:53.442 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:25:53 smithi101 ceph-mon[128807]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-07-14T19:25:53.442 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:25:53 smithi101 ceph-mon[128807]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T19:25:53.442 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:25:53 smithi101 ceph-mon[128807]: osdmap e33: 5 total, 5 up, 5 in 2023-07-14T19:25:53.531 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:25:53 smithi098 ceph-mon[127486]: from='client.24232 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi139:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2023-07-14T19:25:53.531 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:25:53 smithi098 ceph-mon[127486]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-07-14T19:25:53.532 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:25:53 smithi098 ceph-mon[127486]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-07-14T19:25:53.532 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:25:53 smithi098 ceph-mon[127486]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T19:25:53.532 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:25:53 smithi098 ceph-mon[127486]: osdmap e33: 5 total, 5 up, 5 in 2023-07-14T19:25:54.441 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:25:54 smithi101 ceph-mon[128807]: pgmap v109: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2023-07-14T19:25:54.471 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:25:54 smithi139 ceph-mon[129293]: pgmap v109: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2023-07-14T19:25:54.531 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:25:54 smithi098 ceph-mon[127486]: pgmap v109: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2023-07-14T19:25:55.441 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:25:55 smithi101 ceph-mon[128807]: from='client.? 172.21.15.139:0/3671153610' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "415de5fa-607a-45a8-b732-28daf3be6dd3"}]: dispatch 2023-07-14T19:25:55.442 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:25:55 smithi101 ceph-mon[128807]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "415de5fa-607a-45a8-b732-28daf3be6dd3"}]: dispatch 2023-07-14T19:25:55.442 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:25:55 smithi101 ceph-mon[128807]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "415de5fa-607a-45a8-b732-28daf3be6dd3"}]': finished 2023-07-14T19:25:55.442 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:25:55 smithi101 ceph-mon[128807]: osdmap e34: 6 total, 5 up, 6 in 2023-07-14T19:25:55.442 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:25:55 smithi101 ceph-mon[128807]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2023-07-14T19:25:55.470 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:25:55 smithi139 ceph-mon[129293]: from='client.? 172.21.15.139:0/3671153610' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "415de5fa-607a-45a8-b732-28daf3be6dd3"}]: dispatch 2023-07-14T19:25:55.471 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:25:55 smithi139 ceph-mon[129293]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "415de5fa-607a-45a8-b732-28daf3be6dd3"}]: dispatch 2023-07-14T19:25:55.471 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:25:55 smithi139 ceph-mon[129293]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "415de5fa-607a-45a8-b732-28daf3be6dd3"}]': finished 2023-07-14T19:25:55.471 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:25:55 smithi139 ceph-mon[129293]: osdmap e34: 6 total, 5 up, 6 in 2023-07-14T19:25:55.471 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:25:55 smithi139 ceph-mon[129293]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2023-07-14T19:25:55.531 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:25:55 smithi098 ceph-mon[127486]: from='client.? 172.21.15.139:0/3671153610' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "415de5fa-607a-45a8-b732-28daf3be6dd3"}]: dispatch 2023-07-14T19:25:55.531 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:25:55 smithi098 ceph-mon[127486]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "415de5fa-607a-45a8-b732-28daf3be6dd3"}]: dispatch 2023-07-14T19:25:55.531 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:25:55 smithi098 ceph-mon[127486]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "415de5fa-607a-45a8-b732-28daf3be6dd3"}]': finished 2023-07-14T19:25:55.532 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:25:55 smithi098 ceph-mon[127486]: osdmap e34: 6 total, 5 up, 6 in 2023-07-14T19:25:55.532 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:25:55 smithi098 ceph-mon[127486]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2023-07-14T19:25:56.441 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:25:56 smithi101 ceph-mon[128807]: from='client.? 172.21.15.139:0/3559583820' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-07-14T19:25:56.441 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:25:56 smithi101 ceph-mon[128807]: pgmap v111: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2023-07-14T19:25:56.470 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:25:56 smithi139 ceph-mon[129293]: from='client.? 172.21.15.139:0/3559583820' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-07-14T19:25:56.471 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:25:56 smithi139 ceph-mon[129293]: pgmap v111: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2023-07-14T19:25:56.531 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:25:56 smithi098 ceph-mon[127486]: from='client.? 172.21.15.139:0/3559583820' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-07-14T19:25:56.531 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:25:56 smithi098 ceph-mon[127486]: pgmap v111: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2023-07-14T19:25:58.898 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:25:58 smithi139 ceph-mon[129293]: pgmap v112: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2023-07-14T19:25:58.941 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:25:58 smithi101 ceph-mon[128807]: pgmap v112: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2023-07-14T19:25:59.031 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:25:58 smithi098 ceph-mon[127486]: pgmap v112: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2023-07-14T19:26:00.720 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:26:00 smithi139 ceph-mon[129293]: pgmap v113: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2023-07-14T19:26:00.941 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:26:00 smithi101 ceph-mon[128807]: pgmap v113: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2023-07-14T19:26:01.031 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:26:00 smithi098 ceph-mon[127486]: pgmap v113: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2023-07-14T19:26:01.941 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:26:01 smithi101 ceph-mon[128807]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2023-07-14T19:26:01.942 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:26:01 smithi101 ceph-mon[128807]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T19:26:01.957 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:26:01 smithi139 ceph-mon[129293]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2023-07-14T19:26:01.958 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:26:01 smithi139 ceph-mon[129293]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T19:26:02.031 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:26:01 smithi098 ceph-mon[127486]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2023-07-14T19:26:02.031 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:26:01 smithi098 ceph-mon[127486]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T19:26:02.941 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:26:02 smithi101 ceph-mon[128807]: Deploying daemon osd.5 on smithi139 2023-07-14T19:26:02.941 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:26:02 smithi101 ceph-mon[128807]: pgmap v114: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2023-07-14T19:26:02.970 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:26:02 smithi139 ceph-mon[129293]: Deploying daemon osd.5 on smithi139 2023-07-14T19:26:02.970 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:26:02 smithi139 ceph-mon[129293]: pgmap v114: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2023-07-14T19:26:03.031 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:26:02 smithi098 ceph-mon[127486]: Deploying daemon osd.5 on smithi139 2023-07-14T19:26:03.031 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:26:02 smithi098 ceph-mon[127486]: pgmap v114: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2023-07-14T19:26:04.941 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:26:04 smithi101 ceph-mon[128807]: pgmap v115: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2023-07-14T19:26:04.971 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:26:04 smithi139 ceph-mon[129293]: pgmap v115: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2023-07-14T19:26:05.031 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:26:04 smithi098 ceph-mon[127486]: pgmap v115: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2023-07-14T19:26:06.608 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:26:06 smithi139 ceph-mon[129293]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' 2023-07-14T19:26:06.609 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:26:06 smithi139 ceph-mon[129293]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' 2023-07-14T19:26:06.609 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:26:06 smithi139 ceph-mon[129293]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-14T19:26:06.609 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:26:06 smithi139 ceph-mon[129293]: pgmap v116: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2023-07-14T19:26:06.781 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:26:06 smithi098 ceph-mon[127486]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' 2023-07-14T19:26:06.782 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:26:06 smithi098 ceph-mon[127486]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' 2023-07-14T19:26:06.782 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:26:06 smithi098 ceph-mon[127486]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-14T19:26:06.782 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:26:06 smithi098 ceph-mon[127486]: pgmap v116: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2023-07-14T19:26:06.941 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:26:06 smithi101 ceph-mon[128807]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' 2023-07-14T19:26:06.942 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:26:06 smithi101 ceph-mon[128807]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' 2023-07-14T19:26:06.942 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:26:06 smithi101 ceph-mon[128807]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-14T19:26:06.942 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:26:06 smithi101 ceph-mon[128807]: pgmap v116: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2023-07-14T19:26:08.307 INFO:teuthology.orchestra.run.smithi139.stderr:2023-07-14T19:26:08.306+0000 7f833dffb700 1 -- 172.21.15.139:0/1413382702 <== mgr.14152 v2:172.21.15.98:6800/452042164 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7f8340060a80 con 0x7f83300a1150 2023-07-14T19:26:08.308 INFO:teuthology.orchestra.run.smithi139.stdout:Created osd(s) 5 on host 'smithi139' 2023-07-14T19:26:08.309 INFO:teuthology.orchestra.run.smithi139.stderr:2023-07-14T19:26:08.309+0000 7f8346cef700 1 -- 172.21.15.139:0/1413382702 >> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] conn(0x7f83300a1150 msgr2=0x7f83300a3600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:26:08.309 INFO:teuthology.orchestra.run.smithi139.stderr:2023-07-14T19:26:08.309+0000 7f8346cef700 1 --2- 172.21.15.139:0/1413382702 >> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] conn(0x7f83300a1150 0x7f83300a3600 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f832c005950 tx=0x7f832c00b410 comp rx=0 tx=0).stop 2023-07-14T19:26:08.309 INFO:teuthology.orchestra.run.smithi139.stderr:2023-07-14T19:26:08.309+0000 7f8346cef700 1 -- 172.21.15.139:0/1413382702 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f83400fee10 msgr2=0x7f8340194980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:26:08.310 INFO:teuthology.orchestra.run.smithi139.stderr:2023-07-14T19:26:08.309+0000 7f8346cef700 1 --2- 172.21.15.139:0/1413382702 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f83400fee10 0x7f8340194980 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f833400eae0 tx=0x7f833400edf0 comp rx=0 tx=0).stop 2023-07-14T19:26:08.310 INFO:teuthology.orchestra.run.smithi139.stderr:2023-07-14T19:26:08.309+0000 7f8346cef700 1 -- 172.21.15.139:0/1413382702 shutdown_connections 2023-07-14T19:26:08.310 INFO:teuthology.orchestra.run.smithi139.stderr:2023-07-14T19:26:08.309+0000 7f8346cef700 1 --2- 172.21.15.139:0/1413382702 >> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] conn(0x7f83300a1150 0x7f83300a3600 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:26:08.310 INFO:teuthology.orchestra.run.smithi139.stderr:2023-07-14T19:26:08.309+0000 7f8346cef700 1 --2- 172.21.15.139:0/1413382702 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f83400fee10 0x7f8340194980 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:26:08.310 INFO:teuthology.orchestra.run.smithi139.stderr:2023-07-14T19:26:08.309+0000 7f8346cef700 1 --2- 172.21.15.139:0/1413382702 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f8340100240 0x7f834019bf40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:26:08.310 INFO:teuthology.orchestra.run.smithi139.stderr:2023-07-14T19:26:08.309+0000 7f8346cef700 1 --2- 172.21.15.139:0/1413382702 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f83400ff7c0 0x7f8340194ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:26:08.311 INFO:teuthology.orchestra.run.smithi139.stderr:2023-07-14T19:26:08.309+0000 7f8346cef700 1 -- 172.21.15.139:0/1413382702 >> 172.21.15.139:0/1413382702 conn(0x7f83400f9990 msgr2=0x7f83400fbba0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:26:08.311 INFO:teuthology.orchestra.run.smithi139.stderr:2023-07-14T19:26:08.309+0000 7f8346cef700 1 -- 172.21.15.139:0/1413382702 shutdown_connections 2023-07-14T19:26:08.311 INFO:teuthology.orchestra.run.smithi139.stderr:2023-07-14T19:26:08.309+0000 7f8346cef700 1 -- 172.21.15.139:0/1413382702 wait complete. 2023-07-14T19:26:08.763 DEBUG:teuthology.orchestra.run.smithi139:osd.5> sudo journalctl -f -n 0 -u ceph-9adea648-227b-11ee-9b33-001a4aab830c@osd.5.service 2023-07-14T19:26:08.766 INFO:tasks.cephadm:Deploying osd.6 on smithi098 with /dev/vg_nvme/lv_2... 2023-07-14T19:26:08.766 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81c9387d2e7576da065747ec9c16d1c7d69016ec shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 9adea648-227b-11ee-9b33-001a4aab830c -- ceph-volume lvm zap /dev/vg_nvme/lv_2 2023-07-14T19:26:08.796 INFO:journalctl@ceph.osd.5.smithi139.stdout:-- Logs begin at Fri 2023-07-14 18:56:10 UTC. -- 2023-07-14T19:26:09.031 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:26:08 smithi098 ceph-mon[127486]: pgmap v117: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2023-07-14T19:26:09.031 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:26:08 smithi098 ceph-mon[127486]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' 2023-07-14T19:26:09.031 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:26:08 smithi098 ceph-mon[127486]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' 2023-07-14T19:26:09.032 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:26:08 smithi098 ceph-mon[127486]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T19:26:09.032 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:26:08 smithi098 ceph-mon[127486]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-14T19:26:09.032 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:26:08 smithi098 ceph-mon[127486]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' 2023-07-14T19:26:09.032 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:26:08 smithi098 ceph-mon[127486]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' 2023-07-14T19:26:09.032 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:26:08 smithi098 ceph-mon[127486]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' 2023-07-14T19:26:09.032 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:26:08 smithi098 ceph-mon[127486]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-14T19:26:09.191 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:26:08 smithi101 ceph-mon[128807]: pgmap v117: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2023-07-14T19:26:09.192 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:26:08 smithi101 ceph-mon[128807]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' 2023-07-14T19:26:09.192 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:26:08 smithi101 ceph-mon[128807]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' 2023-07-14T19:26:09.192 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:26:08 smithi101 ceph-mon[128807]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T19:26:09.192 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:26:08 smithi101 ceph-mon[128807]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-14T19:26:09.192 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:26:08 smithi101 ceph-mon[128807]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' 2023-07-14T19:26:09.193 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:26:08 smithi101 ceph-mon[128807]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' 2023-07-14T19:26:09.193 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:26:08 smithi101 ceph-mon[128807]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' 2023-07-14T19:26:09.193 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:26:08 smithi101 ceph-mon[128807]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-14T19:26:09.206 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:26:08 smithi139 ceph-mon[129293]: pgmap v117: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2023-07-14T19:26:09.206 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:26:08 smithi139 ceph-mon[129293]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' 2023-07-14T19:26:09.206 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:26:08 smithi139 ceph-mon[129293]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' 2023-07-14T19:26:09.206 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:26:08 smithi139 ceph-mon[129293]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T19:26:09.207 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:26:08 smithi139 ceph-mon[129293]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-14T19:26:09.207 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:26:08 smithi139 ceph-mon[129293]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' 2023-07-14T19:26:09.207 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:26:08 smithi139 ceph-mon[129293]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' 2023-07-14T19:26:09.207 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:26:08 smithi139 ceph-mon[129293]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' 2023-07-14T19:26:09.207 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:26:08 smithi139 ceph-mon[129293]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-14T19:26:09.334 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/9adea648-227b-11ee-9b33-001a4aab830c/mon.a/config 2023-07-14T19:26:09.471 INFO:journalctl@ceph.osd.5.smithi139.stdout:Jul 14 19:26:09 smithi139 ceph-9adea648-227b-11ee-9b33-001a4aab830c-osd-5[140626]: 2023-07-14T19:26:09.446+0000 7f0e5d273640 -1 osd.5 0 log_to_monitors true 2023-07-14T19:26:09.893 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:26:09 smithi098 ceph-mon[127486]: from='osd.5 [v2:172.21.15.139:6808/3532263042,v1:172.21.15.139:6809/3532263042]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2023-07-14T19:26:09.894 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:26:09 smithi098 ceph-mon[127486]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2023-07-14T19:26:10.021 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:26:09 smithi139 ceph-mon[129293]: from='osd.5 [v2:172.21.15.139:6808/3532263042,v1:172.21.15.139:6809/3532263042]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2023-07-14T19:26:10.022 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:26:09 smithi139 ceph-mon[129293]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2023-07-14T19:26:10.191 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:26:09 smithi101 ceph-mon[128807]: from='osd.5 [v2:172.21.15.139:6808/3532263042,v1:172.21.15.139:6809/3532263042]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2023-07-14T19:26:10.192 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:26:09 smithi101 ceph-mon[128807]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2023-07-14T19:26:10.923 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:26:10 smithi139 ceph-mon[129293]: pgmap v118: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2023-07-14T19:26:10.923 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:26:10 smithi139 ceph-mon[129293]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2023-07-14T19:26:10.923 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:26:10 smithi139 ceph-mon[129293]: osdmap e35: 6 total, 5 up, 6 in 2023-07-14T19:26:10.924 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:26:10 smithi139 ceph-mon[129293]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2023-07-14T19:26:10.924 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:26:10 smithi139 ceph-mon[129293]: from='osd.5 [v2:172.21.15.139:6808/3532263042,v1:172.21.15.139:6809/3532263042]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi139", "root=default"]}]: dispatch 2023-07-14T19:26:10.924 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:26:10 smithi139 ceph-mon[129293]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi139", "root=default"]}]: dispatch 2023-07-14T19:26:10.924 INFO:journalctl@ceph.osd.5.smithi139.stdout:Jul 14 19:26:10 smithi139 ceph-9adea648-227b-11ee-9b33-001a4aab830c-osd-5[140626]: 2023-07-14T19:26:10.774+0000 7f0e520ea700 -1 osd.5 0 waiting for initial osdmap 2023-07-14T19:26:10.924 INFO:journalctl@ceph.osd.5.smithi139.stdout:Jul 14 19:26:10 smithi139 ceph-9adea648-227b-11ee-9b33-001a4aab830c-osd-5[140626]: 2023-07-14T19:26:10.781+0000 7f0e4d6df700 -1 osd.5 36 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-07-14T19:26:10.925 INFO:teuthology.orchestra.run.smithi098.stderr:--> Zapping: /dev/vg_nvme/lv_2 2023-07-14T19:26:10.925 INFO:teuthology.orchestra.run.smithi098.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 conv=fsync 2023-07-14T19:26:10.925 INFO:teuthology.orchestra.run.smithi098.stderr: stderr: 10+0 records in 2023-07-14T19:26:10.925 INFO:teuthology.orchestra.run.smithi098.stderr:10+0 records out 2023-07-14T19:26:10.925 INFO:teuthology.orchestra.run.smithi098.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0109705 s, 956 MB/s 2023-07-14T19:26:10.926 INFO:teuthology.orchestra.run.smithi098.stderr:--> Zapping successful for: 2023-07-14T19:26:11.031 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:26:10 smithi098 ceph-mon[127486]: pgmap v118: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2023-07-14T19:26:11.031 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:26:10 smithi098 ceph-mon[127486]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2023-07-14T19:26:11.032 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:26:10 smithi098 ceph-mon[127486]: osdmap e35: 6 total, 5 up, 6 in 2023-07-14T19:26:11.032 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:26:10 smithi098 ceph-mon[127486]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2023-07-14T19:26:11.032 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:26:10 smithi098 ceph-mon[127486]: from='osd.5 [v2:172.21.15.139:6808/3532263042,v1:172.21.15.139:6809/3532263042]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi139", "root=default"]}]: dispatch 2023-07-14T19:26:11.032 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:26:10 smithi098 ceph-mon[127486]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi139", "root=default"]}]: dispatch 2023-07-14T19:26:11.191 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:26:10 smithi101 ceph-mon[128807]: pgmap v118: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2023-07-14T19:26:11.192 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:26:10 smithi101 ceph-mon[128807]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2023-07-14T19:26:11.192 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:26:10 smithi101 ceph-mon[128807]: osdmap e35: 6 total, 5 up, 6 in 2023-07-14T19:26:11.192 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:26:10 smithi101 ceph-mon[128807]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2023-07-14T19:26:11.192 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:26:10 smithi101 ceph-mon[128807]: from='osd.5 [v2:172.21.15.139:6808/3532263042,v1:172.21.15.139:6809/3532263042]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi139", "root=default"]}]: dispatch 2023-07-14T19:26:11.192 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:26:10 smithi101 ceph-mon[128807]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi139", "root=default"]}]: dispatch 2023-07-14T19:26:11.376 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81c9387d2e7576da065747ec9c16d1c7d69016ec shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 9adea648-227b-11ee-9b33-001a4aab830c -- ceph orch daemon add osd smithi098:vg_nvme/lv_2 2023-07-14T19:26:11.926 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/9adea648-227b-11ee-9b33-001a4aab830c/mon.a/config 2023-07-14T19:26:12.031 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:26:11 smithi098 ceph-mon[127486]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi139", "root=default"]}]': finished 2023-07-14T19:26:12.032 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:26:11 smithi098 ceph-mon[127486]: osdmap e36: 6 total, 5 up, 6 in 2023-07-14T19:26:12.032 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:26:11 smithi098 ceph-mon[127486]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2023-07-14T19:26:12.032 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:26:11 smithi098 ceph-mon[127486]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2023-07-14T19:26:12.032 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:26:11 smithi098 ceph-mon[127486]: Detected new or changed devices on smithi139 2023-07-14T19:26:12.032 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:26:11 smithi098 ceph-mon[127486]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' 2023-07-14T19:26:12.032 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:26:11 smithi098 ceph-mon[127486]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' 2023-07-14T19:26:12.032 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:26:11 smithi098 ceph-mon[127486]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2023-07-14T19:26:12.033 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:26:11 smithi098 ceph-mon[127486]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2023-07-14T19:26:12.033 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:26:11 smithi098 ceph-mon[127486]: Adjusting osd_memory_target on smithi139 to 8584M 2023-07-14T19:26:12.033 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:26:11 smithi098 ceph-mon[127486]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' 2023-07-14T19:26:12.033 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:26:11 smithi098 ceph-mon[127486]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T19:26:12.033 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:26:11 smithi098 ceph-mon[127486]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-14T19:26:12.033 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:26:11 smithi098 ceph-mon[127486]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' 2023-07-14T19:26:12.033 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:26:11 smithi098 ceph-mon[127486]: osd.5 [v2:172.21.15.139:6808/3532263042,v1:172.21.15.139:6809/3532263042] boot 2023-07-14T19:26:12.034 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:26:11 smithi098 ceph-mon[127486]: osdmap e37: 6 total, 6 up, 6 in 2023-07-14T19:26:12.034 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:26:11 smithi098 ceph-mon[127486]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2023-07-14T19:26:12.191 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:26:11 smithi101 ceph-mon[128807]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi139", "root=default"]}]': finished 2023-07-14T19:26:12.192 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:26:11 smithi101 ceph-mon[128807]: osdmap e36: 6 total, 5 up, 6 in 2023-07-14T19:26:12.192 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:26:11 smithi101 ceph-mon[128807]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2023-07-14T19:26:12.192 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:26:11 smithi101 ceph-mon[128807]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2023-07-14T19:26:12.192 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:26:11 smithi101 ceph-mon[128807]: Detected new or changed devices on smithi139 2023-07-14T19:26:12.193 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:26:11 smithi101 ceph-mon[128807]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' 2023-07-14T19:26:12.193 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:26:11 smithi101 ceph-mon[128807]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' 2023-07-14T19:26:12.193 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:26:11 smithi101 ceph-mon[128807]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2023-07-14T19:26:12.193 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:26:11 smithi101 ceph-mon[128807]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2023-07-14T19:26:12.193 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:26:11 smithi101 ceph-mon[128807]: Adjusting osd_memory_target on smithi139 to 8584M 2023-07-14T19:26:12.193 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:26:11 smithi101 ceph-mon[128807]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' 2023-07-14T19:26:12.193 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:26:11 smithi101 ceph-mon[128807]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T19:26:12.194 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:26:11 smithi101 ceph-mon[128807]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-14T19:26:12.194 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:26:11 smithi101 ceph-mon[128807]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' 2023-07-14T19:26:12.194 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:26:11 smithi101 ceph-mon[128807]: osd.5 [v2:172.21.15.139:6808/3532263042,v1:172.21.15.139:6809/3532263042] boot 2023-07-14T19:26:12.194 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:26:11 smithi101 ceph-mon[128807]: osdmap e37: 6 total, 6 up, 6 in 2023-07-14T19:26:12.194 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:26:11 smithi101 ceph-mon[128807]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2023-07-14T19:26:12.220 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:26:11 smithi139 ceph-mon[129293]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi139", "root=default"]}]': finished 2023-07-14T19:26:12.221 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:26:11 smithi139 ceph-mon[129293]: osdmap e36: 6 total, 5 up, 6 in 2023-07-14T19:26:12.221 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:26:11 smithi139 ceph-mon[129293]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2023-07-14T19:26:12.221 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:26:11 smithi139 ceph-mon[129293]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2023-07-14T19:26:12.221 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:26:11 smithi139 ceph-mon[129293]: Detected new or changed devices on smithi139 2023-07-14T19:26:12.221 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:26:11 smithi139 ceph-mon[129293]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' 2023-07-14T19:26:12.221 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:26:11 smithi139 ceph-mon[129293]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' 2023-07-14T19:26:12.222 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:26:11 smithi139 ceph-mon[129293]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2023-07-14T19:26:12.222 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:26:11 smithi139 ceph-mon[129293]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2023-07-14T19:26:12.222 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:26:11 smithi139 ceph-mon[129293]: Adjusting osd_memory_target on smithi139 to 8584M 2023-07-14T19:26:12.222 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:26:11 smithi139 ceph-mon[129293]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' 2023-07-14T19:26:12.222 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:26:11 smithi139 ceph-mon[129293]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T19:26:12.222 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:26:11 smithi139 ceph-mon[129293]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-14T19:26:12.222 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:26:11 smithi139 ceph-mon[129293]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' 2023-07-14T19:26:12.222 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:26:11 smithi139 ceph-mon[129293]: osd.5 [v2:172.21.15.139:6808/3532263042,v1:172.21.15.139:6809/3532263042] boot 2023-07-14T19:26:12.223 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:26:11 smithi139 ceph-mon[129293]: osdmap e37: 6 total, 6 up, 6 in 2023-07-14T19:26:12.223 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:26:11 smithi139 ceph-mon[129293]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2023-07-14T19:26:13.031 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:26:12 smithi098 ceph-mon[127486]: purged_snaps scrub starts 2023-07-14T19:26:13.031 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:26:12 smithi098 ceph-mon[127486]: purged_snaps scrub ok 2023-07-14T19:26:13.032 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:26:12 smithi098 ceph-mon[127486]: pgmap v122: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2023-07-14T19:26:13.032 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:26:12 smithi098 ceph-mon[127486]: osdmap e38: 6 total, 6 up, 6 in 2023-07-14T19:26:13.109 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:26:13.108+0000 7f42dee25700 1 -- 172.21.15.98:0/3300202865 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f42d8105980 msgr2=0x7f42d8107de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:26:13.109 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:26:13.108+0000 7f42dee25700 1 --2- 172.21.15.98:0/3300202865 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f42d8105980 0x7f42d8107de0 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f42d000b0a0 tx=0x7f42d000b3b0 comp rx=0 tx=0).stop 2023-07-14T19:26:13.110 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:26:13.108+0000 7f42dee25700 1 -- 172.21.15.98:0/3300202865 shutdown_connections 2023-07-14T19:26:13.110 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:26:13.108+0000 7f42dee25700 1 --2- 172.21.15.98:0/3300202865 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f42d8100740 0x7f42d8102b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:26:13.110 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:26:13.108+0000 7f42dee25700 1 --2- 172.21.15.98:0/3300202865 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f42d8105980 0x7f42d8107de0 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:26:13.110 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:26:13.108+0000 7f42dee25700 1 --2- 172.21.15.98:0/3300202865 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f42d8103060 0x7f42d8105440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:26:13.110 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:26:13.108+0000 7f42dee25700 1 -- 172.21.15.98:0/3300202865 >> 172.21.15.98:0/3300202865 conn(0x7f42d80fa1e0 msgr2=0x7f42d80fc630 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:26:13.110 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:26:13.108+0000 7f42dee25700 1 -- 172.21.15.98:0/3300202865 shutdown_connections 2023-07-14T19:26:13.110 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:26:13.108+0000 7f42dee25700 1 -- 172.21.15.98:0/3300202865 wait complete. 2023-07-14T19:26:13.112 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:26:13.109+0000 7f42dee25700 1 Processor -- start 2023-07-14T19:26:13.112 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:26:13.109+0000 7f42dee25700 1 -- start start 2023-07-14T19:26:13.113 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:26:13.109+0000 7f42dee25700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f42d8100740 0x7f42d8195ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:26:13.113 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:26:13.109+0000 7f42dee25700 1 --2- >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f42d8103060 0x7f42d8195ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:26:13.113 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:26:13.109+0000 7f42dee25700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f42d8105980 0x7f42d819d070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:26:13.113 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:26:13.109+0000 7f42dee25700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f42d819d640 con 0x7f42d8100740 2023-07-14T19:26:13.113 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:26:13.109+0000 7f42dee25700 1 -- --> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f42d819d780 con 0x7f42d8103060 2023-07-14T19:26:13.113 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:26:13.109+0000 7f42dee25700 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f42d819d910 con 0x7f42d8105980 2023-07-14T19:26:13.113 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:26:13.109+0000 7f42dcbc1700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f42d8100740 0x7f42d8195ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:26:13.114 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:26:13.109+0000 7f42dcbc1700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f42d8100740 0x7f42d8195ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:44952/0 (socket says 172.21.15.98:44952) 2023-07-14T19:26:13.114 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:26:13.109+0000 7f42dcbc1700 1 -- 172.21.15.98:0/3304595595 learned_addr learned my addr 172.21.15.98:0/3304595595 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-07-14T19:26:13.114 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:26:13.109+0000 7f42d7fff700 1 --2- 172.21.15.98:0/3304595595 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f42d8103060 0x7f42d8195ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:26:13.114 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:26:13.110+0000 7f42dd3c2700 1 --2- 172.21.15.98:0/3304595595 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f42d8105980 0x7f42d819d070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:26:13.114 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:26:13.110+0000 7f42dcbc1700 1 -- 172.21.15.98:0/3304595595 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f42d8105980 msgr2=0x7f42d819d070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:26:13.114 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:26:13.110+0000 7f42dcbc1700 1 --2- 172.21.15.98:0/3304595595 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f42d8105980 0x7f42d819d070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:26:13.114 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:26:13.110+0000 7f42dcbc1700 1 -- 172.21.15.98:0/3304595595 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f42d8103060 msgr2=0x7f42d8195ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:26:13.115 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:26:13.110+0000 7f42dcbc1700 1 --2- 172.21.15.98:0/3304595595 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f42d8103060 0x7f42d8195ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:26:13.115 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:26:13.110+0000 7f42dcbc1700 1 -- 172.21.15.98:0/3304595595 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f42d0009d00 con 0x7f42d8100740 2023-07-14T19:26:13.115 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:26:13.110+0000 7f42d7fff700 1 --2- 172.21.15.98:0/3304595595 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f42d8103060 0x7f42d8195ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-07-14T19:26:13.115 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:26:13.110+0000 7f42dd3c2700 1 --2- 172.21.15.98:0/3304595595 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f42d8105980 0x7f42d819d070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-07-14T19:26:13.115 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:26:13.110+0000 7f42dcbc1700 1 --2- 172.21.15.98:0/3304595595 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f42d8100740 0x7f42d8195ab0 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f42cc00c950 tx=0x7f42cc00cd10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:26:13.115 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:26:13.110+0000 7f42d5ffb700 1 -- 172.21.15.98:0/3304595595 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f42cc007ac0 con 0x7f42d8100740 2023-07-14T19:26:13.116 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:26:13.110+0000 7f42d5ffb700 1 -- 172.21.15.98:0/3304595595 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f42cc004cf0 con 0x7f42d8100740 2023-07-14T19:26:13.116 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:26:13.110+0000 7f42dee25700 1 -- 172.21.15.98:0/3304595595 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f42d819dbf0 con 0x7f42d8100740 2023-07-14T19:26:13.116 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:26:13.110+0000 7f42d5ffb700 1 -- 172.21.15.98:0/3304595595 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f42cc005550 con 0x7f42d8100740 2023-07-14T19:26:13.116 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:26:13.110+0000 7f42dee25700 1 -- 172.21.15.98:0/3304595595 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f42d819e0f0 con 0x7f42d8100740 2023-07-14T19:26:13.116 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:26:13.111+0000 7f42dee25700 1 -- 172.21.15.98:0/3304595595 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f42d80fbd50 con 0x7f42d8100740 2023-07-14T19:26:13.117 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:26:13.112+0000 7f42d5ffb700 1 -- 172.21.15.98:0/3304595595 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 16) v1 ==== 135715+0+0 (secure 0 0 0) 0x7f42cc004750 con 0x7f42d8100740 2023-07-14T19:26:13.118 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:26:13.113+0000 7f42d5ffb700 1 --2- 172.21.15.98:0/3304595595 >> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] conn(0x7f42c80a11d0 0x7f42c80a3680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:26:13.118 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:26:13.113+0000 7f42d7fff700 1 --2- 172.21.15.98:0/3304595595 >> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] conn(0x7f42c80a11d0 0x7f42c80a3680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:26:13.118 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:26:13.113+0000 7f42d5ffb700 1 -- 172.21.15.98:0/3304595595 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(38..38 src has 1..38) v4 ==== 4562+0+0 (secure 0 0 0) 0x7f42cc0cb710 con 0x7f42d8100740 2023-07-14T19:26:13.118 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:26:13.113+0000 7f42d7fff700 1 --2- 172.21.15.98:0/3304595595 >> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] conn(0x7f42c80a11d0 0x7f42c80a3680 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f42c4005950 tx=0x7f42c40058e0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:26:13.118 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:26:13.117+0000 7f42d5ffb700 1 -- 172.21.15.98:0/3304595595 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f42cc095c60 con 0x7f42d8100740 2023-07-14T19:26:13.195 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:26:12 smithi101 ceph-mon[128807]: purged_snaps scrub starts 2023-07-14T19:26:13.195 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:26:12 smithi101 ceph-mon[128807]: purged_snaps scrub ok 2023-07-14T19:26:13.195 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:26:12 smithi101 ceph-mon[128807]: pgmap v122: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2023-07-14T19:26:13.196 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:26:12 smithi101 ceph-mon[128807]: osdmap e38: 6 total, 6 up, 6 in 2023-07-14T19:26:13.220 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:26:12 smithi139 ceph-mon[129293]: purged_snaps scrub starts 2023-07-14T19:26:13.221 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:26:12 smithi139 ceph-mon[129293]: purged_snaps scrub ok 2023-07-14T19:26:13.221 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:26:12 smithi139 ceph-mon[129293]: pgmap v122: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2023-07-14T19:26:13.221 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:26:12 smithi139 ceph-mon[129293]: osdmap e38: 6 total, 6 up, 6 in 2023-07-14T19:26:13.310 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:26:13.310+0000 7f42dee25700 1 -- 172.21.15.98:0/3304595595 --> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi098:vg_nvme/lv_2", "target": ["mon-mgr", ""]}) v1 -- 0x7f42d8060a80 con 0x7f42c80a11d0 2023-07-14T19:26:14.443 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:26:14 smithi098 ceph-mon[127486]: osdmap e39: 6 total, 6 up, 6 in 2023-07-14T19:26:14.444 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:26:14 smithi098 ceph-mon[127486]: from='client.14370 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi098:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2023-07-14T19:26:14.444 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:26:14 smithi098 ceph-mon[127486]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-07-14T19:26:14.444 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:26:14 smithi098 ceph-mon[127486]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-07-14T19:26:14.444 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:26:14 smithi098 ceph-mon[127486]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T19:26:14.445 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:26:14 smithi098 ceph-mon[127486]: pgmap v125: 1 pgs: 1 remapped+peering; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2023-07-14T19:26:14.691 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:26:14 smithi101 ceph-mon[128807]: osdmap e39: 6 total, 6 up, 6 in 2023-07-14T19:26:14.692 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:26:14 smithi101 ceph-mon[128807]: from='client.14370 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi098:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2023-07-14T19:26:14.692 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:26:14 smithi101 ceph-mon[128807]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-07-14T19:26:14.692 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:26:14 smithi101 ceph-mon[128807]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-07-14T19:26:14.692 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:26:14 smithi101 ceph-mon[128807]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T19:26:14.692 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:26:14 smithi101 ceph-mon[128807]: pgmap v125: 1 pgs: 1 remapped+peering; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2023-07-14T19:26:14.720 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:26:14 smithi139 ceph-mon[129293]: osdmap e39: 6 total, 6 up, 6 in 2023-07-14T19:26:14.721 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:26:14 smithi139 ceph-mon[129293]: from='client.14370 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi098:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2023-07-14T19:26:14.721 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:26:14 smithi139 ceph-mon[129293]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-07-14T19:26:14.721 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:26:14 smithi139 ceph-mon[129293]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-07-14T19:26:14.721 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:26:14 smithi139 ceph-mon[129293]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T19:26:14.721 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:26:14 smithi139 ceph-mon[129293]: pgmap v125: 1 pgs: 1 remapped+peering; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2023-07-14T19:26:15.531 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:26:15 smithi098 ceph-mon[127486]: osdmap e40: 6 total, 6 up, 6 in 2023-07-14T19:26:15.691 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:26:15 smithi101 ceph-mon[128807]: osdmap e40: 6 total, 6 up, 6 in 2023-07-14T19:26:15.720 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:26:15 smithi139 ceph-mon[129293]: osdmap e40: 6 total, 6 up, 6 in 2023-07-14T19:26:16.531 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:26:16 smithi098 ceph-mon[127486]: osdmap e41: 6 total, 6 up, 6 in 2023-07-14T19:26:16.531 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:26:16 smithi098 ceph-mon[127486]: pgmap v128: 1 pgs: 1 remapped+peering; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2023-07-14T19:26:16.532 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:26:16 smithi098 ceph-mon[127486]: from='client.? 172.21.15.98:0/2398013035' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "d7f9d4ba-c309-4b75-a836-5a36384b528e"}]: dispatch 2023-07-14T19:26:16.532 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:26:16 smithi098 ceph-mon[127486]: from='client.? 172.21.15.98:0/2398013035' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "d7f9d4ba-c309-4b75-a836-5a36384b528e"}]': finished 2023-07-14T19:26:16.532 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:26:16 smithi098 ceph-mon[127486]: osdmap e42: 7 total, 6 up, 7 in 2023-07-14T19:26:16.532 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:26:16 smithi098 ceph-mon[127486]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2023-07-14T19:26:16.691 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:26:16 smithi101 ceph-mon[128807]: osdmap e41: 6 total, 6 up, 6 in 2023-07-14T19:26:16.692 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:26:16 smithi101 ceph-mon[128807]: pgmap v128: 1 pgs: 1 remapped+peering; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2023-07-14T19:26:16.692 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:26:16 smithi101 ceph-mon[128807]: from='client.? 172.21.15.98:0/2398013035' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "d7f9d4ba-c309-4b75-a836-5a36384b528e"}]: dispatch 2023-07-14T19:26:16.692 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:26:16 smithi101 ceph-mon[128807]: from='client.? 172.21.15.98:0/2398013035' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "d7f9d4ba-c309-4b75-a836-5a36384b528e"}]': finished 2023-07-14T19:26:16.692 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:26:16 smithi101 ceph-mon[128807]: osdmap e42: 7 total, 6 up, 7 in 2023-07-14T19:26:16.692 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:26:16 smithi101 ceph-mon[128807]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2023-07-14T19:26:16.720 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:26:16 smithi139 ceph-mon[129293]: osdmap e41: 6 total, 6 up, 6 in 2023-07-14T19:26:16.721 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:26:16 smithi139 ceph-mon[129293]: pgmap v128: 1 pgs: 1 remapped+peering; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2023-07-14T19:26:16.721 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:26:16 smithi139 ceph-mon[129293]: from='client.? 172.21.15.98:0/2398013035' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "d7f9d4ba-c309-4b75-a836-5a36384b528e"}]: dispatch 2023-07-14T19:26:16.721 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:26:16 smithi139 ceph-mon[129293]: from='client.? 172.21.15.98:0/2398013035' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "d7f9d4ba-c309-4b75-a836-5a36384b528e"}]': finished 2023-07-14T19:26:16.721 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:26:16 smithi139 ceph-mon[129293]: osdmap e42: 7 total, 6 up, 7 in 2023-07-14T19:26:16.721 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:26:16 smithi139 ceph-mon[129293]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2023-07-14T19:26:17.531 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:26:17 smithi098 ceph-mon[127486]: from='client.? 172.21.15.98:0/4088231213' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-07-14T19:26:17.691 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:26:17 smithi101 ceph-mon[128807]: from='client.? 172.21.15.98:0/4088231213' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-07-14T19:26:17.720 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:26:17 smithi139 ceph-mon[129293]: from='client.? 172.21.15.98:0/4088231213' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-07-14T19:26:18.691 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:26:18 smithi101 ceph-mon[128807]: pgmap v130: 1 pgs: 1 remapped+peering; 577 KiB data, 163 MiB used, 536 GiB / 536 GiB avail 2023-07-14T19:26:18.720 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:26:18 smithi139 ceph-mon[129293]: pgmap v130: 1 pgs: 1 remapped+peering; 577 KiB data, 163 MiB used, 536 GiB / 536 GiB avail 2023-07-14T19:26:18.781 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:26:18 smithi098 ceph-mon[127486]: pgmap v130: 1 pgs: 1 remapped+peering; 577 KiB data, 163 MiB used, 536 GiB / 536 GiB avail 2023-07-14T19:26:20.863 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:26:20 smithi098 ceph-mon[127486]: pgmap v131: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 96 KiB/s, 0 objects/s recovering 2023-07-14T19:26:20.941 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:26:20 smithi101 ceph-mon[128807]: pgmap v131: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 96 KiB/s, 0 objects/s recovering 2023-07-14T19:26:20.971 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:26:20 smithi139 ceph-mon[129293]: pgmap v131: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 96 KiB/s, 0 objects/s recovering 2023-07-14T19:26:22.531 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:26:22 smithi098 ceph-mon[127486]: pgmap v132: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 79 KiB/s, 0 objects/s recovering 2023-07-14T19:26:22.691 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:26:22 smithi101 ceph-mon[128807]: pgmap v132: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 79 KiB/s, 0 objects/s recovering 2023-07-14T19:26:22.720 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:26:22 smithi139 ceph-mon[129293]: pgmap v132: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 79 KiB/s, 0 objects/s recovering 2023-07-14T19:26:23.492 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:26:23 smithi098 ceph-mon[127486]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2023-07-14T19:26:23.492 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:26:23 smithi098 ceph-mon[127486]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T19:26:23.492 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:26:23 smithi098 ceph-mon[127486]: Deploying daemon osd.6 on smithi098 2023-07-14T19:26:23.691 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:26:23 smithi101 ceph-mon[128807]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2023-07-14T19:26:23.692 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:26:23 smithi101 ceph-mon[128807]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T19:26:23.692 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:26:23 smithi101 ceph-mon[128807]: Deploying daemon osd.6 on smithi098 2023-07-14T19:26:23.720 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:26:23 smithi139 ceph-mon[129293]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2023-07-14T19:26:23.721 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:26:23 smithi139 ceph-mon[129293]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T19:26:23.721 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:26:23 smithi139 ceph-mon[129293]: Deploying daemon osd.6 on smithi098 2023-07-14T19:26:24.282 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:26:24 smithi098 ceph-mon[127486]: pgmap v133: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 70 KiB/s, 0 objects/s recovering 2023-07-14T19:26:24.691 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:26:24 smithi101 ceph-mon[128807]: pgmap v133: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 70 KiB/s, 0 objects/s recovering 2023-07-14T19:26:24.720 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:26:24 smithi139 ceph-mon[129293]: pgmap v133: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 70 KiB/s, 0 objects/s recovering 2023-07-14T19:26:26.883 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:26:26 smithi098 ceph-mon[127486]: pgmap v134: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 58 KiB/s, 0 objects/s recovering 2023-07-14T19:26:26.941 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:26:26 smithi101 ceph-mon[128807]: pgmap v134: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 58 KiB/s, 0 objects/s recovering 2023-07-14T19:26:26.970 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:26:26 smithi139 ceph-mon[129293]: pgmap v134: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 58 KiB/s, 0 objects/s recovering 2023-07-14T19:26:28.281 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:26:28 smithi098 ceph-mon[127486]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' 2023-07-14T19:26:28.281 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:26:28 smithi098 ceph-mon[127486]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' 2023-07-14T19:26:28.281 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:26:28 smithi098 ceph-mon[127486]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-14T19:26:28.282 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:26:28 smithi098 ceph-mon[127486]: pgmap v135: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 49 KiB/s, 0 objects/s recovering 2023-07-14T19:26:28.441 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:26:28 smithi101 ceph-mon[128807]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' 2023-07-14T19:26:28.442 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:26:28 smithi101 ceph-mon[128807]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' 2023-07-14T19:26:28.442 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:26:28 smithi101 ceph-mon[128807]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-14T19:26:28.442 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:26:28 smithi101 ceph-mon[128807]: pgmap v135: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 49 KiB/s, 0 objects/s recovering 2023-07-14T19:26:28.470 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:26:28 smithi139 ceph-mon[129293]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' 2023-07-14T19:26:28.471 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:26:28 smithi139 ceph-mon[129293]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' 2023-07-14T19:26:28.471 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:26:28 smithi139 ceph-mon[129293]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-14T19:26:28.471 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:26:28 smithi139 ceph-mon[129293]: pgmap v135: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 49 KiB/s, 0 objects/s recovering 2023-07-14T19:26:30.117 INFO:teuthology.orchestra.run.smithi098.stdout:Created osd(s) 6 on host 'smithi098' 2023-07-14T19:26:30.117 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:26:30.114+0000 7f42d5ffb700 1 -- 172.21.15.98:0/3304595595 <== mgr.14152 v2:172.21.15.98:6800/452042164 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7f42d8060a80 con 0x7f42c80a11d0 2023-07-14T19:26:30.118 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:26:30.117+0000 7f42dee25700 1 -- 172.21.15.98:0/3304595595 >> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] conn(0x7f42c80a11d0 msgr2=0x7f42c80a3680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:26:30.118 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:26:30.117+0000 7f42dee25700 1 --2- 172.21.15.98:0/3304595595 >> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] conn(0x7f42c80a11d0 0x7f42c80a3680 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f42c4005950 tx=0x7f42c40058e0 comp rx=0 tx=0).stop 2023-07-14T19:26:30.119 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:26:30.117+0000 7f42dee25700 1 -- 172.21.15.98:0/3304595595 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f42d8100740 msgr2=0x7f42d8195ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:26:30.119 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:26:30.117+0000 7f42dee25700 1 --2- 172.21.15.98:0/3304595595 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f42d8100740 0x7f42d8195ab0 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f42cc00c950 tx=0x7f42cc00cd10 comp rx=0 tx=0).stop 2023-07-14T19:26:30.119 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:26:30.117+0000 7f42dee25700 1 -- 172.21.15.98:0/3304595595 shutdown_connections 2023-07-14T19:26:30.119 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:26:30.117+0000 7f42dee25700 1 --2- 172.21.15.98:0/3304595595 >> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] conn(0x7f42c80a11d0 0x7f42c80a3680 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:26:30.119 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:26:30.117+0000 7f42dee25700 1 --2- 172.21.15.98:0/3304595595 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f42d8105980 0x7f42d819d070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:26:30.119 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:26:30.117+0000 7f42dee25700 1 --2- 172.21.15.98:0/3304595595 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f42d8100740 0x7f42d8195ab0 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:26:30.119 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:26:30.117+0000 7f42dee25700 1 --2- 172.21.15.98:0/3304595595 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f42d8103060 0x7f42d8195ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:26:30.120 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:26:30.117+0000 7f42dee25700 1 -- 172.21.15.98:0/3304595595 >> 172.21.15.98:0/3304595595 conn(0x7f42d80fa1e0 msgr2=0x7f42d81013a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:26:30.120 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:26:30.117+0000 7f42dee25700 1 -- 172.21.15.98:0/3304595595 shutdown_connections 2023-07-14T19:26:30.120 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:26:30.117+0000 7f42dee25700 1 -- 172.21.15.98:0/3304595595 wait complete. 2023-07-14T19:26:30.502 DEBUG:teuthology.orchestra.run.smithi098:osd.6> sudo journalctl -f -n 0 -u ceph-9adea648-227b-11ee-9b33-001a4aab830c@osd.6.service 2023-07-14T19:26:30.505 INFO:tasks.cephadm:Deploying osd.7 on smithi101 with /dev/vg_nvme/lv_2... 2023-07-14T19:26:30.505 DEBUG:teuthology.orchestra.run.smithi101:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81c9387d2e7576da065747ec9c16d1c7d69016ec shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 9adea648-227b-11ee-9b33-001a4aab830c -- ceph-volume lvm zap /dev/vg_nvme/lv_2 2023-07-14T19:26:30.545 INFO:journalctl@ceph.osd.6.smithi098.stdout:-- Logs begin at Fri 2023-07-14 18:57:10 UTC. -- 2023-07-14T19:26:30.941 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:26:30 smithi101 ceph-mon[128807]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' 2023-07-14T19:26:30.942 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:26:30 smithi101 ceph-mon[128807]: pgmap v136: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 48 KiB/s, 0 objects/s recovering 2023-07-14T19:26:30.942 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:26:30 smithi101 ceph-mon[128807]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' 2023-07-14T19:26:30.942 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:26:30 smithi101 ceph-mon[128807]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' 2023-07-14T19:26:30.942 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:26:30 smithi101 ceph-mon[128807]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' 2023-07-14T19:26:30.970 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:26:30 smithi139 ceph-mon[129293]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' 2023-07-14T19:26:30.971 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:26:30 smithi139 ceph-mon[129293]: pgmap v136: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 48 KiB/s, 0 objects/s recovering 2023-07-14T19:26:30.971 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:26:30 smithi139 ceph-mon[129293]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' 2023-07-14T19:26:30.971 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:26:30 smithi139 ceph-mon[129293]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' 2023-07-14T19:26:30.971 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:26:30 smithi139 ceph-mon[129293]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' 2023-07-14T19:26:30.988 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:26:30 smithi098 ceph-mon[127486]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' 2023-07-14T19:26:30.989 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:26:30 smithi098 ceph-mon[127486]: pgmap v136: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 48 KiB/s, 0 objects/s recovering 2023-07-14T19:26:30.989 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:26:30 smithi098 ceph-mon[127486]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' 2023-07-14T19:26:30.989 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:26:30 smithi098 ceph-mon[127486]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' 2023-07-14T19:26:30.989 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:26:30 smithi098 ceph-mon[127486]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' 2023-07-14T19:26:31.080 INFO:teuthology.orchestra.run.smithi101.stderr:Inferring config /var/lib/ceph/9adea648-227b-11ee-9b33-001a4aab830c/mon.b/config 2023-07-14T19:26:31.281 INFO:journalctl@ceph.osd.6.smithi098.stdout:Jul 14 19:26:31 smithi098 ceph-9adea648-227b-11ee-9b33-001a4aab830c-osd-6[151312]: 2023-07-14T19:26:31.116+0000 7f464f588640 -1 osd.6 0 log_to_monitors true 2023-07-14T19:26:31.623 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:26:31 smithi101 ceph-mon[128807]: from='osd.6 [v2:172.21.15.98:6818/658948743,v1:172.21.15.98:6819/658948743]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2023-07-14T19:26:31.818 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:26:31 smithi098 ceph-mon[127486]: from='osd.6 [v2:172.21.15.98:6818/658948743,v1:172.21.15.98:6819/658948743]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2023-07-14T19:26:31.970 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:26:31 smithi139 ceph-mon[129293]: from='osd.6 [v2:172.21.15.98:6818/658948743,v1:172.21.15.98:6819/658948743]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2023-07-14T19:26:32.679 INFO:teuthology.orchestra.run.smithi101.stderr:--> Zapping: /dev/vg_nvme/lv_2 2023-07-14T19:26:32.680 INFO:teuthology.orchestra.run.smithi101.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 conv=fsync 2023-07-14T19:26:32.680 INFO:teuthology.orchestra.run.smithi101.stderr: stderr: 10+0 records in 2023-07-14T19:26:32.681 INFO:teuthology.orchestra.run.smithi101.stderr:10+0 records out 2023-07-14T19:26:32.681 INFO:teuthology.orchestra.run.smithi101.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0109831 s, 955 MB/s 2023-07-14T19:26:32.681 INFO:teuthology.orchestra.run.smithi101.stderr:--> Zapping successful for: 2023-07-14T19:26:32.757 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:26:32 smithi098 ceph-mon[127486]: pgmap v137: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2023-07-14T19:26:32.757 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:26:32 smithi098 ceph-mon[127486]: from='osd.6 [v2:172.21.15.98:6818/658948743,v1:172.21.15.98:6819/658948743]' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2023-07-14T19:26:32.758 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:26:32 smithi098 ceph-mon[127486]: osdmap e43: 7 total, 6 up, 7 in 2023-07-14T19:26:32.758 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:26:32 smithi098 ceph-mon[127486]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2023-07-14T19:26:32.758 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:26:32 smithi098 ceph-mon[127486]: from='osd.6 [v2:172.21.15.98:6818/658948743,v1:172.21.15.98:6819/658948743]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi098", "root=default"]}]: dispatch 2023-07-14T19:26:32.758 INFO:journalctl@ceph.osd.6.smithi098.stdout:Jul 14 19:26:32 smithi098 ceph-9adea648-227b-11ee-9b33-001a4aab830c-osd-6[151312]: 2023-07-14T19:26:32.581+0000 7f46443ff700 -1 osd.6 0 waiting for initial osdmap 2023-07-14T19:26:32.758 INFO:journalctl@ceph.osd.6.smithi098.stdout:Jul 14 19:26:32 smithi098 ceph-9adea648-227b-11ee-9b33-001a4aab830c-osd-6[151312]: 2023-07-14T19:26:32.593+0000 7f463d1ef700 -1 osd.6 44 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-07-14T19:26:32.941 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:26:32 smithi101 ceph-mon[128807]: pgmap v137: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2023-07-14T19:26:32.942 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:26:32 smithi101 ceph-mon[128807]: from='osd.6 [v2:172.21.15.98:6818/658948743,v1:172.21.15.98:6819/658948743]' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2023-07-14T19:26:32.942 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:26:32 smithi101 ceph-mon[128807]: osdmap e43: 7 total, 6 up, 7 in 2023-07-14T19:26:32.942 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:26:32 smithi101 ceph-mon[128807]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2023-07-14T19:26:32.942 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:26:32 smithi101 ceph-mon[128807]: from='osd.6 [v2:172.21.15.98:6818/658948743,v1:172.21.15.98:6819/658948743]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi098", "root=default"]}]: dispatch 2023-07-14T19:26:32.970 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:26:32 smithi139 ceph-mon[129293]: pgmap v137: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2023-07-14T19:26:32.971 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:26:32 smithi139 ceph-mon[129293]: from='osd.6 [v2:172.21.15.98:6818/658948743,v1:172.21.15.98:6819/658948743]' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2023-07-14T19:26:32.971 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:26:32 smithi139 ceph-mon[129293]: osdmap e43: 7 total, 6 up, 7 in 2023-07-14T19:26:32.971 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:26:32 smithi139 ceph-mon[129293]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2023-07-14T19:26:32.971 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:26:32 smithi139 ceph-mon[129293]: from='osd.6 [v2:172.21.15.98:6818/658948743,v1:172.21.15.98:6819/658948743]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi098", "root=default"]}]: dispatch 2023-07-14T19:26:33.133 DEBUG:teuthology.orchestra.run.smithi101:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81c9387d2e7576da065747ec9c16d1c7d69016ec shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 9adea648-227b-11ee-9b33-001a4aab830c -- ceph orch daemon add osd smithi101:vg_nvme/lv_2 2023-07-14T19:26:33.682 INFO:teuthology.orchestra.run.smithi101.stderr:Inferring config /var/lib/ceph/9adea648-227b-11ee-9b33-001a4aab830c/mon.b/config 2023-07-14T19:26:33.887 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:26:33 smithi101 ceph-mon[128807]: from='osd.6 [v2:172.21.15.98:6818/658948743,v1:172.21.15.98:6819/658948743]' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi098", "root=default"]}]': finished 2023-07-14T19:26:33.887 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:26:33 smithi101 ceph-mon[128807]: osdmap e44: 7 total, 6 up, 7 in 2023-07-14T19:26:33.887 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:26:33 smithi101 ceph-mon[128807]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2023-07-14T19:26:33.888 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:26:33 smithi101 ceph-mon[128807]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2023-07-14T19:26:33.888 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:26:33 smithi101 ceph-mon[128807]: Detected new or changed devices on smithi098 2023-07-14T19:26:33.888 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:26:33 smithi101 ceph-mon[128807]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' 2023-07-14T19:26:33.888 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:26:33 smithi101 ceph-mon[128807]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' 2023-07-14T19:26:33.888 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:26:33 smithi101 ceph-mon[128807]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2023-07-14T19:26:33.888 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:26:33 smithi101 ceph-mon[128807]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2023-07-14T19:26:33.888 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:26:33 smithi101 ceph-mon[128807]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2023-07-14T19:26:33.889 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:26:33 smithi101 ceph-mon[128807]: Adjusting osd_memory_target on smithi098 to 5722M 2023-07-14T19:26:33.889 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:26:33 smithi101 ceph-mon[128807]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' 2023-07-14T19:26:33.889 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:26:33 smithi101 ceph-mon[128807]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T19:26:33.889 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:26:33 smithi101 ceph-mon[128807]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-14T19:26:33.889 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:26:33 smithi101 ceph-mon[128807]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' 2023-07-14T19:26:33.889 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:26:33 smithi101 ceph-mon[128807]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-14T19:26:33.889 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:26:33 smithi101 ceph-mon[128807]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T19:26:33.890 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:26:33 smithi101 ceph-mon[128807]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-14T19:26:33.890 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:26:33 smithi101 ceph-mon[128807]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' 2023-07-14T19:26:33.970 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:26:33 smithi139 ceph-mon[129293]: from='osd.6 [v2:172.21.15.98:6818/658948743,v1:172.21.15.98:6819/658948743]' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi098", "root=default"]}]': finished 2023-07-14T19:26:33.971 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:26:33 smithi139 ceph-mon[129293]: osdmap e44: 7 total, 6 up, 7 in 2023-07-14T19:26:33.971 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:26:33 smithi139 ceph-mon[129293]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2023-07-14T19:26:33.971 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:26:33 smithi139 ceph-mon[129293]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2023-07-14T19:26:33.971 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:26:33 smithi139 ceph-mon[129293]: Detected new or changed devices on smithi098 2023-07-14T19:26:33.971 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:26:33 smithi139 ceph-mon[129293]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' 2023-07-14T19:26:33.971 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:26:33 smithi139 ceph-mon[129293]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' 2023-07-14T19:26:33.972 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:26:33 smithi139 ceph-mon[129293]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2023-07-14T19:26:33.972 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:26:33 smithi139 ceph-mon[129293]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2023-07-14T19:26:33.972 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:26:33 smithi139 ceph-mon[129293]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2023-07-14T19:26:33.972 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:26:33 smithi139 ceph-mon[129293]: Adjusting osd_memory_target on smithi098 to 5722M 2023-07-14T19:26:33.972 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:26:33 smithi139 ceph-mon[129293]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' 2023-07-14T19:26:33.972 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:26:33 smithi139 ceph-mon[129293]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T19:26:33.972 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:26:33 smithi139 ceph-mon[129293]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-14T19:26:33.973 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:26:33 smithi139 ceph-mon[129293]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' 2023-07-14T19:26:33.973 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:26:33 smithi139 ceph-mon[129293]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-14T19:26:33.973 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:26:33 smithi139 ceph-mon[129293]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T19:26:33.973 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:26:33 smithi139 ceph-mon[129293]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-14T19:26:33.973 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:26:33 smithi139 ceph-mon[129293]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' 2023-07-14T19:26:34.031 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:26:33 smithi098 ceph-mon[127486]: from='osd.6 [v2:172.21.15.98:6818/658948743,v1:172.21.15.98:6819/658948743]' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi098", "root=default"]}]': finished 2023-07-14T19:26:34.032 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:26:33 smithi098 ceph-mon[127486]: osdmap e44: 7 total, 6 up, 7 in 2023-07-14T19:26:34.032 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:26:33 smithi098 ceph-mon[127486]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2023-07-14T19:26:34.032 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:26:33 smithi098 ceph-mon[127486]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2023-07-14T19:26:34.032 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:26:33 smithi098 ceph-mon[127486]: Detected new or changed devices on smithi098 2023-07-14T19:26:34.033 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:26:33 smithi098 ceph-mon[127486]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' 2023-07-14T19:26:34.033 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:26:33 smithi098 ceph-mon[127486]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' 2023-07-14T19:26:34.033 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:26:33 smithi098 ceph-mon[127486]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2023-07-14T19:26:34.033 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:26:33 smithi098 ceph-mon[127486]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2023-07-14T19:26:34.033 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:26:33 smithi098 ceph-mon[127486]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2023-07-14T19:26:34.033 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:26:33 smithi098 ceph-mon[127486]: Adjusting osd_memory_target on smithi098 to 5722M 2023-07-14T19:26:34.033 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:26:33 smithi098 ceph-mon[127486]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' 2023-07-14T19:26:34.033 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:26:33 smithi098 ceph-mon[127486]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T19:26:34.034 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:26:33 smithi098 ceph-mon[127486]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-14T19:26:34.034 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:26:33 smithi098 ceph-mon[127486]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' 2023-07-14T19:26:34.034 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:26:33 smithi098 ceph-mon[127486]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-14T19:26:34.034 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:26:33 smithi098 ceph-mon[127486]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T19:26:34.034 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:26:33 smithi098 ceph-mon[127486]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-14T19:26:34.034 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:26:33 smithi098 ceph-mon[127486]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' 2023-07-14T19:26:34.894 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-14T19:26:34.892+0000 7efc20642700 1 -- 172.21.15.101:0/3277870329 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7efc18101850 msgr2=0x7efc18108110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:26:34.894 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-14T19:26:34.892+0000 7efc20642700 1 --2- 172.21.15.101:0/3277870329 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7efc18101850 0x7efc18108110 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7efc1000b0a0 tx=0x7efc1000b3b0 comp rx=0 tx=0).stop 2023-07-14T19:26:34.894 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-14T19:26:34.892+0000 7efc20642700 1 -- 172.21.15.101:0/3277870329 shutdown_connections 2023-07-14T19:26:34.894 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-14T19:26:34.892+0000 7efc20642700 1 --2- 172.21.15.101:0/3277870329 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7efc180ffcc0 0x7efc181000d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:26:34.895 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-14T19:26:34.892+0000 7efc20642700 1 --2- 172.21.15.101:0/3277870329 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7efc18101850 0x7efc18108110 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:26:34.895 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-14T19:26:34.892+0000 7efc20642700 1 --2- 172.21.15.101:0/3277870329 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7efc18100ec0 0x7efc18101310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:26:34.895 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-14T19:26:34.892+0000 7efc20642700 1 -- 172.21.15.101:0/3277870329 >> 172.21.15.101:0/3277870329 conn(0x7efc180fb2d0 msgr2=0x7efc180fd740 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:26:34.895 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-14T19:26:34.892+0000 7efc20642700 1 -- 172.21.15.101:0/3277870329 shutdown_connections 2023-07-14T19:26:34.895 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-14T19:26:34.892+0000 7efc20642700 1 -- 172.21.15.101:0/3277870329 wait complete. 2023-07-14T19:26:34.895 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-14T19:26:34.893+0000 7efc20642700 1 Processor -- start 2023-07-14T19:26:34.895 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-14T19:26:34.893+0000 7efc20642700 1 -- start start 2023-07-14T19:26:34.897 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-14T19:26:34.893+0000 7efc20642700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7efc180ffcc0 0x7efc18195a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:26:34.897 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-14T19:26:34.893+0000 7efc20642700 1 --2- >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7efc18100ec0 0x7efc18195f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:26:34.898 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-14T19:26:34.893+0000 7efc20642700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7efc18101850 0x7efc1819cfe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:26:34.898 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-14T19:26:34.893+0000 7efc20642700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efc1819d5b0 con 0x7efc180ffcc0 2023-07-14T19:26:34.898 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-14T19:26:34.893+0000 7efc20642700 1 -- --> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efc1819d6f0 con 0x7efc18100ec0 2023-07-14T19:26:34.898 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-14T19:26:34.893+0000 7efc20642700 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efc1819d880 con 0x7efc18101850 2023-07-14T19:26:34.898 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-14T19:26:34.893+0000 7efc1dbdd700 1 --2- >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7efc18100ec0 0x7efc18195f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:26:34.898 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-14T19:26:34.893+0000 7efc1dbdd700 1 --2- >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7efc18100ec0 0x7efc18195f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.101:3300/0 says I am v2:172.21.15.101:36206/0 (socket says 172.21.15.101:36206) 2023-07-14T19:26:34.898 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-14T19:26:34.893+0000 7efc1dbdd700 1 -- 172.21.15.101:0/3845646295 learned_addr learned my addr 172.21.15.101:0/3845646295 (peer_addr_for_me v2:172.21.15.101:0/0) 2023-07-14T19:26:34.899 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-14T19:26:34.894+0000 7efc1ebdf700 1 --2- 172.21.15.101:0/3845646295 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7efc18101850 0x7efc1819cfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:26:34.899 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-14T19:26:34.894+0000 7efc1e3de700 1 --2- 172.21.15.101:0/3845646295 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7efc180ffcc0 0x7efc18195a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:26:34.899 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-14T19:26:34.894+0000 7efc1dbdd700 1 -- 172.21.15.101:0/3845646295 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7efc18101850 msgr2=0x7efc1819cfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:26:34.899 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-14T19:26:34.894+0000 7efc1dbdd700 1 --2- 172.21.15.101:0/3845646295 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7efc18101850 0x7efc1819cfe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:26:34.899 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-14T19:26:34.894+0000 7efc1dbdd700 1 -- 172.21.15.101:0/3845646295 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7efc180ffcc0 msgr2=0x7efc18195a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:26:34.899 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-14T19:26:34.894+0000 7efc1dbdd700 1 --2- 172.21.15.101:0/3845646295 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7efc180ffcc0 0x7efc18195a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:26:34.899 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-14T19:26:34.894+0000 7efc1dbdd700 1 -- 172.21.15.101:0/3845646295 --> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efc10009d00 con 0x7efc18100ec0 2023-07-14T19:26:34.899 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-14T19:26:34.894+0000 7efc1dbdd700 1 --2- 172.21.15.101:0/3845646295 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7efc18100ec0 0x7efc18195f60 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7efc1400c9b0 tx=0x7efc1400ccc0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:26:34.900 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-14T19:26:34.894+0000 7efc0f7fe700 1 -- 172.21.15.101:0/3845646295 <== mon.2 v2:172.21.15.101:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efc14007b00 con 0x7efc18100ec0 2023-07-14T19:26:34.900 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-14T19:26:34.894+0000 7efc20642700 1 -- 172.21.15.101:0/3845646295 --> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efc1819db60 con 0x7efc18100ec0 2023-07-14T19:26:34.900 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-14T19:26:34.894+0000 7efc0f7fe700 1 -- 172.21.15.101:0/3845646295 <== mon.2 v2:172.21.15.101:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7efc1400ce50 con 0x7efc18100ec0 2023-07-14T19:26:34.900 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-14T19:26:34.894+0000 7efc0f7fe700 1 -- 172.21.15.101:0/3845646295 <== mon.2 v2:172.21.15.101:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efc14018660 con 0x7efc18100ec0 2023-07-14T19:26:34.900 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-14T19:26:34.894+0000 7efc20642700 1 -- 172.21.15.101:0/3845646295 --> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efc1819e0b0 con 0x7efc18100ec0 2023-07-14T19:26:34.900 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-14T19:26:34.895+0000 7efc20642700 1 -- 172.21.15.101:0/3845646295 --> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efc181071c0 con 0x7efc18100ec0 2023-07-14T19:26:34.900 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-14T19:26:34.896+0000 7efc0f7fe700 1 -- 172.21.15.101:0/3845646295 <== mon.2 v2:172.21.15.101:3300/0 4 ==== mgrmap(e 16) v1 ==== 135715+0+0 (secure 0 0 0) 0x7efc1400f450 con 0x7efc18100ec0 2023-07-14T19:26:34.901 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-14T19:26:34.896+0000 7efc0f7fe700 1 --2- 172.21.15.101:0/3845646295 >> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] conn(0x7efc040a1150 0x7efc040a3600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:26:34.901 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-14T19:26:34.897+0000 7efc0f7fe700 1 -- 172.21.15.101:0/3845646295 <== mon.2 v2:172.21.15.101:3300/0 5 ==== osd_map(45..45 src has 1..45) v4 ==== 4965+0+0 (secure 0 0 0) 0x7efc140cbb40 con 0x7efc18100ec0 2023-07-14T19:26:34.901 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-14T19:26:34.897+0000 7efc1e3de700 1 --2- 172.21.15.101:0/3845646295 >> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] conn(0x7efc040a1150 0x7efc040a3600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:26:34.902 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-14T19:26:34.897+0000 7efc1e3de700 1 --2- 172.21.15.101:0/3845646295 >> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] conn(0x7efc040a1150 0x7efc040a3600 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7efc18100d20 tx=0x7efc0800b410 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:26:34.902 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-14T19:26:34.901+0000 7efc0f7fe700 1 -- 172.21.15.101:0/3845646295 <== mon.2 v2:172.21.15.101:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7efc14095fa0 con 0x7efc18100ec0 2023-07-14T19:26:34.942 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:26:34 smithi101 ceph-mon[128807]: purged_snaps scrub starts 2023-07-14T19:26:34.942 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:26:34 smithi101 ceph-mon[128807]: purged_snaps scrub ok 2023-07-14T19:26:34.942 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:26:34 smithi101 ceph-mon[128807]: pgmap v140: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2023-07-14T19:26:34.942 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:26:34 smithi101 ceph-mon[128807]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2023-07-14T19:26:34.942 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:26:34 smithi101 ceph-mon[128807]: osd.6 [v2:172.21.15.98:6818/658948743,v1:172.21.15.98:6819/658948743] boot 2023-07-14T19:26:34.943 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:26:34 smithi101 ceph-mon[128807]: osdmap e45: 7 total, 7 up, 7 in 2023-07-14T19:26:34.943 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:26:34 smithi101 ceph-mon[128807]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2023-07-14T19:26:34.970 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:26:34 smithi139 ceph-mon[129293]: purged_snaps scrub starts 2023-07-14T19:26:34.970 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:26:34 smithi139 ceph-mon[129293]: purged_snaps scrub ok 2023-07-14T19:26:34.971 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:26:34 smithi139 ceph-mon[129293]: pgmap v140: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2023-07-14T19:26:34.971 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:26:34 smithi139 ceph-mon[129293]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2023-07-14T19:26:34.971 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:26:34 smithi139 ceph-mon[129293]: osd.6 [v2:172.21.15.98:6818/658948743,v1:172.21.15.98:6819/658948743] boot 2023-07-14T19:26:34.971 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:26:34 smithi139 ceph-mon[129293]: osdmap e45: 7 total, 7 up, 7 in 2023-07-14T19:26:34.971 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:26:34 smithi139 ceph-mon[129293]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2023-07-14T19:26:35.031 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:26:34 smithi098 ceph-mon[127486]: purged_snaps scrub starts 2023-07-14T19:26:35.032 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:26:34 smithi098 ceph-mon[127486]: purged_snaps scrub ok 2023-07-14T19:26:35.032 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:26:34 smithi098 ceph-mon[127486]: pgmap v140: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2023-07-14T19:26:35.032 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:26:34 smithi098 ceph-mon[127486]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2023-07-14T19:26:35.033 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:26:34 smithi098 ceph-mon[127486]: osd.6 [v2:172.21.15.98:6818/658948743,v1:172.21.15.98:6819/658948743] boot 2023-07-14T19:26:35.033 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:26:34 smithi098 ceph-mon[127486]: osdmap e45: 7 total, 7 up, 7 in 2023-07-14T19:26:35.033 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:26:34 smithi098 ceph-mon[127486]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2023-07-14T19:26:35.096 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-14T19:26:35.095+0000 7efc20642700 1 -- 172.21.15.101:0/3845646295 --> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi101:vg_nvme/lv_2", "target": ["mon-mgr", ""]}) v1 -- 0x7efc18060a80 con 0x7efc040a1150 2023-07-14T19:26:36.191 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:26:35 smithi101 ceph-mon[128807]: osdmap e46: 7 total, 7 up, 7 in 2023-07-14T19:26:36.192 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:26:35 smithi101 ceph-mon[128807]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-07-14T19:26:36.192 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:26:35 smithi101 ceph-mon[128807]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-07-14T19:26:36.192 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:26:35 smithi101 ceph-mon[128807]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T19:26:36.220 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:26:35 smithi139 ceph-mon[129293]: osdmap e46: 7 total, 7 up, 7 in 2023-07-14T19:26:36.220 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:26:35 smithi139 ceph-mon[129293]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-07-14T19:26:36.221 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:26:35 smithi139 ceph-mon[129293]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-07-14T19:26:36.221 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:26:35 smithi139 ceph-mon[129293]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T19:26:36.288 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:26:35 smithi098 ceph-mon[127486]: osdmap e46: 7 total, 7 up, 7 in 2023-07-14T19:26:36.288 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:26:35 smithi098 ceph-mon[127486]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-07-14T19:26:36.289 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:26:35 smithi098 ceph-mon[127486]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-07-14T19:26:36.289 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:26:35 smithi098 ceph-mon[127486]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T19:26:37.220 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:26:36 smithi139 ceph-mon[129293]: from='client.24284 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi101:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2023-07-14T19:26:37.221 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:26:36 smithi139 ceph-mon[129293]: pgmap v143: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2023-07-14T19:26:37.281 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:26:36 smithi098 ceph-mon[127486]: from='client.24284 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi101:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2023-07-14T19:26:37.282 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:26:36 smithi098 ceph-mon[127486]: pgmap v143: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2023-07-14T19:26:37.442 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:26:36 smithi101 ceph-mon[128807]: from='client.24284 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi101:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2023-07-14T19:26:37.442 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:26:36 smithi101 ceph-mon[128807]: pgmap v143: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2023-07-14T19:26:38.220 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:26:37 smithi139 ceph-mon[129293]: from='client.? 172.21.15.101:0/550206361' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "eedb94d9-c5cd-4a93-8a8c-2648804b7195"}]: dispatch 2023-07-14T19:26:38.221 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:26:37 smithi139 ceph-mon[129293]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "eedb94d9-c5cd-4a93-8a8c-2648804b7195"}]: dispatch 2023-07-14T19:26:38.221 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:26:37 smithi139 ceph-mon[129293]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "eedb94d9-c5cd-4a93-8a8c-2648804b7195"}]': finished 2023-07-14T19:26:38.221 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:26:37 smithi139 ceph-mon[129293]: osdmap e47: 8 total, 7 up, 8 in 2023-07-14T19:26:38.221 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:26:37 smithi139 ceph-mon[129293]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2023-07-14T19:26:38.281 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:26:37 smithi098 ceph-mon[127486]: from='client.? 172.21.15.101:0/550206361' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "eedb94d9-c5cd-4a93-8a8c-2648804b7195"}]: dispatch 2023-07-14T19:26:38.281 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:26:37 smithi098 ceph-mon[127486]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "eedb94d9-c5cd-4a93-8a8c-2648804b7195"}]: dispatch 2023-07-14T19:26:38.282 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:26:37 smithi098 ceph-mon[127486]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "eedb94d9-c5cd-4a93-8a8c-2648804b7195"}]': finished 2023-07-14T19:26:38.282 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:26:37 smithi098 ceph-mon[127486]: osdmap e47: 8 total, 7 up, 8 in 2023-07-14T19:26:38.282 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:26:37 smithi098 ceph-mon[127486]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2023-07-14T19:26:38.441 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:26:37 smithi101 ceph-mon[128807]: from='client.? 172.21.15.101:0/550206361' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "eedb94d9-c5cd-4a93-8a8c-2648804b7195"}]: dispatch 2023-07-14T19:26:38.442 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:26:37 smithi101 ceph-mon[128807]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "eedb94d9-c5cd-4a93-8a8c-2648804b7195"}]: dispatch 2023-07-14T19:26:38.442 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:26:37 smithi101 ceph-mon[128807]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "eedb94d9-c5cd-4a93-8a8c-2648804b7195"}]': finished 2023-07-14T19:26:38.442 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:26:37 smithi101 ceph-mon[128807]: osdmap e47: 8 total, 7 up, 8 in 2023-07-14T19:26:38.442 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:26:37 smithi101 ceph-mon[128807]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2023-07-14T19:26:39.281 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:26:38 smithi098 ceph-mon[127486]: pgmap v145: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2023-07-14T19:26:39.281 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:26:38 smithi098 ceph-mon[127486]: from='client.? 172.21.15.101:0/1924325494' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-07-14T19:26:39.441 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:26:38 smithi101 ceph-mon[128807]: pgmap v145: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2023-07-14T19:26:39.442 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:26:38 smithi101 ceph-mon[128807]: from='client.? 172.21.15.101:0/1924325494' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-07-14T19:26:39.470 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:26:38 smithi139 ceph-mon[129293]: pgmap v145: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2023-07-14T19:26:39.471 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:26:38 smithi139 ceph-mon[129293]: from='client.? 172.21.15.101:0/1924325494' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-07-14T19:26:41.281 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:26:40 smithi098 ceph-mon[127486]: pgmap v146: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2023-07-14T19:26:41.347 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:26:40 smithi101 ceph-mon[128807]: pgmap v146: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2023-07-14T19:26:41.470 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:26:40 smithi139 ceph-mon[129293]: pgmap v146: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2023-07-14T19:26:42.531 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:26:42 smithi098 ceph-mon[127486]: pgmap v147: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2023-07-14T19:26:42.607 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:26:42 smithi101 ceph-mon[128807]: pgmap v147: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2023-07-14T19:26:42.720 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:26:42 smithi139 ceph-mon[129293]: pgmap v147: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2023-07-14T19:26:44.908 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:26:44 smithi101 ceph-mon[128807]: pgmap v148: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2023-07-14T19:26:44.908 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:26:44 smithi101 ceph-mon[128807]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2023-07-14T19:26:44.908 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:26:44 smithi101 ceph-mon[128807]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T19:26:44.970 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:26:44 smithi139 ceph-mon[129293]: pgmap v148: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2023-07-14T19:26:44.971 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:26:44 smithi139 ceph-mon[129293]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2023-07-14T19:26:44.971 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:26:44 smithi139 ceph-mon[129293]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T19:26:45.031 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:26:44 smithi098 ceph-mon[127486]: pgmap v148: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2023-07-14T19:26:45.032 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:26:44 smithi098 ceph-mon[127486]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2023-07-14T19:26:45.032 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:26:44 smithi098 ceph-mon[127486]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T19:26:45.881 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:26:45 smithi101 ceph-mon[128807]: Deploying daemon osd.7 on smithi101 2023-07-14T19:26:45.970 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:26:45 smithi139 ceph-mon[129293]: Deploying daemon osd.7 on smithi101 2023-07-14T19:26:46.031 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:26:45 smithi098 ceph-mon[127486]: Deploying daemon osd.7 on smithi101 2023-07-14T19:26:46.942 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:26:46 smithi101 ceph-mon[128807]: pgmap v149: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2023-07-14T19:26:46.970 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:26:46 smithi139 ceph-mon[129293]: pgmap v149: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2023-07-14T19:26:47.031 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:26:46 smithi098 ceph-mon[127486]: pgmap v149: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2023-07-14T19:26:48.941 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:26:48 smithi101 ceph-mon[128807]: pgmap v150: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2023-07-14T19:26:48.941 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:26:48 smithi101 ceph-mon[128807]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' 2023-07-14T19:26:48.942 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:26:48 smithi101 ceph-mon[128807]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' 2023-07-14T19:26:48.942 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:26:48 smithi101 ceph-mon[128807]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-14T19:26:48.970 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:26:48 smithi139 ceph-mon[129293]: pgmap v150: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2023-07-14T19:26:48.970 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:26:48 smithi139 ceph-mon[129293]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' 2023-07-14T19:26:48.971 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:26:48 smithi139 ceph-mon[129293]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' 2023-07-14T19:26:48.971 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:26:48 smithi139 ceph-mon[129293]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-14T19:26:49.031 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:26:48 smithi098 ceph-mon[127486]: pgmap v150: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2023-07-14T19:26:49.032 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:26:48 smithi098 ceph-mon[127486]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' 2023-07-14T19:26:49.032 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:26:48 smithi098 ceph-mon[127486]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' 2023-07-14T19:26:49.032 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:26:48 smithi098 ceph-mon[127486]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-14T19:26:50.912 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:26:50 smithi101 ceph-mon[128807]: pgmap v151: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2023-07-14T19:26:50.970 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:26:50 smithi139 ceph-mon[129293]: pgmap v151: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2023-07-14T19:26:51.031 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:26:50 smithi098 ceph-mon[127486]: pgmap v151: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2023-07-14T19:26:51.356 INFO:teuthology.orchestra.run.smithi101.stdout:Created osd(s) 7 on host 'smithi101' 2023-07-14T19:26:51.356 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-14T19:26:51.354+0000 7efc0f7fe700 1 -- 172.21.15.101:0/3845646295 <== mgr.14152 v2:172.21.15.98:6800/452042164 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7efc18060a80 con 0x7efc040a1150 2023-07-14T19:26:51.357 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-14T19:26:51.357+0000 7efc20642700 1 -- 172.21.15.101:0/3845646295 >> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] conn(0x7efc040a1150 msgr2=0x7efc040a3600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:26:51.358 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-14T19:26:51.357+0000 7efc20642700 1 --2- 172.21.15.101:0/3845646295 >> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] conn(0x7efc040a1150 0x7efc040a3600 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7efc18100d20 tx=0x7efc0800b410 comp rx=0 tx=0).stop 2023-07-14T19:26:51.358 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-14T19:26:51.357+0000 7efc20642700 1 -- 172.21.15.101:0/3845646295 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7efc18100ec0 msgr2=0x7efc18195f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:26:51.358 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-14T19:26:51.357+0000 7efc20642700 1 --2- 172.21.15.101:0/3845646295 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7efc18100ec0 0x7efc18195f60 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7efc1400c9b0 tx=0x7efc1400ccc0 comp rx=0 tx=0).stop 2023-07-14T19:26:51.358 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-14T19:26:51.357+0000 7efc20642700 1 -- 172.21.15.101:0/3845646295 shutdown_connections 2023-07-14T19:26:51.358 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-14T19:26:51.357+0000 7efc20642700 1 --2- 172.21.15.101:0/3845646295 >> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] conn(0x7efc040a1150 0x7efc040a3600 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:26:51.359 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-14T19:26:51.357+0000 7efc20642700 1 --2- 172.21.15.101:0/3845646295 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7efc18101850 0x7efc1819cfe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:26:51.359 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-14T19:26:51.357+0000 7efc20642700 1 --2- 172.21.15.101:0/3845646295 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7efc180ffcc0 0x7efc18195a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:26:51.359 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-14T19:26:51.357+0000 7efc20642700 1 --2- 172.21.15.101:0/3845646295 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7efc18100ec0 0x7efc18195f60 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:26:51.359 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-14T19:26:51.357+0000 7efc20642700 1 -- 172.21.15.101:0/3845646295 >> 172.21.15.101:0/3845646295 conn(0x7efc180fb2d0 msgr2=0x7efc18104300 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:26:51.359 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-14T19:26:51.357+0000 7efc20642700 1 -- 172.21.15.101:0/3845646295 shutdown_connections 2023-07-14T19:26:51.359 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-14T19:26:51.357+0000 7efc20642700 1 -- 172.21.15.101:0/3845646295 wait complete. 2023-07-14T19:26:51.786 DEBUG:teuthology.orchestra.run.smithi101:osd.7> sudo journalctl -f -n 0 -u ceph-9adea648-227b-11ee-9b33-001a4aab830c@osd.7.service 2023-07-14T19:26:51.789 INFO:tasks.cephadm:Deploying osd.8 on smithi139 with /dev/vg_nvme/lv_2... 2023-07-14T19:26:51.790 DEBUG:teuthology.orchestra.run.smithi139:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81c9387d2e7576da065747ec9c16d1c7d69016ec shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 9adea648-227b-11ee-9b33-001a4aab830c -- ceph-volume lvm zap /dev/vg_nvme/lv_2 2023-07-14T19:26:51.823 INFO:journalctl@ceph.osd.7.smithi101.stdout:-- Logs begin at Fri 2023-07-14 18:56:58 UTC. -- 2023-07-14T19:26:52.191 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:26:52 smithi101 ceph-mon[128807]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' 2023-07-14T19:26:52.192 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:26:52 smithi101 ceph-mon[128807]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' 2023-07-14T19:26:52.192 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:26:52 smithi101 ceph-mon[128807]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T19:26:52.192 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:26:52 smithi101 ceph-mon[128807]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-14T19:26:52.193 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:26:52 smithi101 ceph-mon[128807]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' 2023-07-14T19:26:52.193 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:26:52 smithi101 ceph-mon[128807]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' 2023-07-14T19:26:52.193 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:26:52 smithi101 ceph-mon[128807]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' 2023-07-14T19:26:52.193 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:26:52 smithi101 ceph-mon[128807]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-14T19:26:52.194 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:26:52 smithi101 ceph-mon[128807]: pgmap v152: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2023-07-14T19:26:52.220 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:26:52 smithi139 ceph-mon[129293]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' 2023-07-14T19:26:52.221 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:26:52 smithi139 ceph-mon[129293]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' 2023-07-14T19:26:52.221 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:26:52 smithi139 ceph-mon[129293]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T19:26:52.221 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:26:52 smithi139 ceph-mon[129293]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-14T19:26:52.221 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:26:52 smithi139 ceph-mon[129293]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' 2023-07-14T19:26:52.221 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:26:52 smithi139 ceph-mon[129293]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' 2023-07-14T19:26:52.221 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:26:52 smithi139 ceph-mon[129293]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' 2023-07-14T19:26:52.221 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:26:52 smithi139 ceph-mon[129293]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-14T19:26:52.222 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:26:52 smithi139 ceph-mon[129293]: pgmap v152: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2023-07-14T19:26:52.281 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:26:52 smithi098 ceph-mon[127486]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' 2023-07-14T19:26:52.282 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:26:52 smithi098 ceph-mon[127486]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' 2023-07-14T19:26:52.282 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:26:52 smithi098 ceph-mon[127486]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T19:26:52.282 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:26:52 smithi098 ceph-mon[127486]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-14T19:26:52.283 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:26:52 smithi098 ceph-mon[127486]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' 2023-07-14T19:26:52.283 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:26:52 smithi098 ceph-mon[127486]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' 2023-07-14T19:26:52.283 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:26:52 smithi098 ceph-mon[127486]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' 2023-07-14T19:26:52.283 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:26:52 smithi098 ceph-mon[127486]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-14T19:26:52.284 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:26:52 smithi098 ceph-mon[127486]: pgmap v152: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2023-07-14T19:26:52.365 INFO:teuthology.orchestra.run.smithi139.stderr:Inferring config /var/lib/ceph/9adea648-227b-11ee-9b33-001a4aab830c/mon.c/config 2023-07-14T19:26:52.645 INFO:journalctl@ceph.osd.7.smithi101.stdout:Jul 14 19:26:52 smithi101 ceph-9adea648-227b-11ee-9b33-001a4aab830c-osd-7[144864]: 2023-07-14T19:26:52.524+0000 7f42ff62e640 -1 osd.7 0 log_to_monitors true 2023-07-14T19:26:53.401 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:26:53 smithi139 ceph-mon[129293]: from='osd.7 [v2:172.21.15.101:6816/2881214066,v1:172.21.15.101:6817/2881214066]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2023-07-14T19:26:53.691 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:26:53 smithi101 ceph-mon[128807]: from='osd.7 [v2:172.21.15.101:6816/2881214066,v1:172.21.15.101:6817/2881214066]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2023-07-14T19:26:53.781 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:26:53 smithi098 ceph-mon[127486]: from='osd.7 [v2:172.21.15.101:6816/2881214066,v1:172.21.15.101:6817/2881214066]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2023-07-14T19:26:53.817 INFO:teuthology.orchestra.run.smithi139.stderr:--> Zapping: /dev/vg_nvme/lv_2 2023-07-14T19:26:53.817 INFO:teuthology.orchestra.run.smithi139.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 conv=fsync 2023-07-14T19:26:53.819 INFO:teuthology.orchestra.run.smithi139.stderr: stderr: 10+0 records in 2023-07-14T19:26:53.819 INFO:teuthology.orchestra.run.smithi139.stderr:10+0 records out 2023-07-14T19:26:53.819 INFO:teuthology.orchestra.run.smithi139.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0112524 s, 932 MB/s 2023-07-14T19:26:53.819 INFO:teuthology.orchestra.run.smithi139.stderr:--> Zapping successful for: 2023-07-14T19:26:54.214 DEBUG:teuthology.orchestra.run.smithi139:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81c9387d2e7576da065747ec9c16d1c7d69016ec shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 9adea648-227b-11ee-9b33-001a4aab830c -- ceph orch daemon add osd smithi139:vg_nvme/lv_2 2023-07-14T19:26:54.355 INFO:journalctl@ceph.osd.7.smithi101.stdout:Jul 14 19:26:54 smithi101 ceph-9adea648-227b-11ee-9b33-001a4aab830c-osd-7[144864]: 2023-07-14T19:26:54.338+0000 7f42f44a5700 -1 osd.7 0 waiting for initial osdmap 2023-07-14T19:26:54.355 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:26:54 smithi101 ceph-mon[128807]: from='osd.7 [v2:172.21.15.101:6816/2881214066,v1:172.21.15.101:6817/2881214066]' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2023-07-14T19:26:54.356 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:26:54 smithi101 ceph-mon[128807]: osdmap e48: 8 total, 7 up, 8 in 2023-07-14T19:26:54.356 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:26:54 smithi101 ceph-mon[128807]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2023-07-14T19:26:54.356 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:26:54 smithi101 ceph-mon[128807]: from='osd.7 [v2:172.21.15.101:6816/2881214066,v1:172.21.15.101:6817/2881214066]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi101", "root=default"]}]: dispatch 2023-07-14T19:26:54.356 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:26:54 smithi101 ceph-mon[128807]: pgmap v154: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2023-07-14T19:26:54.357 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:26:54 smithi101 ceph-mon[128807]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' 2023-07-14T19:26:54.357 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:26:54 smithi101 ceph-mon[128807]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' 2023-07-14T19:26:54.357 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:26:54 smithi101 ceph-mon[128807]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2023-07-14T19:26:54.357 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:26:54 smithi101 ceph-mon[128807]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2023-07-14T19:26:54.357 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:26:54 smithi101 ceph-mon[128807]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2023-07-14T19:26:54.358 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:26:54 smithi101 ceph-mon[128807]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' 2023-07-14T19:26:54.358 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:26:54 smithi101 ceph-mon[128807]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T19:26:54.358 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:26:54 smithi101 ceph-mon[128807]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-14T19:26:54.358 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:26:54 smithi101 ceph-mon[128807]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' 2023-07-14T19:26:54.470 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:26:54 smithi139 ceph-mon[129293]: from='osd.7 [v2:172.21.15.101:6816/2881214066,v1:172.21.15.101:6817/2881214066]' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2023-07-14T19:26:54.471 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:26:54 smithi139 ceph-mon[129293]: osdmap e48: 8 total, 7 up, 8 in 2023-07-14T19:26:54.471 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:26:54 smithi139 ceph-mon[129293]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2023-07-14T19:26:54.471 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:26:54 smithi139 ceph-mon[129293]: from='osd.7 [v2:172.21.15.101:6816/2881214066,v1:172.21.15.101:6817/2881214066]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi101", "root=default"]}]: dispatch 2023-07-14T19:26:54.471 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:26:54 smithi139 ceph-mon[129293]: pgmap v154: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2023-07-14T19:26:54.471 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:26:54 smithi139 ceph-mon[129293]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' 2023-07-14T19:26:54.471 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:26:54 smithi139 ceph-mon[129293]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' 2023-07-14T19:26:54.471 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:26:54 smithi139 ceph-mon[129293]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2023-07-14T19:26:54.472 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:26:54 smithi139 ceph-mon[129293]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2023-07-14T19:26:54.472 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:26:54 smithi139 ceph-mon[129293]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2023-07-14T19:26:54.472 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:26:54 smithi139 ceph-mon[129293]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' 2023-07-14T19:26:54.472 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:26:54 smithi139 ceph-mon[129293]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T19:26:54.472 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:26:54 smithi139 ceph-mon[129293]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-14T19:26:54.472 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:26:54 smithi139 ceph-mon[129293]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' 2023-07-14T19:26:54.691 INFO:journalctl@ceph.osd.7.smithi101.stdout:Jul 14 19:26:54 smithi101 ceph-9adea648-227b-11ee-9b33-001a4aab830c-osd-7[144864]: 2023-07-14T19:26:54.353+0000 7f42eda96700 -1 osd.7 49 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-07-14T19:26:54.765 INFO:teuthology.orchestra.run.smithi139.stderr:Inferring config /var/lib/ceph/9adea648-227b-11ee-9b33-001a4aab830c/mon.c/config 2023-07-14T19:26:54.781 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:26:54 smithi098 ceph-mon[127486]: from='osd.7 [v2:172.21.15.101:6816/2881214066,v1:172.21.15.101:6817/2881214066]' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2023-07-14T19:26:54.782 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:26:54 smithi098 ceph-mon[127486]: osdmap e48: 8 total, 7 up, 8 in 2023-07-14T19:26:54.782 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:26:54 smithi098 ceph-mon[127486]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2023-07-14T19:26:54.782 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:26:54 smithi098 ceph-mon[127486]: from='osd.7 [v2:172.21.15.101:6816/2881214066,v1:172.21.15.101:6817/2881214066]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi101", "root=default"]}]: dispatch 2023-07-14T19:26:54.782 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:26:54 smithi098 ceph-mon[127486]: pgmap v154: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2023-07-14T19:26:54.782 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:26:54 smithi098 ceph-mon[127486]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' 2023-07-14T19:26:54.782 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:26:54 smithi098 ceph-mon[127486]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' 2023-07-14T19:26:54.783 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:26:54 smithi098 ceph-mon[127486]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2023-07-14T19:26:54.783 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:26:54 smithi098 ceph-mon[127486]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2023-07-14T19:26:54.783 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:26:54 smithi098 ceph-mon[127486]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2023-07-14T19:26:54.783 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:26:54 smithi098 ceph-mon[127486]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' 2023-07-14T19:26:54.783 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:26:54 smithi098 ceph-mon[127486]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T19:26:54.784 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:26:54 smithi098 ceph-mon[127486]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-14T19:26:54.784 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:26:54 smithi098 ceph-mon[127486]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' 2023-07-14T19:26:55.691 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:26:55 smithi101 ceph-mon[128807]: Detected new or changed devices on smithi101 2023-07-14T19:26:55.691 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:26:55 smithi101 ceph-mon[128807]: Adjusting osd_memory_target on smithi101 to 5722M 2023-07-14T19:26:55.692 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:26:55 smithi101 ceph-mon[128807]: from='osd.7 [v2:172.21.15.101:6816/2881214066,v1:172.21.15.101:6817/2881214066]' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi101", "root=default"]}]': finished 2023-07-14T19:26:55.692 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:26:55 smithi101 ceph-mon[128807]: osdmap e49: 8 total, 7 up, 8 in 2023-07-14T19:26:55.692 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:26:55 smithi101 ceph-mon[128807]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2023-07-14T19:26:55.692 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:26:55 smithi101 ceph-mon[128807]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2023-07-14T19:26:55.692 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:26:55 smithi101 ceph-mon[128807]: osd.7 [v2:172.21.15.101:6816/2881214066,v1:172.21.15.101:6817/2881214066] boot 2023-07-14T19:26:55.692 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:26:55 smithi101 ceph-mon[128807]: osdmap e50: 8 total, 8 up, 8 in 2023-07-14T19:26:55.693 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:26:55 smithi101 ceph-mon[128807]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2023-07-14T19:26:55.720 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:26:55 smithi139 ceph-mon[129293]: Detected new or changed devices on smithi101 2023-07-14T19:26:55.721 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:26:55 smithi139 ceph-mon[129293]: Adjusting osd_memory_target on smithi101 to 5722M 2023-07-14T19:26:55.721 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:26:55 smithi139 ceph-mon[129293]: from='osd.7 [v2:172.21.15.101:6816/2881214066,v1:172.21.15.101:6817/2881214066]' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi101", "root=default"]}]': finished 2023-07-14T19:26:55.721 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:26:55 smithi139 ceph-mon[129293]: osdmap e49: 8 total, 7 up, 8 in 2023-07-14T19:26:55.721 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:26:55 smithi139 ceph-mon[129293]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2023-07-14T19:26:55.721 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:26:55 smithi139 ceph-mon[129293]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2023-07-14T19:26:55.722 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:26:55 smithi139 ceph-mon[129293]: osd.7 [v2:172.21.15.101:6816/2881214066,v1:172.21.15.101:6817/2881214066] boot 2023-07-14T19:26:55.722 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:26:55 smithi139 ceph-mon[129293]: osdmap e50: 8 total, 8 up, 8 in 2023-07-14T19:26:55.722 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:26:55 smithi139 ceph-mon[129293]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2023-07-14T19:26:55.781 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:26:55 smithi098 ceph-mon[127486]: Detected new or changed devices on smithi101 2023-07-14T19:26:55.782 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:26:55 smithi098 ceph-mon[127486]: Adjusting osd_memory_target on smithi101 to 5722M 2023-07-14T19:26:55.782 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:26:55 smithi098 ceph-mon[127486]: from='osd.7 [v2:172.21.15.101:6816/2881214066,v1:172.21.15.101:6817/2881214066]' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi101", "root=default"]}]': finished 2023-07-14T19:26:55.782 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:26:55 smithi098 ceph-mon[127486]: osdmap e49: 8 total, 7 up, 8 in 2023-07-14T19:26:55.783 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:26:55 smithi098 ceph-mon[127486]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2023-07-14T19:26:55.783 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:26:55 smithi098 ceph-mon[127486]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2023-07-14T19:26:55.783 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:26:55 smithi098 ceph-mon[127486]: osd.7 [v2:172.21.15.101:6816/2881214066,v1:172.21.15.101:6817/2881214066] boot 2023-07-14T19:26:55.783 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:26:55 smithi098 ceph-mon[127486]: osdmap e50: 8 total, 8 up, 8 in 2023-07-14T19:26:55.784 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:26:55 smithi098 ceph-mon[127486]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2023-07-14T19:26:55.937 INFO:teuthology.orchestra.run.smithi139.stderr:2023-07-14T19:26:55.937+0000 7f7e43812700 1 -- 172.21.15.139:0/257635625 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f7e3c0ffdb0 msgr2=0x7f7e3c100200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:26:55.938 INFO:teuthology.orchestra.run.smithi139.stderr:2023-07-14T19:26:55.937+0000 7f7e43812700 1 --2- 172.21.15.139:0/257635625 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f7e3c0ffdb0 0x7f7e3c100200 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f7e2c009990 tx=0x7f7e2c009ca0 comp rx=0 tx=0).stop 2023-07-14T19:26:55.938 INFO:teuthology.orchestra.run.smithi139.stderr:2023-07-14T19:26:55.937+0000 7f7e43812700 1 -- 172.21.15.139:0/257635625 shutdown_connections 2023-07-14T19:26:55.938 INFO:teuthology.orchestra.run.smithi139.stderr:2023-07-14T19:26:55.937+0000 7f7e43812700 1 --2- 172.21.15.139:0/257635625 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f7e3c100740 0x7f7e3c107000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:26:55.938 INFO:teuthology.orchestra.run.smithi139.stderr:2023-07-14T19:26:55.937+0000 7f7e43812700 1 --2- 172.21.15.139:0/257635625 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7e3c0febb0 0x7f7e3c0fefc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:26:55.938 INFO:teuthology.orchestra.run.smithi139.stderr:2023-07-14T19:26:55.937+0000 7f7e43812700 1 --2- 172.21.15.139:0/257635625 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f7e3c0ffdb0 0x7f7e3c100200 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:26:55.938 INFO:teuthology.orchestra.run.smithi139.stderr:2023-07-14T19:26:55.937+0000 7f7e43812700 1 -- 172.21.15.139:0/257635625 >> 172.21.15.139:0/257635625 conn(0x7f7e3c0fa1e0 msgr2=0x7f7e3c0fc630 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:26:55.939 INFO:teuthology.orchestra.run.smithi139.stderr:2023-07-14T19:26:55.937+0000 7f7e43812700 1 -- 172.21.15.139:0/257635625 shutdown_connections 2023-07-14T19:26:55.939 INFO:teuthology.orchestra.run.smithi139.stderr:2023-07-14T19:26:55.937+0000 7f7e43812700 1 -- 172.21.15.139:0/257635625 wait complete. 2023-07-14T19:26:55.939 INFO:teuthology.orchestra.run.smithi139.stderr:2023-07-14T19:26:55.937+0000 7f7e43812700 1 Processor -- start 2023-07-14T19:26:55.939 INFO:teuthology.orchestra.run.smithi139.stderr:2023-07-14T19:26:55.937+0000 7f7e43812700 1 -- start start 2023-07-14T19:26:55.940 INFO:teuthology.orchestra.run.smithi139.stderr:2023-07-14T19:26:55.938+0000 7f7e43812700 1 --2- >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f7e3c0febb0 0x7f7e3c1948a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:26:55.941 INFO:teuthology.orchestra.run.smithi139.stderr:2023-07-14T19:26:55.938+0000 7f7e43812700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7e3c0ffdb0 0x7f7e3c194de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:26:55.941 INFO:teuthology.orchestra.run.smithi139.stderr:2023-07-14T19:26:55.938+0000 7f7e43812700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f7e3c100740 0x7f7e3c19be60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:26:55.941 INFO:teuthology.orchestra.run.smithi139.stderr:2023-07-14T19:26:55.938+0000 7f7e43812700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7e3c19c430 con 0x7f7e3c0ffdb0 2023-07-14T19:26:55.941 INFO:teuthology.orchestra.run.smithi139.stderr:2023-07-14T19:26:55.938+0000 7f7e43812700 1 -- --> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7e3c19c570 con 0x7f7e3c0febb0 2023-07-14T19:26:55.941 INFO:teuthology.orchestra.run.smithi139.stderr:2023-07-14T19:26:55.938+0000 7f7e43812700 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7e3c068180 con 0x7f7e3c100740 2023-07-14T19:26:55.941 INFO:teuthology.orchestra.run.smithi139.stderr:2023-07-14T19:26:55.938+0000 7f7e41daf700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f7e3c100740 0x7f7e3c19be60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:26:55.941 INFO:teuthology.orchestra.run.smithi139.stderr:2023-07-14T19:26:55.938+0000 7f7e41daf700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f7e3c100740 0x7f7e3c19be60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.139:3300/0 says I am v2:172.21.15.139:37292/0 (socket says 172.21.15.139:37292) 2023-07-14T19:26:55.942 INFO:teuthology.orchestra.run.smithi139.stderr:2023-07-14T19:26:55.938+0000 7f7e41daf700 1 -- 172.21.15.139:0/1570811873 learned_addr learned my addr 172.21.15.139:0/1570811873 (peer_addr_for_me v2:172.21.15.139:0/0) 2023-07-14T19:26:55.942 INFO:teuthology.orchestra.run.smithi139.stderr:2023-07-14T19:26:55.938+0000 7f7e40dad700 1 --2- 172.21.15.139:0/1570811873 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7e3c0ffdb0 0x7f7e3c194de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:26:55.942 INFO:teuthology.orchestra.run.smithi139.stderr:2023-07-14T19:26:55.938+0000 7f7e415ae700 1 --2- 172.21.15.139:0/1570811873 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f7e3c0febb0 0x7f7e3c1948a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:26:55.942 INFO:teuthology.orchestra.run.smithi139.stderr:2023-07-14T19:26:55.938+0000 7f7e41daf700 1 -- 172.21.15.139:0/1570811873 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f7e3c0febb0 msgr2=0x7f7e3c1948a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:26:55.942 INFO:teuthology.orchestra.run.smithi139.stderr:2023-07-14T19:26:55.938+0000 7f7e41daf700 1 --2- 172.21.15.139:0/1570811873 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f7e3c0febb0 0x7f7e3c1948a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:26:55.942 INFO:teuthology.orchestra.run.smithi139.stderr:2023-07-14T19:26:55.938+0000 7f7e41daf700 1 -- 172.21.15.139:0/1570811873 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7e3c0ffdb0 msgr2=0x7f7e3c194de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:26:55.942 INFO:teuthology.orchestra.run.smithi139.stderr:2023-07-14T19:26:55.938+0000 7f7e41daf700 1 --2- 172.21.15.139:0/1570811873 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7e3c0ffdb0 0x7f7e3c194de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:26:55.943 INFO:teuthology.orchestra.run.smithi139.stderr:2023-07-14T19:26:55.938+0000 7f7e41daf700 1 -- 172.21.15.139:0/1570811873 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7e2c0095f0 con 0x7f7e3c100740 2023-07-14T19:26:55.943 INFO:teuthology.orchestra.run.smithi139.stderr:2023-07-14T19:26:55.938+0000 7f7e40dad700 1 --2- 172.21.15.139:0/1570811873 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7e3c0ffdb0 0x7f7e3c194de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-07-14T19:26:55.943 INFO:teuthology.orchestra.run.smithi139.stderr:2023-07-14T19:26:55.938+0000 7f7e41daf700 1 --2- 172.21.15.139:0/1570811873 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f7e3c100740 0x7f7e3c19be60 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f7e3400ada0 tx=0x7f7e3400c330 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:26:55.943 INFO:teuthology.orchestra.run.smithi139.stderr:2023-07-14T19:26:55.938+0000 7f7e327fc700 1 -- 172.21.15.139:0/1570811873 <== mon.1 v2:172.21.15.139:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e3400f070 con 0x7f7e3c100740 2023-07-14T19:26:55.943 INFO:teuthology.orchestra.run.smithi139.stderr:2023-07-14T19:26:55.938+0000 7f7e43812700 1 -- 172.21.15.139:0/1570811873 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7e3c068460 con 0x7f7e3c100740 2023-07-14T19:26:55.943 INFO:teuthology.orchestra.run.smithi139.stderr:2023-07-14T19:26:55.938+0000 7f7e327fc700 1 -- 172.21.15.139:0/1570811873 <== mon.1 v2:172.21.15.139:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f7e34007b30 con 0x7f7e3c100740 2023-07-14T19:26:55.943 INFO:teuthology.orchestra.run.smithi139.stderr:2023-07-14T19:26:55.938+0000 7f7e327fc700 1 -- 172.21.15.139:0/1570811873 <== mon.1 v2:172.21.15.139:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e34005440 con 0x7f7e3c100740 2023-07-14T19:26:55.944 INFO:teuthology.orchestra.run.smithi139.stderr:2023-07-14T19:26:55.938+0000 7f7e43812700 1 -- 172.21.15.139:0/1570811873 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7e3c068910 con 0x7f7e3c100740 2023-07-14T19:26:55.944 INFO:teuthology.orchestra.run.smithi139.stderr:2023-07-14T19:26:55.939+0000 7f7e43812700 1 -- 172.21.15.139:0/1570811873 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7e3c10a370 con 0x7f7e3c100740 2023-07-14T19:26:55.944 INFO:teuthology.orchestra.run.smithi139.stderr:2023-07-14T19:26:55.941+0000 7f7e327fc700 1 -- 172.21.15.139:0/1570811873 <== mon.1 v2:172.21.15.139:3300/0 4 ==== mgrmap(e 16) v1 ==== 135715+0+0 (secure 0 0 0) 0x7f7e34004ad0 con 0x7f7e3c100740 2023-07-14T19:26:55.945 INFO:teuthology.orchestra.run.smithi139.stderr:2023-07-14T19:26:55.941+0000 7f7e327fc700 1 --2- 172.21.15.139:0/1570811873 >> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] conn(0x7f7e280a11d0 0x7f7e280a3680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:26:55.945 INFO:teuthology.orchestra.run.smithi139.stderr:2023-07-14T19:26:55.941+0000 7f7e327fc700 1 -- 172.21.15.139:0/1570811873 <== mon.1 v2:172.21.15.139:3300/0 5 ==== osd_map(50..50 src has 1..50) v4 ==== 5430+0+0 (secure 0 0 0) 0x7f7e340cbb10 con 0x7f7e3c100740 2023-07-14T19:26:55.945 INFO:teuthology.orchestra.run.smithi139.stderr:2023-07-14T19:26:55.941+0000 7f7e415ae700 1 --2- 172.21.15.139:0/1570811873 >> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] conn(0x7f7e280a11d0 0x7f7e280a3680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:26:55.945 INFO:teuthology.orchestra.run.smithi139.stderr:2023-07-14T19:26:55.942+0000 7f7e415ae700 1 --2- 172.21.15.139:0/1570811873 >> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] conn(0x7f7e280a11d0 0x7f7e280a3680 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f7e38005950 tx=0x7f7e380058e0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:26:55.946 INFO:teuthology.orchestra.run.smithi139.stderr:2023-07-14T19:26:55.944+0000 7f7e327fc700 1 -- 172.21.15.139:0/1570811873 <== mon.1 v2:172.21.15.139:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f7e34095d90 con 0x7f7e3c100740 2023-07-14T19:26:56.138 INFO:teuthology.orchestra.run.smithi139.stderr:2023-07-14T19:26:56.137+0000 7f7e43812700 1 -- 172.21.15.139:0/1570811873 --> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi139:vg_nvme/lv_2", "target": ["mon-mgr", ""]}) v1 -- 0x7f7e3c060a80 con 0x7f7e280a11d0 2023-07-14T19:26:56.691 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:26:56 smithi101 ceph-mon[128807]: purged_snaps scrub starts 2023-07-14T19:26:56.692 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:26:56 smithi101 ceph-mon[128807]: purged_snaps scrub ok 2023-07-14T19:26:56.692 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:26:56 smithi101 ceph-mon[128807]: pgmap v157: 1 pgs: 1 active+clean; 577 KiB data, 216 MiB used, 715 GiB / 715 GiB avail 2023-07-14T19:26:56.692 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:26:56 smithi101 ceph-mon[128807]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-07-14T19:26:56.692 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:26:56 smithi101 ceph-mon[128807]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-07-14T19:26:56.692 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:26:56 smithi101 ceph-mon[128807]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T19:26:56.692 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:26:56 smithi101 ceph-mon[128807]: osdmap e51: 8 total, 8 up, 8 in 2023-07-14T19:26:56.720 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:26:56 smithi139 ceph-mon[129293]: purged_snaps scrub starts 2023-07-14T19:26:56.721 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:26:56 smithi139 ceph-mon[129293]: purged_snaps scrub ok 2023-07-14T19:26:56.721 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:26:56 smithi139 ceph-mon[129293]: pgmap v157: 1 pgs: 1 active+clean; 577 KiB data, 216 MiB used, 715 GiB / 715 GiB avail 2023-07-14T19:26:56.721 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:26:56 smithi139 ceph-mon[129293]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-07-14T19:26:56.721 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:26:56 smithi139 ceph-mon[129293]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-07-14T19:26:56.721 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:26:56 smithi139 ceph-mon[129293]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T19:26:56.722 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:26:56 smithi139 ceph-mon[129293]: osdmap e51: 8 total, 8 up, 8 in 2023-07-14T19:26:56.781 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:26:56 smithi098 ceph-mon[127486]: purged_snaps scrub starts 2023-07-14T19:26:56.782 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:26:56 smithi098 ceph-mon[127486]: purged_snaps scrub ok 2023-07-14T19:26:56.782 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:26:56 smithi098 ceph-mon[127486]: pgmap v157: 1 pgs: 1 active+clean; 577 KiB data, 216 MiB used, 715 GiB / 715 GiB avail 2023-07-14T19:26:56.782 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:26:56 smithi098 ceph-mon[127486]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-07-14T19:26:56.782 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:26:56 smithi098 ceph-mon[127486]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-07-14T19:26:56.783 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:26:56 smithi098 ceph-mon[127486]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T19:26:56.783 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:26:56 smithi098 ceph-mon[127486]: osdmap e51: 8 total, 8 up, 8 in 2023-07-14T19:26:57.691 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:26:57 smithi101 ceph-mon[128807]: from='client.24286 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi139:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2023-07-14T19:26:57.691 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:26:57 smithi101 ceph-mon[128807]: osdmap e52: 8 total, 8 up, 8 in 2023-07-14T19:26:57.715 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:26:57 smithi139 ceph-mon[129293]: from='client.24286 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi139:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2023-07-14T19:26:57.715 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:26:57 smithi139 ceph-mon[129293]: osdmap e52: 8 total, 8 up, 8 in 2023-07-14T19:26:57.781 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:26:57 smithi098 ceph-mon[127486]: from='client.24286 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi139:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2023-07-14T19:26:57.781 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:26:57 smithi098 ceph-mon[127486]: osdmap e52: 8 total, 8 up, 8 in 2023-07-14T19:26:58.691 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:26:58 smithi101 ceph-mon[128807]: pgmap v160: 1 pgs: 1 remapped+peering; 577 KiB data, 216 MiB used, 715 GiB / 715 GiB avail 2023-07-14T19:26:58.720 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:26:58 smithi139 ceph-mon[129293]: pgmap v160: 1 pgs: 1 remapped+peering; 577 KiB data, 216 MiB used, 715 GiB / 715 GiB avail 2023-07-14T19:26:58.781 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:26:58 smithi098 ceph-mon[127486]: pgmap v160: 1 pgs: 1 remapped+peering; 577 KiB data, 216 MiB used, 715 GiB / 715 GiB avail 2023-07-14T19:26:59.691 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:26:59 smithi101 ceph-mon[128807]: osdmap e53: 8 total, 8 up, 8 in 2023-07-14T19:26:59.692 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:26:59 smithi101 ceph-mon[128807]: from='client.? 172.21.15.139:0/3553859210' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "6bfd77c2-5770-466e-aa6f-84b50c74ffb6"}]: dispatch 2023-07-14T19:26:59.692 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:26:59 smithi101 ceph-mon[128807]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "6bfd77c2-5770-466e-aa6f-84b50c74ffb6"}]: dispatch 2023-07-14T19:26:59.693 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:26:59 smithi101 ceph-mon[128807]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "6bfd77c2-5770-466e-aa6f-84b50c74ffb6"}]': finished 2023-07-14T19:26:59.693 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:26:59 smithi101 ceph-mon[128807]: osdmap e54: 9 total, 8 up, 9 in 2023-07-14T19:26:59.693 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:26:59 smithi101 ceph-mon[128807]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2023-07-14T19:26:59.694 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:26:59 smithi101 ceph-mon[128807]: from='client.? 172.21.15.139:0/4284267155' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-07-14T19:26:59.720 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:26:59 smithi139 ceph-mon[129293]: osdmap e53: 8 total, 8 up, 8 in 2023-07-14T19:26:59.721 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:26:59 smithi139 ceph-mon[129293]: from='client.? 172.21.15.139:0/3553859210' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "6bfd77c2-5770-466e-aa6f-84b50c74ffb6"}]: dispatch 2023-07-14T19:26:59.721 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:26:59 smithi139 ceph-mon[129293]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "6bfd77c2-5770-466e-aa6f-84b50c74ffb6"}]: dispatch 2023-07-14T19:26:59.721 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:26:59 smithi139 ceph-mon[129293]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "6bfd77c2-5770-466e-aa6f-84b50c74ffb6"}]': finished 2023-07-14T19:26:59.721 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:26:59 smithi139 ceph-mon[129293]: osdmap e54: 9 total, 8 up, 9 in 2023-07-14T19:26:59.721 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:26:59 smithi139 ceph-mon[129293]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2023-07-14T19:26:59.722 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:26:59 smithi139 ceph-mon[129293]: from='client.? 172.21.15.139:0/4284267155' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-07-14T19:26:59.781 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:26:59 smithi098 ceph-mon[127486]: osdmap e53: 8 total, 8 up, 8 in 2023-07-14T19:26:59.781 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:26:59 smithi098 ceph-mon[127486]: from='client.? 172.21.15.139:0/3553859210' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "6bfd77c2-5770-466e-aa6f-84b50c74ffb6"}]: dispatch 2023-07-14T19:26:59.782 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:26:59 smithi098 ceph-mon[127486]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "6bfd77c2-5770-466e-aa6f-84b50c74ffb6"}]: dispatch 2023-07-14T19:26:59.782 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:26:59 smithi098 ceph-mon[127486]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "6bfd77c2-5770-466e-aa6f-84b50c74ffb6"}]': finished 2023-07-14T19:26:59.782 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:26:59 smithi098 ceph-mon[127486]: osdmap e54: 9 total, 8 up, 9 in 2023-07-14T19:26:59.782 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:26:59 smithi098 ceph-mon[127486]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2023-07-14T19:26:59.782 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:26:59 smithi098 ceph-mon[127486]: from='client.? 172.21.15.139:0/4284267155' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-07-14T19:27:00.691 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:27:00 smithi101 ceph-mon[128807]: pgmap v163: 1 pgs: 1 remapped+peering; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2023-07-14T19:27:00.720 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:27:00 smithi139 ceph-mon[129293]: pgmap v163: 1 pgs: 1 remapped+peering; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2023-07-14T19:27:00.781 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:27:00 smithi098 ceph-mon[127486]: pgmap v163: 1 pgs: 1 remapped+peering; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2023-07-14T19:27:02.531 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:27:02 smithi098 ceph-mon[127486]: pgmap v164: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2023-07-14T19:27:02.666 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:27:02 smithi139 ceph-mon[129293]: pgmap v164: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2023-07-14T19:27:02.691 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:27:02 smithi101 ceph-mon[128807]: pgmap v164: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2023-07-14T19:27:04.941 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:27:04 smithi101 ceph-mon[128807]: pgmap v165: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2023-07-14T19:27:04.971 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:27:04 smithi139 ceph-mon[129293]: pgmap v165: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2023-07-14T19:27:05.031 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:27:04 smithi098 ceph-mon[127486]: pgmap v165: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2023-07-14T19:27:05.941 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:27:05 smithi101 ceph-mon[128807]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.8"}]: dispatch 2023-07-14T19:27:05.941 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:27:05 smithi101 ceph-mon[128807]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T19:27:05.947 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:27:05 smithi139 ceph-mon[129293]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.8"}]: dispatch 2023-07-14T19:27:05.947 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:27:05 smithi139 ceph-mon[129293]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T19:27:06.031 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:27:05 smithi098 ceph-mon[127486]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.8"}]: dispatch 2023-07-14T19:27:06.031 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:27:05 smithi098 ceph-mon[127486]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T19:27:06.721 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:27:06 smithi139 ceph-mon[129293]: Deploying daemon osd.8 on smithi139 2023-07-14T19:27:06.721 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:27:06 smithi139 ceph-mon[129293]: pgmap v166: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2023-07-14T19:27:06.941 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:27:06 smithi101 ceph-mon[128807]: Deploying daemon osd.8 on smithi139 2023-07-14T19:27:06.941 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:27:06 smithi101 ceph-mon[128807]: pgmap v166: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2023-07-14T19:27:07.031 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:27:06 smithi098 ceph-mon[127486]: Deploying daemon osd.8 on smithi139 2023-07-14T19:27:07.031 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:27:06 smithi098 ceph-mon[127486]: pgmap v166: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2023-07-14T19:27:08.721 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:27:08 smithi139 ceph-mon[129293]: pgmap v167: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2023-07-14T19:27:08.941 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:27:08 smithi101 ceph-mon[128807]: pgmap v167: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2023-07-14T19:27:09.031 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:27:08 smithi098 ceph-mon[127486]: pgmap v167: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2023-07-14T19:27:10.470 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:27:10 smithi139 ceph-mon[129293]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' 2023-07-14T19:27:10.470 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:27:10 smithi139 ceph-mon[129293]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' 2023-07-14T19:27:10.471 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:27:10 smithi139 ceph-mon[129293]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-14T19:27:10.471 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:27:10 smithi139 ceph-mon[129293]: pgmap v168: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2023-07-14T19:27:10.691 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:27:10 smithi101 ceph-mon[128807]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' 2023-07-14T19:27:10.692 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:27:10 smithi101 ceph-mon[128807]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' 2023-07-14T19:27:10.692 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:27:10 smithi101 ceph-mon[128807]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-14T19:27:10.692 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:27:10 smithi101 ceph-mon[128807]: pgmap v168: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2023-07-14T19:27:10.781 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:27:10 smithi098 ceph-mon[127486]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' 2023-07-14T19:27:10.781 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:27:10 smithi098 ceph-mon[127486]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' 2023-07-14T19:27:10.782 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:27:10 smithi098 ceph-mon[127486]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-14T19:27:10.782 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:27:10 smithi098 ceph-mon[127486]: pgmap v168: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2023-07-14T19:27:12.045 INFO:teuthology.orchestra.run.smithi139.stderr:2023-07-14T19:27:12.044+0000 7f7e327fc700 1 -- 172.21.15.139:0/1570811873 <== mgr.14152 v2:172.21.15.98:6800/452042164 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7f7e3c060a80 con 0x7f7e280a11d0 2023-07-14T19:27:12.045 INFO:teuthology.orchestra.run.smithi139.stdout:Created osd(s) 8 on host 'smithi139' 2023-07-14T19:27:12.047 INFO:teuthology.orchestra.run.smithi139.stderr:2023-07-14T19:27:12.046+0000 7f7e43812700 1 -- 172.21.15.139:0/1570811873 >> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] conn(0x7f7e280a11d0 msgr2=0x7f7e280a3680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:27:12.047 INFO:teuthology.orchestra.run.smithi139.stderr:2023-07-14T19:27:12.046+0000 7f7e43812700 1 --2- 172.21.15.139:0/1570811873 >> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] conn(0x7f7e280a11d0 0x7f7e280a3680 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f7e38005950 tx=0x7f7e380058e0 comp rx=0 tx=0).stop 2023-07-14T19:27:12.047 INFO:teuthology.orchestra.run.smithi139.stderr:2023-07-14T19:27:12.046+0000 7f7e43812700 1 -- 172.21.15.139:0/1570811873 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f7e3c100740 msgr2=0x7f7e3c19be60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:27:12.047 INFO:teuthology.orchestra.run.smithi139.stderr:2023-07-14T19:27:12.046+0000 7f7e43812700 1 --2- 172.21.15.139:0/1570811873 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f7e3c100740 0x7f7e3c19be60 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f7e3400ada0 tx=0x7f7e3400c330 comp rx=0 tx=0).stop 2023-07-14T19:27:12.048 INFO:teuthology.orchestra.run.smithi139.stderr:2023-07-14T19:27:12.046+0000 7f7e43812700 1 -- 172.21.15.139:0/1570811873 shutdown_connections 2023-07-14T19:27:12.048 INFO:teuthology.orchestra.run.smithi139.stderr:2023-07-14T19:27:12.046+0000 7f7e43812700 1 --2- 172.21.15.139:0/1570811873 >> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] conn(0x7f7e280a11d0 0x7f7e280a3680 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:27:12.048 INFO:teuthology.orchestra.run.smithi139.stderr:2023-07-14T19:27:12.046+0000 7f7e43812700 1 --2- 172.21.15.139:0/1570811873 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f7e3c100740 0x7f7e3c19be60 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:27:12.048 INFO:teuthology.orchestra.run.smithi139.stderr:2023-07-14T19:27:12.046+0000 7f7e43812700 1 --2- 172.21.15.139:0/1570811873 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f7e3c0febb0 0x7f7e3c1948a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:27:12.048 INFO:teuthology.orchestra.run.smithi139.stderr:2023-07-14T19:27:12.046+0000 7f7e43812700 1 --2- 172.21.15.139:0/1570811873 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7e3c0ffdb0 0x7f7e3c194de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:27:12.048 INFO:teuthology.orchestra.run.smithi139.stderr:2023-07-14T19:27:12.046+0000 7f7e43812700 1 -- 172.21.15.139:0/1570811873 >> 172.21.15.139:0/1570811873 conn(0x7f7e3c0fa1e0 msgr2=0x7f7e3c0fc480 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:27:12.048 INFO:teuthology.orchestra.run.smithi139.stderr:2023-07-14T19:27:12.046+0000 7f7e43812700 1 -- 172.21.15.139:0/1570811873 shutdown_connections 2023-07-14T19:27:12.049 INFO:teuthology.orchestra.run.smithi139.stderr:2023-07-14T19:27:12.046+0000 7f7e43812700 1 -- 172.21.15.139:0/1570811873 wait complete. 2023-07-14T19:27:12.542 DEBUG:teuthology.orchestra.run.smithi139:osd.8> sudo journalctl -f -n 0 -u ceph-9adea648-227b-11ee-9b33-001a4aab830c@osd.8.service 2023-07-14T19:27:12.544 INFO:tasks.cephadm:Deploying osd.9 on smithi098 with /dev/vg_nvme/lv_1... 2023-07-14T19:27:12.545 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81c9387d2e7576da065747ec9c16d1c7d69016ec shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 9adea648-227b-11ee-9b33-001a4aab830c -- ceph-volume lvm zap /dev/vg_nvme/lv_1 2023-07-14T19:27:12.575 INFO:journalctl@ceph.osd.8.smithi139.stdout:-- Logs begin at Fri 2023-07-14 18:56:10 UTC. -- 2023-07-14T19:27:12.936 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:27:12 smithi139 ceph-mon[129293]: pgmap v169: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2023-07-14T19:27:12.937 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:27:12 smithi139 ceph-mon[129293]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' 2023-07-14T19:27:12.937 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:27:12 smithi139 ceph-mon[129293]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' 2023-07-14T19:27:12.937 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:27:12 smithi139 ceph-mon[129293]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T19:27:12.937 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:27:12 smithi139 ceph-mon[129293]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-14T19:27:12.937 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:27:12 smithi139 ceph-mon[129293]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' 2023-07-14T19:27:12.938 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:27:12 smithi139 ceph-mon[129293]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' 2023-07-14T19:27:12.938 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:27:12 smithi139 ceph-mon[129293]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' 2023-07-14T19:27:12.938 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:27:12 smithi139 ceph-mon[129293]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-14T19:27:13.031 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:27:12 smithi098 ceph-mon[127486]: pgmap v169: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2023-07-14T19:27:13.032 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:27:12 smithi098 ceph-mon[127486]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' 2023-07-14T19:27:13.032 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:27:12 smithi098 ceph-mon[127486]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' 2023-07-14T19:27:13.032 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:27:12 smithi098 ceph-mon[127486]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T19:27:13.033 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:27:12 smithi098 ceph-mon[127486]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-14T19:27:13.033 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:27:12 smithi098 ceph-mon[127486]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' 2023-07-14T19:27:13.033 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:27:12 smithi098 ceph-mon[127486]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' 2023-07-14T19:27:13.033 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:27:12 smithi098 ceph-mon[127486]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' 2023-07-14T19:27:13.033 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:27:12 smithi098 ceph-mon[127486]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-14T19:27:13.111 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/9adea648-227b-11ee-9b33-001a4aab830c/mon.a/config 2023-07-14T19:27:13.192 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:27:12 smithi101 ceph-mon[128807]: pgmap v169: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2023-07-14T19:27:13.192 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:27:12 smithi101 ceph-mon[128807]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' 2023-07-14T19:27:13.192 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:27:12 smithi101 ceph-mon[128807]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' 2023-07-14T19:27:13.192 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:27:12 smithi101 ceph-mon[128807]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T19:27:13.193 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:27:12 smithi101 ceph-mon[128807]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-14T19:27:13.193 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:27:12 smithi101 ceph-mon[128807]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' 2023-07-14T19:27:13.193 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:27:12 smithi101 ceph-mon[128807]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' 2023-07-14T19:27:13.193 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:27:12 smithi101 ceph-mon[128807]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' 2023-07-14T19:27:13.193 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:27:12 smithi101 ceph-mon[128807]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-14T19:27:13.720 INFO:journalctl@ceph.osd.8.smithi139.stdout:Jul 14 19:27:13 smithi139 ceph-9adea648-227b-11ee-9b33-001a4aab830c-osd-8[146146]: 2023-07-14T19:27:13.338+0000 7f35e9e3c640 -1 osd.8 0 log_to_monitors true 2023-07-14T19:27:13.903 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:27:13 smithi098 ceph-mon[127486]: from='osd.8 [v2:172.21.15.139:6816/3669514524,v1:172.21.15.139:6817/3669514524]' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2023-07-14T19:27:13.903 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:27:13 smithi098 ceph-mon[127486]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2023-07-14T19:27:14.191 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:27:13 smithi101 ceph-mon[128807]: from='osd.8 [v2:172.21.15.139:6816/3669514524,v1:172.21.15.139:6817/3669514524]' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2023-07-14T19:27:14.191 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:27:13 smithi101 ceph-mon[128807]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2023-07-14T19:27:14.221 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:27:13 smithi139 ceph-mon[129293]: from='osd.8 [v2:172.21.15.139:6816/3669514524,v1:172.21.15.139:6817/3669514524]' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2023-07-14T19:27:14.221 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:27:13 smithi139 ceph-mon[129293]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2023-07-14T19:27:14.581 INFO:teuthology.orchestra.run.smithi098.stderr:--> Zapping: /dev/vg_nvme/lv_1 2023-07-14T19:27:14.582 INFO:teuthology.orchestra.run.smithi098.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_1 bs=1M count=10 conv=fsync 2023-07-14T19:27:14.582 INFO:teuthology.orchestra.run.smithi098.stderr: stderr: 10+0 records in 2023-07-14T19:27:14.582 INFO:teuthology.orchestra.run.smithi098.stderr:10+0 records out 2023-07-14T19:27:14.582 INFO:teuthology.orchestra.run.smithi098.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0125239 s, 837 MB/s 2023-07-14T19:27:14.582 INFO:teuthology.orchestra.run.smithi098.stderr:--> Zapping successful for: 2023-07-14T19:27:14.970 INFO:journalctl@ceph.osd.8.smithi139.stdout:Jul 14 19:27:14 smithi139 ceph-9adea648-227b-11ee-9b33-001a4aab830c-osd-8[146146]: 2023-07-14T19:27:14.801+0000 7f35e04b6700 -1 osd.8 0 waiting for initial osdmap 2023-07-14T19:27:14.970 INFO:journalctl@ceph.osd.8.smithi139.stdout:Jul 14 19:27:14 smithi139 ceph-9adea648-227b-11ee-9b33-001a4aab830c-osd-8[146146]: 2023-07-14T19:27:14.812+0000 7f35d8aa5700 -1 osd.8 56 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-07-14T19:27:14.971 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:27:14 smithi139 ceph-mon[129293]: pgmap v170: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2023-07-14T19:27:14.971 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:27:14 smithi139 ceph-mon[129293]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]': finished 2023-07-14T19:27:14.971 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:27:14 smithi139 ceph-mon[129293]: osdmap e55: 9 total, 8 up, 9 in 2023-07-14T19:27:14.971 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:27:14 smithi139 ceph-mon[129293]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2023-07-14T19:27:14.971 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:27:14 smithi139 ceph-mon[129293]: from='osd.8 [v2:172.21.15.139:6816/3669514524,v1:172.21.15.139:6817/3669514524]' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi139", "root=default"]}]: dispatch 2023-07-14T19:27:14.972 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:27:14 smithi139 ceph-mon[129293]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi139", "root=default"]}]: dispatch 2023-07-14T19:27:14.972 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:27:14 smithi139 ceph-mon[129293]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' 2023-07-14T19:27:14.972 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:27:14 smithi139 ceph-mon[129293]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' 2023-07-14T19:27:14.972 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:27:14 smithi139 ceph-mon[129293]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2023-07-14T19:27:14.972 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:27:14 smithi139 ceph-mon[129293]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2023-07-14T19:27:14.972 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:27:14 smithi139 ceph-mon[129293]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2023-07-14T19:27:14.972 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:27:14 smithi139 ceph-mon[129293]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' 2023-07-14T19:27:14.973 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:27:14 smithi139 ceph-mon[129293]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T19:27:14.973 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:27:14 smithi139 ceph-mon[129293]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-14T19:27:14.973 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:27:14 smithi139 ceph-mon[129293]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' 2023-07-14T19:27:14.983 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:27:14 smithi098 ceph-mon[127486]: pgmap v170: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2023-07-14T19:27:14.984 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:27:14 smithi098 ceph-mon[127486]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]': finished 2023-07-14T19:27:14.984 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:27:14 smithi098 ceph-mon[127486]: osdmap e55: 9 total, 8 up, 9 in 2023-07-14T19:27:14.984 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:27:14 smithi098 ceph-mon[127486]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2023-07-14T19:27:14.984 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:27:14 smithi098 ceph-mon[127486]: from='osd.8 [v2:172.21.15.139:6816/3669514524,v1:172.21.15.139:6817/3669514524]' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi139", "root=default"]}]: dispatch 2023-07-14T19:27:14.984 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:27:14 smithi098 ceph-mon[127486]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi139", "root=default"]}]: dispatch 2023-07-14T19:27:14.985 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:27:14 smithi098 ceph-mon[127486]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' 2023-07-14T19:27:14.985 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:27:14 smithi098 ceph-mon[127486]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' 2023-07-14T19:27:14.985 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:27:14 smithi098 ceph-mon[127486]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2023-07-14T19:27:14.985 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:27:14 smithi098 ceph-mon[127486]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2023-07-14T19:27:14.985 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:27:14 smithi098 ceph-mon[127486]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2023-07-14T19:27:14.985 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:27:14 smithi098 ceph-mon[127486]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' 2023-07-14T19:27:14.985 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:27:14 smithi098 ceph-mon[127486]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T19:27:14.985 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:27:14 smithi098 ceph-mon[127486]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-14T19:27:14.986 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:27:14 smithi098 ceph-mon[127486]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' 2023-07-14T19:27:15.055 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81c9387d2e7576da065747ec9c16d1c7d69016ec shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 9adea648-227b-11ee-9b33-001a4aab830c -- ceph orch daemon add osd smithi098:vg_nvme/lv_1 2023-07-14T19:27:15.192 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:27:14 smithi101 ceph-mon[128807]: pgmap v170: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2023-07-14T19:27:15.192 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:27:14 smithi101 ceph-mon[128807]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]': finished 2023-07-14T19:27:15.192 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:27:14 smithi101 ceph-mon[128807]: osdmap e55: 9 total, 8 up, 9 in 2023-07-14T19:27:15.193 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:27:14 smithi101 ceph-mon[128807]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2023-07-14T19:27:15.193 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:27:14 smithi101 ceph-mon[128807]: from='osd.8 [v2:172.21.15.139:6816/3669514524,v1:172.21.15.139:6817/3669514524]' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi139", "root=default"]}]: dispatch 2023-07-14T19:27:15.193 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:27:14 smithi101 ceph-mon[128807]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi139", "root=default"]}]: dispatch 2023-07-14T19:27:15.194 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:27:14 smithi101 ceph-mon[128807]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' 2023-07-14T19:27:15.194 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:27:14 smithi101 ceph-mon[128807]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' 2023-07-14T19:27:15.194 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:27:14 smithi101 ceph-mon[128807]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2023-07-14T19:27:15.195 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:27:14 smithi101 ceph-mon[128807]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2023-07-14T19:27:15.195 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:27:14 smithi101 ceph-mon[128807]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2023-07-14T19:27:15.195 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:27:14 smithi101 ceph-mon[128807]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' 2023-07-14T19:27:15.195 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:27:14 smithi101 ceph-mon[128807]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T19:27:15.196 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:27:14 smithi101 ceph-mon[128807]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-14T19:27:15.196 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:27:14 smithi101 ceph-mon[128807]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' 2023-07-14T19:27:15.606 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/9adea648-227b-11ee-9b33-001a4aab830c/mon.a/config 2023-07-14T19:27:16.151 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:27:15 smithi098 ceph-mon[127486]: Detected new or changed devices on smithi139 2023-07-14T19:27:16.151 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:27:15 smithi098 ceph-mon[127486]: Adjusting osd_memory_target on smithi139 to 5722M 2023-07-14T19:27:16.152 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:27:15 smithi098 ceph-mon[127486]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi139", "root=default"]}]': finished 2023-07-14T19:27:16.152 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:27:15 smithi098 ceph-mon[127486]: osdmap e56: 9 total, 8 up, 9 in 2023-07-14T19:27:16.152 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:27:15 smithi098 ceph-mon[127486]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2023-07-14T19:27:16.152 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:27:15 smithi098 ceph-mon[127486]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2023-07-14T19:27:16.152 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:27:15 smithi098 ceph-mon[127486]: osd.8 [v2:172.21.15.139:6816/3669514524,v1:172.21.15.139:6817/3669514524] boot 2023-07-14T19:27:16.152 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:27:15 smithi098 ceph-mon[127486]: osdmap e57: 9 total, 9 up, 9 in 2023-07-14T19:27:16.153 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:27:15 smithi098 ceph-mon[127486]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2023-07-14T19:27:16.191 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:27:15 smithi101 ceph-mon[128807]: Detected new or changed devices on smithi139 2023-07-14T19:27:16.192 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:27:15 smithi101 ceph-mon[128807]: Adjusting osd_memory_target on smithi139 to 5722M 2023-07-14T19:27:16.192 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:27:15 smithi101 ceph-mon[128807]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi139", "root=default"]}]': finished 2023-07-14T19:27:16.192 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:27:15 smithi101 ceph-mon[128807]: osdmap e56: 9 total, 8 up, 9 in 2023-07-14T19:27:16.192 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:27:15 smithi101 ceph-mon[128807]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2023-07-14T19:27:16.192 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:27:15 smithi101 ceph-mon[128807]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2023-07-14T19:27:16.192 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:27:15 smithi101 ceph-mon[128807]: osd.8 [v2:172.21.15.139:6816/3669514524,v1:172.21.15.139:6817/3669514524] boot 2023-07-14T19:27:16.193 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:27:15 smithi101 ceph-mon[128807]: osdmap e57: 9 total, 9 up, 9 in 2023-07-14T19:27:16.193 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:27:15 smithi101 ceph-mon[128807]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2023-07-14T19:27:16.220 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:27:15 smithi139 ceph-mon[129293]: Detected new or changed devices on smithi139 2023-07-14T19:27:16.220 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:27:15 smithi139 ceph-mon[129293]: Adjusting osd_memory_target on smithi139 to 5722M 2023-07-14T19:27:16.221 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:27:15 smithi139 ceph-mon[129293]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi139", "root=default"]}]': finished 2023-07-14T19:27:16.221 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:27:15 smithi139 ceph-mon[129293]: osdmap e56: 9 total, 8 up, 9 in 2023-07-14T19:27:16.221 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:27:15 smithi139 ceph-mon[129293]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2023-07-14T19:27:16.221 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:27:15 smithi139 ceph-mon[129293]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2023-07-14T19:27:16.221 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:27:15 smithi139 ceph-mon[129293]: osd.8 [v2:172.21.15.139:6816/3669514524,v1:172.21.15.139:6817/3669514524] boot 2023-07-14T19:27:16.221 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:27:15 smithi139 ceph-mon[129293]: osdmap e57: 9 total, 9 up, 9 in 2023-07-14T19:27:16.221 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:27:15 smithi139 ceph-mon[129293]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2023-07-14T19:27:16.781 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:27:16.780+0000 7f096506a700 1 -- 172.21.15.98:0/3599719879 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f09600ff540 msgr2=0x7f09600ff990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:27:16.782 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:27:16.780+0000 7f096506a700 1 --2- 172.21.15.98:0/3599719879 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f09600ff540 0x7f09600ff990 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f0950009990 tx=0x7f0950009ca0 comp rx=0 tx=0).stop 2023-07-14T19:27:16.782 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:27:16.780+0000 7f096506a700 1 -- 172.21.15.98:0/3599719879 shutdown_connections 2023-07-14T19:27:16.782 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:27:16.780+0000 7f096506a700 1 --2- 172.21.15.98:0/3599719879 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f09600fe340 0x7f09600fe750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:27:16.782 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:27:16.780+0000 7f096506a700 1 --2- 172.21.15.98:0/3599719879 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f09600ffed0 0x7f0960106790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:27:16.782 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:27:16.780+0000 7f096506a700 1 --2- 172.21.15.98:0/3599719879 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f09600ff540 0x7f09600ff990 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:27:16.783 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:27:16.780+0000 7f096506a700 1 -- 172.21.15.98:0/3599719879 >> 172.21.15.98:0/3599719879 conn(0x7f09600f9990 msgr2=0x7f09600fbdc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:27:16.783 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:27:16.780+0000 7f096506a700 1 -- 172.21.15.98:0/3599719879 shutdown_connections 2023-07-14T19:27:16.783 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:27:16.780+0000 7f096506a700 1 -- 172.21.15.98:0/3599719879 wait complete. 2023-07-14T19:27:16.786 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:27:16.781+0000 7f096506a700 1 Processor -- start 2023-07-14T19:27:16.786 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:27:16.782+0000 7f096506a700 1 -- start start 2023-07-14T19:27:16.786 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:27:16.782+0000 7f096506a700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f09600fe340 0x7f09600fbda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:27:16.786 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:27:16.782+0000 7f096506a700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f09600ff540 0x7f096019cb40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:27:16.787 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:27:16.782+0000 7f096506a700 1 --2- >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f09600ffed0 0x7f096019f090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:27:16.787 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:27:16.782+0000 7f096506a700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f096019f740 con 0x7f09600fe340 2023-07-14T19:27:16.787 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:27:16.782+0000 7f096506a700 1 -- --> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f096019f8b0 con 0x7f09600ffed0 2023-07-14T19:27:16.787 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:27:16.782+0000 7f096506a700 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f096019fa70 con 0x7f09600ff540 2023-07-14T19:27:16.787 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:27:16.782+0000 7f095ed9d700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f09600fe340 0x7f09600fbda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:27:16.788 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:27:16.782+0000 7f095ed9d700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f09600fe340 0x7f09600fbda0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:45090/0 (socket says 172.21.15.98:45090) 2023-07-14T19:27:16.788 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:27:16.782+0000 7f095ed9d700 1 -- 172.21.15.98:0/870131415 learned_addr learned my addr 172.21.15.98:0/870131415 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-07-14T19:27:16.788 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:27:16.782+0000 7f095f59e700 1 --2- 172.21.15.98:0/870131415 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f09600ffed0 0x7f096019f090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:27:16.789 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:27:16.782+0000 7f095e59c700 1 --2- 172.21.15.98:0/870131415 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f09600ff540 0x7f096019cb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:27:16.789 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:27:16.782+0000 7f095ed9d700 1 -- 172.21.15.98:0/870131415 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f09600ff540 msgr2=0x7f096019cb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:27:16.789 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:27:16.782+0000 7f095ed9d700 1 --2- 172.21.15.98:0/870131415 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f09600ff540 0x7f096019cb40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:27:16.789 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:27:16.782+0000 7f095ed9d700 1 -- 172.21.15.98:0/870131415 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f09600ffed0 msgr2=0x7f096019f090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:27:16.789 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:27:16.782+0000 7f095ed9d700 1 --2- 172.21.15.98:0/870131415 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f09600ffed0 0x7f096019f090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:27:16.789 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:27:16.782+0000 7f095ed9d700 1 -- 172.21.15.98:0/870131415 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f09500095f0 con 0x7f09600fe340 2023-07-14T19:27:16.789 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:27:16.782+0000 7f095f59e700 1 --2- 172.21.15.98:0/870131415 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f09600ffed0 0x7f096019f090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-07-14T19:27:16.790 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:27:16.782+0000 7f095ed9d700 1 --2- 172.21.15.98:0/870131415 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f09600fe340 0x7f09600fbda0 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f095400c970 tx=0x7f095400cd30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:27:16.790 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:27:16.783+0000 7f0947fff700 1 -- 172.21.15.98:0/870131415 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0954004220 con 0x7f09600fe340 2023-07-14T19:27:16.790 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:27:16.783+0000 7f0947fff700 1 -- 172.21.15.98:0/870131415 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f0954007d60 con 0x7f09600fe340 2023-07-14T19:27:16.790 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:27:16.783+0000 7f096506a700 1 -- 172.21.15.98:0/870131415 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f096019fd50 con 0x7f09600fe340 2023-07-14T19:27:16.790 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:27:16.783+0000 7f0947fff700 1 -- 172.21.15.98:0/870131415 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0954003d00 con 0x7f09600fe340 2023-07-14T19:27:16.790 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:27:16.783+0000 7f096506a700 1 -- 172.21.15.98:0/870131415 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f09601a01f0 con 0x7f09600fe340 2023-07-14T19:27:16.790 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:27:16.784+0000 7f096506a700 1 -- 172.21.15.98:0/870131415 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0960105ec0 con 0x7f09600fe340 2023-07-14T19:27:16.791 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:27:16.785+0000 7f0947fff700 1 -- 172.21.15.98:0/870131415 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 16) v1 ==== 135715+0+0 (secure 0 0 0) 0x7f0954004380 con 0x7f09600fe340 2023-07-14T19:27:16.791 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:27:16.785+0000 7f0947fff700 1 --2- 172.21.15.98:0/870131415 >> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] conn(0x7f09480a1150 0x7f09480a3600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:27:16.791 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:27:16.785+0000 7f095e59c700 1 --2- 172.21.15.98:0/870131415 >> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] conn(0x7f09480a1150 0x7f09480a3600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:27:16.791 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:27:16.785+0000 7f0947fff700 1 -- 172.21.15.98:0/870131415 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(58..58 src has 1..58) v4 ==== 5858+0+0 (secure 0 0 0) 0x7f09540cbf40 con 0x7f09600fe340 2023-07-14T19:27:16.791 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:27:16.786+0000 7f095e59c700 1 --2- 172.21.15.98:0/870131415 >> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] conn(0x7f09480a1150 0x7f09480a3600 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f095000b5c0 tx=0x7f0950005fb0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:27:16.792 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:27:16.789+0000 7f0947fff700 1 -- 172.21.15.98:0/870131415 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f0954096020 con 0x7f09600fe340 2023-07-14T19:27:16.987 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:27:16.986+0000 7f096506a700 1 -- 172.21.15.98:0/870131415 --> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi098:vg_nvme/lv_1", "target": ["mon-mgr", ""]}) v1 -- 0x7f0960060a80 con 0x7f09480a1150 2023-07-14T19:27:17.191 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:27:16 smithi101 ceph-mon[128807]: purged_snaps scrub starts 2023-07-14T19:27:17.191 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:27:16 smithi101 ceph-mon[128807]: purged_snaps scrub ok 2023-07-14T19:27:17.192 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:27:16 smithi101 ceph-mon[128807]: pgmap v173: 1 pgs: 1 peering; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2023-07-14T19:27:17.192 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:27:16 smithi101 ceph-mon[128807]: osdmap e58: 9 total, 9 up, 9 in 2023-07-14T19:27:17.220 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:27:16 smithi139 ceph-mon[129293]: purged_snaps scrub starts 2023-07-14T19:27:17.220 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:27:16 smithi139 ceph-mon[129293]: purged_snaps scrub ok 2023-07-14T19:27:17.221 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:27:16 smithi139 ceph-mon[129293]: pgmap v173: 1 pgs: 1 peering; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2023-07-14T19:27:17.221 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:27:16 smithi139 ceph-mon[129293]: osdmap e58: 9 total, 9 up, 9 in 2023-07-14T19:27:17.281 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:27:16 smithi098 ceph-mon[127486]: purged_snaps scrub starts 2023-07-14T19:27:17.281 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:27:16 smithi098 ceph-mon[127486]: purged_snaps scrub ok 2023-07-14T19:27:17.281 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:27:16 smithi098 ceph-mon[127486]: pgmap v173: 1 pgs: 1 peering; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2023-07-14T19:27:17.282 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:27:16 smithi098 ceph-mon[127486]: osdmap e58: 9 total, 9 up, 9 in 2023-07-14T19:27:18.075 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:27:17 smithi098 ceph-mon[127486]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-07-14T19:27:18.075 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:27:17 smithi098 ceph-mon[127486]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-07-14T19:27:18.075 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:27:17 smithi098 ceph-mon[127486]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T19:27:18.076 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:27:17 smithi098 ceph-mon[127486]: osdmap e59: 9 total, 9 up, 9 in 2023-07-14T19:27:18.191 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:27:17 smithi101 ceph-mon[128807]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-07-14T19:27:18.192 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:27:17 smithi101 ceph-mon[128807]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-07-14T19:27:18.192 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:27:17 smithi101 ceph-mon[128807]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T19:27:18.192 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:27:17 smithi101 ceph-mon[128807]: osdmap e59: 9 total, 9 up, 9 in 2023-07-14T19:27:18.220 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:27:17 smithi139 ceph-mon[129293]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-07-14T19:27:18.221 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:27:17 smithi139 ceph-mon[129293]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-07-14T19:27:18.221 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:27:17 smithi139 ceph-mon[129293]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T19:27:18.221 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:27:17 smithi139 ceph-mon[129293]: osdmap e59: 9 total, 9 up, 9 in 2023-07-14T19:27:19.032 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:27:18 smithi098 ceph-mon[127486]: from='client.14436 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi098:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2023-07-14T19:27:19.032 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:27:18 smithi098 ceph-mon[127486]: pgmap v177: 1 pgs: 1 peering; 577 KiB data, 244 MiB used, 804 GiB / 805 GiB avail 2023-07-14T19:27:19.032 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:27:18 smithi098 ceph-mon[127486]: osdmap e60: 9 total, 9 up, 9 in 2023-07-14T19:27:19.191 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:27:18 smithi101 ceph-mon[128807]: from='client.14436 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi098:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2023-07-14T19:27:19.192 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:27:18 smithi101 ceph-mon[128807]: pgmap v177: 1 pgs: 1 peering; 577 KiB data, 244 MiB used, 804 GiB / 805 GiB avail 2023-07-14T19:27:19.192 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:27:18 smithi101 ceph-mon[128807]: osdmap e60: 9 total, 9 up, 9 in 2023-07-14T19:27:19.220 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:27:18 smithi139 ceph-mon[129293]: from='client.14436 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi098:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2023-07-14T19:27:19.220 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:27:18 smithi139 ceph-mon[129293]: pgmap v177: 1 pgs: 1 peering; 577 KiB data, 244 MiB used, 804 GiB / 805 GiB avail 2023-07-14T19:27:19.221 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:27:18 smithi139 ceph-mon[129293]: osdmap e60: 9 total, 9 up, 9 in 2023-07-14T19:27:20.691 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:27:20 smithi101 ceph-mon[128807]: osdmap e61: 9 total, 9 up, 9 in 2023-07-14T19:27:20.692 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:27:20 smithi101 ceph-mon[128807]: from='client.? 172.21.15.98:0/2736763648' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "19c5b54a-2242-4b23-b721-63be9559653d"}]: dispatch 2023-07-14T19:27:20.692 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:27:20 smithi101 ceph-mon[128807]: from='client.? 172.21.15.98:0/2736763648' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "19c5b54a-2242-4b23-b721-63be9559653d"}]': finished 2023-07-14T19:27:20.692 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:27:20 smithi101 ceph-mon[128807]: osdmap e62: 10 total, 9 up, 10 in 2023-07-14T19:27:20.692 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:27:20 smithi101 ceph-mon[128807]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2023-07-14T19:27:20.692 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:27:20 smithi101 ceph-mon[128807]: pgmap v181: 1 pgs: 1 peering; 577 KiB data, 244 MiB used, 804 GiB / 805 GiB avail 2023-07-14T19:27:20.693 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:27:20 smithi101 ceph-mon[128807]: from='client.? 172.21.15.98:0/3804318805' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-07-14T19:27:20.720 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:27:20 smithi139 ceph-mon[129293]: osdmap e61: 9 total, 9 up, 9 in 2023-07-14T19:27:20.721 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:27:20 smithi139 ceph-mon[129293]: from='client.? 172.21.15.98:0/2736763648' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "19c5b54a-2242-4b23-b721-63be9559653d"}]: dispatch 2023-07-14T19:27:20.721 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:27:20 smithi139 ceph-mon[129293]: from='client.? 172.21.15.98:0/2736763648' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "19c5b54a-2242-4b23-b721-63be9559653d"}]': finished 2023-07-14T19:27:20.721 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:27:20 smithi139 ceph-mon[129293]: osdmap e62: 10 total, 9 up, 10 in 2023-07-14T19:27:20.721 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:27:20 smithi139 ceph-mon[129293]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2023-07-14T19:27:20.721 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:27:20 smithi139 ceph-mon[129293]: pgmap v181: 1 pgs: 1 peering; 577 KiB data, 244 MiB used, 804 GiB / 805 GiB avail 2023-07-14T19:27:20.721 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:27:20 smithi139 ceph-mon[129293]: from='client.? 172.21.15.98:0/3804318805' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-07-14T19:27:20.781 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:27:20 smithi098 ceph-mon[127486]: osdmap e61: 9 total, 9 up, 9 in 2023-07-14T19:27:20.781 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:27:20 smithi098 ceph-mon[127486]: from='client.? 172.21.15.98:0/2736763648' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "19c5b54a-2242-4b23-b721-63be9559653d"}]: dispatch 2023-07-14T19:27:20.781 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:27:20 smithi098 ceph-mon[127486]: from='client.? 172.21.15.98:0/2736763648' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "19c5b54a-2242-4b23-b721-63be9559653d"}]': finished 2023-07-14T19:27:20.781 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:27:20 smithi098 ceph-mon[127486]: osdmap e62: 10 total, 9 up, 10 in 2023-07-14T19:27:20.782 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:27:20 smithi098 ceph-mon[127486]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2023-07-14T19:27:20.782 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:27:20 smithi098 ceph-mon[127486]: pgmap v181: 1 pgs: 1 peering; 577 KiB data, 244 MiB used, 804 GiB / 805 GiB avail 2023-07-14T19:27:20.782 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:27:20 smithi098 ceph-mon[127486]: from='client.? 172.21.15.98:0/3804318805' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-07-14T19:27:22.531 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:27:22 smithi098 ceph-mon[127486]: pgmap v182: 1 pgs: 1 active+clean; 577 KiB data, 245 MiB used, 804 GiB / 805 GiB avail; 134 KiB/s, 0 objects/s recovering 2023-07-14T19:27:22.691 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:27:22 smithi101 ceph-mon[128807]: pgmap v182: 1 pgs: 1 active+clean; 577 KiB data, 245 MiB used, 804 GiB / 805 GiB avail; 134 KiB/s, 0 objects/s recovering 2023-07-14T19:27:22.720 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:27:22 smithi139 ceph-mon[129293]: pgmap v182: 1 pgs: 1 active+clean; 577 KiB data, 245 MiB used, 804 GiB / 805 GiB avail; 134 KiB/s, 0 objects/s recovering 2023-07-14T19:27:24.892 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:27:24 smithi098 ceph-mon[127486]: pgmap v183: 1 pgs: 1 active+clean; 577 KiB data, 245 MiB used, 804 GiB / 805 GiB avail; 96 KiB/s, 0 objects/s recovering 2023-07-14T19:27:24.941 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:27:24 smithi101 ceph-mon[128807]: pgmap v183: 1 pgs: 1 active+clean; 577 KiB data, 245 MiB used, 804 GiB / 805 GiB avail; 96 KiB/s, 0 objects/s recovering 2023-07-14T19:27:24.970 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:27:24 smithi139 ceph-mon[129293]: pgmap v183: 1 pgs: 1 active+clean; 577 KiB data, 245 MiB used, 804 GiB / 805 GiB avail; 96 KiB/s, 0 objects/s recovering 2023-07-14T19:27:26.941 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:27:26 smithi101 ceph-mon[128807]: pgmap v184: 1 pgs: 1 active+clean; 577 KiB data, 245 MiB used, 804 GiB / 805 GiB avail; 79 KiB/s, 0 objects/s recovering 2023-07-14T19:27:26.942 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:27:26 smithi101 ceph-mon[128807]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.9"}]: dispatch 2023-07-14T19:27:26.942 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:27:26 smithi101 ceph-mon[128807]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T19:27:26.970 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:27:26 smithi139 ceph-mon[129293]: pgmap v184: 1 pgs: 1 active+clean; 577 KiB data, 245 MiB used, 804 GiB / 805 GiB avail; 79 KiB/s, 0 objects/s recovering 2023-07-14T19:27:26.970 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:27:26 smithi139 ceph-mon[129293]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.9"}]: dispatch 2023-07-14T19:27:26.971 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:27:26 smithi139 ceph-mon[129293]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T19:27:26.992 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:27:26 smithi098 ceph-mon[127486]: pgmap v184: 1 pgs: 1 active+clean; 577 KiB data, 245 MiB used, 804 GiB / 805 GiB avail; 79 KiB/s, 0 objects/s recovering 2023-07-14T19:27:26.992 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:27:26 smithi098 ceph-mon[127486]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.9"}]: dispatch 2023-07-14T19:27:26.992 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:27:26 smithi098 ceph-mon[127486]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T19:27:27.941 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:27:27 smithi101 ceph-mon[128807]: Deploying daemon osd.9 on smithi098 2023-07-14T19:27:27.970 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:27:27 smithi139 ceph-mon[129293]: Deploying daemon osd.9 on smithi098 2023-07-14T19:27:28.026 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:27:27 smithi098 ceph-mon[127486]: Deploying daemon osd.9 on smithi098 2023-07-14T19:27:28.757 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:27:28 smithi098 ceph-mon[127486]: pgmap v185: 1 pgs: 1 active+clean; 577 KiB data, 245 MiB used, 804 GiB / 805 GiB avail; 70 KiB/s, 0 objects/s recovering 2023-07-14T19:27:28.941 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:27:28 smithi101 ceph-mon[128807]: pgmap v185: 1 pgs: 1 active+clean; 577 KiB data, 245 MiB used, 804 GiB / 805 GiB avail; 70 KiB/s, 0 objects/s recovering 2023-07-14T19:27:28.970 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:27:28 smithi139 ceph-mon[129293]: pgmap v185: 1 pgs: 1 active+clean; 577 KiB data, 245 MiB used, 804 GiB / 805 GiB avail; 70 KiB/s, 0 objects/s recovering 2023-07-14T19:27:30.781 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:27:30 smithi098 ceph-mon[127486]: pgmap v186: 1 pgs: 1 active+clean; 577 KiB data, 245 MiB used, 804 GiB / 805 GiB avail; 57 KiB/s, 0 objects/s recovering 2023-07-14T19:27:30.941 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:27:30 smithi101 ceph-mon[128807]: pgmap v186: 1 pgs: 1 active+clean; 577 KiB data, 245 MiB used, 804 GiB / 805 GiB avail; 57 KiB/s, 0 objects/s recovering 2023-07-14T19:27:30.970 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:27:30 smithi139 ceph-mon[129293]: pgmap v186: 1 pgs: 1 active+clean; 577 KiB data, 245 MiB used, 804 GiB / 805 GiB avail; 57 KiB/s, 0 objects/s recovering 2023-07-14T19:27:31.781 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:27:31 smithi098 ceph-mon[127486]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' 2023-07-14T19:27:31.782 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:27:31 smithi098 ceph-mon[127486]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' 2023-07-14T19:27:31.782 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:27:31 smithi098 ceph-mon[127486]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-14T19:27:31.941 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:27:31 smithi101 ceph-mon[128807]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' 2023-07-14T19:27:31.942 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:27:31 smithi101 ceph-mon[128807]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' 2023-07-14T19:27:31.942 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:27:31 smithi101 ceph-mon[128807]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-14T19:27:31.970 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:27:31 smithi139 ceph-mon[129293]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' 2023-07-14T19:27:31.970 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:27:31 smithi139 ceph-mon[129293]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' 2023-07-14T19:27:31.970 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:27:31 smithi139 ceph-mon[129293]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-14T19:27:32.782 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:27:32 smithi098 ceph-mon[127486]: pgmap v187: 1 pgs: 1 active+clean; 577 KiB data, 245 MiB used, 804 GiB / 805 GiB avail; 48 KiB/s, 0 objects/s recovering 2023-07-14T19:27:32.941 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:27:32 smithi101 ceph-mon[128807]: pgmap v187: 1 pgs: 1 active+clean; 577 KiB data, 245 MiB used, 804 GiB / 805 GiB avail; 48 KiB/s, 0 objects/s recovering 2023-07-14T19:27:32.970 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:27:32 smithi139 ceph-mon[129293]: pgmap v187: 1 pgs: 1 active+clean; 577 KiB data, 245 MiB used, 804 GiB / 805 GiB avail; 48 KiB/s, 0 objects/s recovering 2023-07-14T19:27:33.388 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:27:33.387+0000 7f0947fff700 1 -- 172.21.15.98:0/870131415 <== mgr.14152 v2:172.21.15.98:6800/452042164 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7f0960060a80 con 0x7f09480a1150 2023-07-14T19:27:33.389 INFO:teuthology.orchestra.run.smithi098.stdout:Created osd(s) 9 on host 'smithi098' 2023-07-14T19:27:33.392 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:27:33.391+0000 7f096506a700 1 -- 172.21.15.98:0/870131415 >> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] conn(0x7f09480a1150 msgr2=0x7f09480a3600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:27:33.392 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:27:33.391+0000 7f096506a700 1 --2- 172.21.15.98:0/870131415 >> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] conn(0x7f09480a1150 0x7f09480a3600 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f095000b5c0 tx=0x7f0950005fb0 comp rx=0 tx=0).stop 2023-07-14T19:27:33.392 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:27:33.391+0000 7f096506a700 1 -- 172.21.15.98:0/870131415 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f09600fe340 msgr2=0x7f09600fbda0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:27:33.392 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:27:33.391+0000 7f096506a700 1 --2- 172.21.15.98:0/870131415 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f09600fe340 0x7f09600fbda0 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f095400c970 tx=0x7f095400cd30 comp rx=0 tx=0).stop 2023-07-14T19:27:33.392 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:27:33.391+0000 7f096506a700 1 -- 172.21.15.98:0/870131415 shutdown_connections 2023-07-14T19:27:33.393 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:27:33.391+0000 7f096506a700 1 --2- 172.21.15.98:0/870131415 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f09600ffed0 0x7f096019f090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:27:33.393 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:27:33.391+0000 7f096506a700 1 --2- 172.21.15.98:0/870131415 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f09600fe340 0x7f09600fbda0 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:27:33.393 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:27:33.391+0000 7f096506a700 1 --2- 172.21.15.98:0/870131415 >> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] conn(0x7f09480a1150 0x7f09480a3600 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:27:33.393 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:27:33.391+0000 7f096506a700 1 --2- 172.21.15.98:0/870131415 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f09600ff540 0x7f096019cb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:27:33.393 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:27:33.391+0000 7f096506a700 1 -- 172.21.15.98:0/870131415 >> 172.21.15.98:0/870131415 conn(0x7f09600f9990 msgr2=0x7f0960103000 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:27:33.393 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:27:33.391+0000 7f096506a700 1 -- 172.21.15.98:0/870131415 shutdown_connections 2023-07-14T19:27:33.393 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:27:33.391+0000 7f096506a700 1 -- 172.21.15.98:0/870131415 wait complete. 2023-07-14T19:27:33.785 DEBUG:teuthology.orchestra.run.smithi098:osd.9> sudo journalctl -f -n 0 -u ceph-9adea648-227b-11ee-9b33-001a4aab830c@osd.9.service 2023-07-14T19:27:33.787 INFO:tasks.cephadm:Deploying osd.10 on smithi101 with /dev/vg_nvme/lv_1... 2023-07-14T19:27:33.788 DEBUG:teuthology.orchestra.run.smithi101:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81c9387d2e7576da065747ec9c16d1c7d69016ec shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 9adea648-227b-11ee-9b33-001a4aab830c -- ceph-volume lvm zap /dev/vg_nvme/lv_1 2023-07-14T19:27:33.819 INFO:journalctl@ceph.osd.9.smithi098.stdout:-- Logs begin at Fri 2023-07-14 18:57:10 UTC. -- 2023-07-14T19:27:34.352 INFO:teuthology.orchestra.run.smithi101.stderr:Inferring config /var/lib/ceph/9adea648-227b-11ee-9b33-001a4aab830c/mon.b/config 2023-07-14T19:27:34.516 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:27:34 smithi098 ceph-mon[127486]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' 2023-07-14T19:27:34.517 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:27:34 smithi098 ceph-mon[127486]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' 2023-07-14T19:27:34.517 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:27:34 smithi098 ceph-mon[127486]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' 2023-07-14T19:27:34.517 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:27:34 smithi098 ceph-mon[127486]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' 2023-07-14T19:27:34.517 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:27:34 smithi098 ceph-mon[127486]: pgmap v188: 1 pgs: 1 active+clean; 577 KiB data, 245 MiB used, 804 GiB / 805 GiB avail 2023-07-14T19:27:34.563 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:27:34 smithi101 ceph-mon[128807]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' 2023-07-14T19:27:34.563 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:27:34 smithi101 ceph-mon[128807]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' 2023-07-14T19:27:34.563 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:27:34 smithi101 ceph-mon[128807]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' 2023-07-14T19:27:34.564 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:27:34 smithi101 ceph-mon[128807]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' 2023-07-14T19:27:34.564 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:27:34 smithi101 ceph-mon[128807]: pgmap v188: 1 pgs: 1 active+clean; 577 KiB data, 245 MiB used, 804 GiB / 805 GiB avail 2023-07-14T19:27:34.720 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:27:34 smithi139 ceph-mon[129293]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' 2023-07-14T19:27:34.720 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:27:34 smithi139 ceph-mon[129293]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' 2023-07-14T19:27:34.721 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:27:34 smithi139 ceph-mon[129293]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' 2023-07-14T19:27:34.721 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:27:34 smithi139 ceph-mon[129293]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' 2023-07-14T19:27:34.721 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:27:34 smithi139 ceph-mon[129293]: pgmap v188: 1 pgs: 1 active+clean; 577 KiB data, 245 MiB used, 804 GiB / 805 GiB avail 2023-07-14T19:27:34.781 INFO:journalctl@ceph.osd.9.smithi098.stdout:Jul 14 19:27:34 smithi098 ceph-9adea648-227b-11ee-9b33-001a4aab830c-osd-9[156860]: 2023-07-14T19:27:34.647+0000 7fd88552c640 -1 osd.9 0 log_to_monitors true 2023-07-14T19:27:35.691 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:27:35 smithi101 ceph-mon[128807]: from='osd.9 [v2:172.21.15.98:6826/2893994080,v1:172.21.15.98:6827/2893994080]' entity='osd.9' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]: dispatch 2023-07-14T19:27:35.720 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:27:35 smithi139 ceph-mon[129293]: from='osd.9 [v2:172.21.15.98:6826/2893994080,v1:172.21.15.98:6827/2893994080]' entity='osd.9' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]: dispatch 2023-07-14T19:27:35.781 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:27:35 smithi098 ceph-mon[127486]: from='osd.9 [v2:172.21.15.98:6826/2893994080,v1:172.21.15.98:6827/2893994080]' entity='osd.9' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]: dispatch 2023-07-14T19:27:35.845 INFO:teuthology.orchestra.run.smithi101.stderr:--> Zapping: /dev/vg_nvme/lv_1 2023-07-14T19:27:35.846 INFO:teuthology.orchestra.run.smithi101.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_1 bs=1M count=10 conv=fsync 2023-07-14T19:27:35.846 INFO:teuthology.orchestra.run.smithi101.stderr: stderr: 10+0 records in 2023-07-14T19:27:35.846 INFO:teuthology.orchestra.run.smithi101.stderr:10+0 records out 2023-07-14T19:27:35.846 INFO:teuthology.orchestra.run.smithi101.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0107657 s, 974 MB/s 2023-07-14T19:27:35.846 INFO:teuthology.orchestra.run.smithi101.stderr:--> Zapping successful for: 2023-07-14T19:27:36.296 DEBUG:teuthology.orchestra.run.smithi101:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81c9387d2e7576da065747ec9c16d1c7d69016ec shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 9adea648-227b-11ee-9b33-001a4aab830c -- ceph orch daemon add osd smithi101:vg_nvme/lv_1 2023-07-14T19:27:36.532 INFO:journalctl@ceph.osd.9.smithi098.stdout:Jul 14 19:27:36 smithi098 ceph-9adea648-227b-11ee-9b33-001a4aab830c-osd-9[156860]: 2023-07-14T19:27:36.397+0000 7fd87a3a3700 -1 osd.9 0 waiting for initial osdmap 2023-07-14T19:27:36.532 INFO:journalctl@ceph.osd.9.smithi098.stdout:Jul 14 19:27:36 smithi098 ceph-9adea648-227b-11ee-9b33-001a4aab830c-osd-9[156860]: 2023-07-14T19:27:36.410+0000 7fd874195700 -1 osd.9 64 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-07-14T19:27:36.533 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:27:36 smithi098 ceph-mon[127486]: from='osd.9 [v2:172.21.15.98:6826/2893994080,v1:172.21.15.98:6827/2893994080]' entity='osd.9' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]': finished 2023-07-14T19:27:36.533 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:27:36 smithi098 ceph-mon[127486]: osdmap e63: 10 total, 9 up, 10 in 2023-07-14T19:27:36.533 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:27:36 smithi098 ceph-mon[127486]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2023-07-14T19:27:36.533 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:27:36 smithi098 ceph-mon[127486]: from='osd.9 [v2:172.21.15.98:6826/2893994080,v1:172.21.15.98:6827/2893994080]' entity='osd.9' cmd=[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi098", "root=default"]}]: dispatch 2023-07-14T19:27:36.533 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:27:36 smithi098 ceph-mon[127486]: pgmap v190: 1 pgs: 1 active+clean; 577 KiB data, 245 MiB used, 804 GiB / 805 GiB avail 2023-07-14T19:27:36.533 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:27:36 smithi098 ceph-mon[127486]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' 2023-07-14T19:27:36.534 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:27:36 smithi098 ceph-mon[127486]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' 2023-07-14T19:27:36.534 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:27:36 smithi098 ceph-mon[127486]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2023-07-14T19:27:36.534 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:27:36 smithi098 ceph-mon[127486]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2023-07-14T19:27:36.534 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:27:36 smithi098 ceph-mon[127486]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2023-07-14T19:27:36.534 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:27:36 smithi098 ceph-mon[127486]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2023-07-14T19:27:36.534 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:27:36 smithi098 ceph-mon[127486]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' 2023-07-14T19:27:36.534 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:27:36 smithi098 ceph-mon[127486]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T19:27:36.535 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:27:36 smithi098 ceph-mon[127486]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-14T19:27:36.535 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:27:36 smithi098 ceph-mon[127486]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' 2023-07-14T19:27:36.691 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:27:36 smithi101 ceph-mon[128807]: from='osd.9 [v2:172.21.15.98:6826/2893994080,v1:172.21.15.98:6827/2893994080]' entity='osd.9' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]': finished 2023-07-14T19:27:36.692 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:27:36 smithi101 ceph-mon[128807]: osdmap e63: 10 total, 9 up, 10 in 2023-07-14T19:27:36.692 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:27:36 smithi101 ceph-mon[128807]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2023-07-14T19:27:36.692 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:27:36 smithi101 ceph-mon[128807]: from='osd.9 [v2:172.21.15.98:6826/2893994080,v1:172.21.15.98:6827/2893994080]' entity='osd.9' cmd=[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi098", "root=default"]}]: dispatch 2023-07-14T19:27:36.692 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:27:36 smithi101 ceph-mon[128807]: pgmap v190: 1 pgs: 1 active+clean; 577 KiB data, 245 MiB used, 804 GiB / 805 GiB avail 2023-07-14T19:27:36.692 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:27:36 smithi101 ceph-mon[128807]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' 2023-07-14T19:27:36.692 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:27:36 smithi101 ceph-mon[128807]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' 2023-07-14T19:27:36.693 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:27:36 smithi101 ceph-mon[128807]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2023-07-14T19:27:36.693 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:27:36 smithi101 ceph-mon[128807]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2023-07-14T19:27:36.693 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:27:36 smithi101 ceph-mon[128807]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2023-07-14T19:27:36.693 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:27:36 smithi101 ceph-mon[128807]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2023-07-14T19:27:36.693 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:27:36 smithi101 ceph-mon[128807]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' 2023-07-14T19:27:36.693 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:27:36 smithi101 ceph-mon[128807]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T19:27:36.693 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:27:36 smithi101 ceph-mon[128807]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-14T19:27:36.693 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:27:36 smithi101 ceph-mon[128807]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' 2023-07-14T19:27:36.720 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:27:36 smithi139 ceph-mon[129293]: from='osd.9 [v2:172.21.15.98:6826/2893994080,v1:172.21.15.98:6827/2893994080]' entity='osd.9' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]': finished 2023-07-14T19:27:36.721 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:27:36 smithi139 ceph-mon[129293]: osdmap e63: 10 total, 9 up, 10 in 2023-07-14T19:27:36.721 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:27:36 smithi139 ceph-mon[129293]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2023-07-14T19:27:36.721 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:27:36 smithi139 ceph-mon[129293]: from='osd.9 [v2:172.21.15.98:6826/2893994080,v1:172.21.15.98:6827/2893994080]' entity='osd.9' cmd=[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi098", "root=default"]}]: dispatch 2023-07-14T19:27:36.721 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:27:36 smithi139 ceph-mon[129293]: pgmap v190: 1 pgs: 1 active+clean; 577 KiB data, 245 MiB used, 804 GiB / 805 GiB avail 2023-07-14T19:27:36.721 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:27:36 smithi139 ceph-mon[129293]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' 2023-07-14T19:27:36.721 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:27:36 smithi139 ceph-mon[129293]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' 2023-07-14T19:27:36.721 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:27:36 smithi139 ceph-mon[129293]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2023-07-14T19:27:36.722 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:27:36 smithi139 ceph-mon[129293]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2023-07-14T19:27:36.722 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:27:36 smithi139 ceph-mon[129293]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2023-07-14T19:27:36.722 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:27:36 smithi139 ceph-mon[129293]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2023-07-14T19:27:36.722 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:27:36 smithi139 ceph-mon[129293]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' 2023-07-14T19:27:36.722 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:27:36 smithi139 ceph-mon[129293]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T19:27:36.722 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:27:36 smithi139 ceph-mon[129293]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-14T19:27:36.722 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:27:36 smithi139 ceph-mon[129293]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' 2023-07-14T19:27:36.856 INFO:teuthology.orchestra.run.smithi101.stderr:Inferring config /var/lib/ceph/9adea648-227b-11ee-9b33-001a4aab830c/mon.b/config 2023-07-14T19:27:37.691 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:27:37 smithi101 ceph-mon[128807]: Detected new or changed devices on smithi098 2023-07-14T19:27:37.692 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:27:37 smithi101 ceph-mon[128807]: Adjusting osd_memory_target on smithi098 to 4292M 2023-07-14T19:27:37.692 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:27:37 smithi101 ceph-mon[128807]: from='osd.9 [v2:172.21.15.98:6826/2893994080,v1:172.21.15.98:6827/2893994080]' entity='osd.9' cmd='[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi098", "root=default"]}]': finished 2023-07-14T19:27:37.692 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:27:37 smithi101 ceph-mon[128807]: osdmap e64: 10 total, 9 up, 10 in 2023-07-14T19:27:37.692 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:27:37 smithi101 ceph-mon[128807]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2023-07-14T19:27:37.692 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:27:37 smithi101 ceph-mon[128807]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2023-07-14T19:27:37.692 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:27:37 smithi101 ceph-mon[128807]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-14T19:27:37.692 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:27:37 smithi101 ceph-mon[128807]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T19:27:37.693 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:27:37 smithi101 ceph-mon[128807]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-14T19:27:37.693 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:27:37 smithi101 ceph-mon[128807]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' 2023-07-14T19:27:37.720 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:27:37 smithi139 ceph-mon[129293]: Detected new or changed devices on smithi098 2023-07-14T19:27:37.721 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:27:37 smithi139 ceph-mon[129293]: Adjusting osd_memory_target on smithi098 to 4292M 2023-07-14T19:27:37.721 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:27:37 smithi139 ceph-mon[129293]: from='osd.9 [v2:172.21.15.98:6826/2893994080,v1:172.21.15.98:6827/2893994080]' entity='osd.9' cmd='[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi098", "root=default"]}]': finished 2023-07-14T19:27:37.721 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:27:37 smithi139 ceph-mon[129293]: osdmap e64: 10 total, 9 up, 10 in 2023-07-14T19:27:37.722 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:27:37 smithi139 ceph-mon[129293]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2023-07-14T19:27:37.722 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:27:37 smithi139 ceph-mon[129293]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2023-07-14T19:27:37.722 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:27:37 smithi139 ceph-mon[129293]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-14T19:27:37.723 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:27:37 smithi139 ceph-mon[129293]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T19:27:37.723 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:27:37 smithi139 ceph-mon[129293]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-14T19:27:37.723 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:27:37 smithi139 ceph-mon[129293]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' 2023-07-14T19:27:37.781 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:27:37 smithi098 ceph-mon[127486]: Detected new or changed devices on smithi098 2023-07-14T19:27:37.781 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:27:37 smithi098 ceph-mon[127486]: Adjusting osd_memory_target on smithi098 to 4292M 2023-07-14T19:27:37.782 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:27:37 smithi098 ceph-mon[127486]: from='osd.9 [v2:172.21.15.98:6826/2893994080,v1:172.21.15.98:6827/2893994080]' entity='osd.9' cmd='[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi098", "root=default"]}]': finished 2023-07-14T19:27:37.782 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:27:37 smithi098 ceph-mon[127486]: osdmap e64: 10 total, 9 up, 10 in 2023-07-14T19:27:37.782 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:27:37 smithi098 ceph-mon[127486]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2023-07-14T19:27:37.782 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:27:37 smithi098 ceph-mon[127486]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2023-07-14T19:27:37.782 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:27:37 smithi098 ceph-mon[127486]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-14T19:27:37.783 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:27:37 smithi098 ceph-mon[127486]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T19:27:37.783 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:27:37 smithi098 ceph-mon[127486]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-14T19:27:37.783 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:27:37 smithi098 ceph-mon[127486]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' 2023-07-14T19:27:38.065 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-14T19:27:38.062+0000 7f59b8190700 1 -- 172.21.15.101:0/2465609227 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f59b00ffe10 msgr2=0x7f59b0100260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:27:38.066 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-14T19:27:38.062+0000 7f59b8190700 1 --2- 172.21.15.101:0/2465609227 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f59b00ffe10 0x7f59b0100260 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f59ac009990 tx=0x7f59ac009ca0 comp rx=0 tx=0).stop 2023-07-14T19:27:38.066 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-14T19:27:38.062+0000 7f59b8190700 1 -- 172.21.15.101:0/2465609227 shutdown_connections 2023-07-14T19:27:38.066 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-14T19:27:38.062+0000 7f59b8190700 1 --2- 172.21.15.101:0/2465609227 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f59b00fec10 0x7f59b00ff020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:27:38.066 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-14T19:27:38.062+0000 7f59b8190700 1 --2- 172.21.15.101:0/2465609227 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f59b01007a0 0x7f59b0107060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:27:38.066 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-14T19:27:38.062+0000 7f59b8190700 1 --2- 172.21.15.101:0/2465609227 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f59b00ffe10 0x7f59b0100260 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:27:38.067 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-14T19:27:38.062+0000 7f59b8190700 1 -- 172.21.15.101:0/2465609227 >> 172.21.15.101:0/2465609227 conn(0x7f59b00fa260 msgr2=0x7f59b00fc690 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:27:38.067 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-14T19:27:38.062+0000 7f59b8190700 1 -- 172.21.15.101:0/2465609227 shutdown_connections 2023-07-14T19:27:38.067 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-14T19:27:38.062+0000 7f59b8190700 1 -- 172.21.15.101:0/2465609227 wait complete. 2023-07-14T19:27:38.067 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-14T19:27:38.063+0000 7f59b8190700 1 Processor -- start 2023-07-14T19:27:38.067 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-14T19:27:38.063+0000 7f59b8190700 1 -- start start 2023-07-14T19:27:38.067 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-14T19:27:38.063+0000 7f59b8190700 1 --2- >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f59b00fec10 0x7f59b01959f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:27:38.068 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-14T19:27:38.063+0000 7f59b8190700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f59b00ffe10 0x7f59b0195f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:27:38.068 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-14T19:27:38.063+0000 7f59b8190700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f59b01007a0 0x7f59b019cfb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:27:38.068 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-14T19:27:38.063+0000 7f59b8190700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f59b019d580 con 0x7f59b01007a0 2023-07-14T19:27:38.069 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-14T19:27:38.063+0000 7f59b8190700 1 -- --> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f59b019d6c0 con 0x7f59b00fec10 2023-07-14T19:27:38.069 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-14T19:27:38.063+0000 7f59b8190700 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f59b019d850 con 0x7f59b00ffe10 2023-07-14T19:27:38.069 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-14T19:27:38.063+0000 7f59b5f2c700 1 --2- >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f59b00fec10 0x7f59b01959f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:27:38.069 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-14T19:27:38.063+0000 7f59b5f2c700 1 --2- >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f59b00fec10 0x7f59b01959f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.101:3300/0 says I am v2:172.21.15.101:36344/0 (socket says 172.21.15.101:36344) 2023-07-14T19:27:38.069 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-14T19:27:38.063+0000 7f59b5f2c700 1 -- 172.21.15.101:0/2330330211 learned_addr learned my addr 172.21.15.101:0/2330330211 (peer_addr_for_me v2:172.21.15.101:0/0) 2023-07-14T19:27:38.069 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-14T19:27:38.063+0000 7f59b672d700 1 --2- 172.21.15.101:0/2330330211 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f59b01007a0 0x7f59b019cfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:27:38.069 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-14T19:27:38.064+0000 7f59b572b700 1 --2- 172.21.15.101:0/2330330211 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f59b00ffe10 0x7f59b0195f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:27:38.070 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-14T19:27:38.064+0000 7f59b5f2c700 1 -- 172.21.15.101:0/2330330211 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f59b00ffe10 msgr2=0x7f59b0195f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:27:38.070 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-14T19:27:38.064+0000 7f59b5f2c700 1 --2- 172.21.15.101:0/2330330211 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f59b00ffe10 0x7f59b0195f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:27:38.070 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-14T19:27:38.064+0000 7f59b5f2c700 1 -- 172.21.15.101:0/2330330211 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f59b01007a0 msgr2=0x7f59b019cfb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:27:38.070 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-14T19:27:38.064+0000 7f59b5f2c700 1 --2- 172.21.15.101:0/2330330211 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f59b01007a0 0x7f59b019cfb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:27:38.070 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-14T19:27:38.064+0000 7f59b5f2c700 1 -- 172.21.15.101:0/2330330211 --> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f59ac0095f0 con 0x7f59b00fec10 2023-07-14T19:27:38.070 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-14T19:27:38.064+0000 7f59b672d700 1 --2- 172.21.15.101:0/2330330211 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f59b01007a0 0x7f59b019cfb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-07-14T19:27:38.070 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-14T19:27:38.064+0000 7f59b5f2c700 1 --2- 172.21.15.101:0/2330330211 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f59b00fec10 0x7f59b01959f0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f59a400d980 tx=0x7f59a400dd40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:27:38.071 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-14T19:27:38.064+0000 7f59a2ffd700 1 -- 172.21.15.101:0/2330330211 <== mon.2 v2:172.21.15.101:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f59a400de60 con 0x7f59b00fec10 2023-07-14T19:27:38.071 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-14T19:27:38.064+0000 7f59a2ffd700 1 -- 172.21.15.101:0/2330330211 <== mon.2 v2:172.21.15.101:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f59a40044e0 con 0x7f59b00fec10 2023-07-14T19:27:38.071 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-14T19:27:38.064+0000 7f59a2ffd700 1 -- 172.21.15.101:0/2330330211 <== mon.2 v2:172.21.15.101:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f59a4003bc0 con 0x7f59b00fec10 2023-07-14T19:27:38.071 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-14T19:27:38.064+0000 7f59b8190700 1 -- 172.21.15.101:0/2330330211 --> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f59b019db30 con 0x7f59b00fec10 2023-07-14T19:27:38.071 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-14T19:27:38.064+0000 7f59b8190700 1 -- 172.21.15.101:0/2330330211 --> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f59b019dfe0 con 0x7f59b00fec10 2023-07-14T19:27:38.072 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-14T19:27:38.065+0000 7f59b8190700 1 -- 172.21.15.101:0/2330330211 --> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f59b010a3d0 con 0x7f59b00fec10 2023-07-14T19:27:38.073 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-14T19:27:38.067+0000 7f59a2ffd700 1 -- 172.21.15.101:0/2330330211 <== mon.2 v2:172.21.15.101:3300/0 4 ==== mgrmap(e 16) v1 ==== 135715+0+0 (secure 0 0 0) 0x7f59a4009bb0 con 0x7f59b00fec10 2023-07-14T19:27:38.073 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-14T19:27:38.067+0000 7f59a2ffd700 1 --2- 172.21.15.101:0/2330330211 >> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] conn(0x7f599c0a1150 0x7f599c0a3600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:27:38.073 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-14T19:27:38.067+0000 7f59a2ffd700 1 -- 172.21.15.101:0/2330330211 <== mon.2 v2:172.21.15.101:3300/0 5 ==== osd_map(65..65 src has 1..65) v4 ==== 6261+0+0 (secure 0 0 0) 0x7f59a40cc190 con 0x7f59b00fec10 2023-07-14T19:27:38.074 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-14T19:27:38.067+0000 7f59b572b700 1 --2- 172.21.15.101:0/2330330211 >> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] conn(0x7f599c0a1150 0x7f599c0a3600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:27:38.074 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-14T19:27:38.068+0000 7f59b572b700 1 --2- 172.21.15.101:0/2330330211 >> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] conn(0x7f599c0a1150 0x7f599c0a3600 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f59ac006010 tx=0x7f59ac00b540 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:27:38.074 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-14T19:27:38.071+0000 7f59a2ffd700 1 -- 172.21.15.101:0/2330330211 <== mon.2 v2:172.21.15.101:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f59a40960e0 con 0x7f59b00fec10 2023-07-14T19:27:38.268 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-14T19:27:38.267+0000 7f59b8190700 1 -- 172.21.15.101:0/2330330211 --> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi101:vg_nvme/lv_1", "target": ["mon-mgr", ""]}) v1 -- 0x7f59b0069190 con 0x7f599c0a1150 2023-07-14T19:27:38.691 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:27:38 smithi101 ceph-mon[128807]: purged_snaps scrub starts 2023-07-14T19:27:38.692 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:27:38 smithi101 ceph-mon[128807]: purged_snaps scrub ok 2023-07-14T19:27:38.692 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:27:38 smithi101 ceph-mon[128807]: osd.9 [v2:172.21.15.98:6826/2893994080,v1:172.21.15.98:6827/2893994080] boot 2023-07-14T19:27:38.692 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:27:38 smithi101 ceph-mon[128807]: osdmap e65: 10 total, 10 up, 10 in 2023-07-14T19:27:38.692 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:27:38 smithi101 ceph-mon[128807]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2023-07-14T19:27:38.692 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:27:38 smithi101 ceph-mon[128807]: pgmap v193: 1 pgs: 1 active+clean; 577 KiB data, 245 MiB used, 804 GiB / 805 GiB avail 2023-07-14T19:27:38.692 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:27:38 smithi101 ceph-mon[128807]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-07-14T19:27:38.693 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:27:38 smithi101 ceph-mon[128807]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-07-14T19:27:38.693 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:27:38 smithi101 ceph-mon[128807]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T19:27:38.720 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:27:38 smithi139 ceph-mon[129293]: purged_snaps scrub starts 2023-07-14T19:27:38.721 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:27:38 smithi139 ceph-mon[129293]: purged_snaps scrub ok 2023-07-14T19:27:38.721 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:27:38 smithi139 ceph-mon[129293]: osd.9 [v2:172.21.15.98:6826/2893994080,v1:172.21.15.98:6827/2893994080] boot 2023-07-14T19:27:38.721 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:27:38 smithi139 ceph-mon[129293]: osdmap e65: 10 total, 10 up, 10 in 2023-07-14T19:27:38.721 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:27:38 smithi139 ceph-mon[129293]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2023-07-14T19:27:38.721 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:27:38 smithi139 ceph-mon[129293]: pgmap v193: 1 pgs: 1 active+clean; 577 KiB data, 245 MiB used, 804 GiB / 805 GiB avail 2023-07-14T19:27:38.721 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:27:38 smithi139 ceph-mon[129293]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-07-14T19:27:38.722 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:27:38 smithi139 ceph-mon[129293]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-07-14T19:27:38.722 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:27:38 smithi139 ceph-mon[129293]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T19:27:38.781 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:27:38 smithi098 ceph-mon[127486]: purged_snaps scrub starts 2023-07-14T19:27:38.781 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:27:38 smithi098 ceph-mon[127486]: purged_snaps scrub ok 2023-07-14T19:27:38.782 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:27:38 smithi098 ceph-mon[127486]: osd.9 [v2:172.21.15.98:6826/2893994080,v1:172.21.15.98:6827/2893994080] boot 2023-07-14T19:27:38.782 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:27:38 smithi098 ceph-mon[127486]: osdmap e65: 10 total, 10 up, 10 in 2023-07-14T19:27:38.782 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:27:38 smithi098 ceph-mon[127486]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2023-07-14T19:27:38.782 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:27:38 smithi098 ceph-mon[127486]: pgmap v193: 1 pgs: 1 active+clean; 577 KiB data, 245 MiB used, 804 GiB / 805 GiB avail 2023-07-14T19:27:38.782 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:27:38 smithi098 ceph-mon[127486]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-07-14T19:27:38.782 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:27:38 smithi098 ceph-mon[127486]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-07-14T19:27:38.782 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:27:38 smithi098 ceph-mon[127486]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T19:27:39.691 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:27:39 smithi101 ceph-mon[128807]: from='client.24347 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi101:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2023-07-14T19:27:39.691 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:27:39 smithi101 ceph-mon[128807]: osdmap e66: 10 total, 10 up, 10 in 2023-07-14T19:27:39.720 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:27:39 smithi139 ceph-mon[129293]: from='client.24347 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi101:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2023-07-14T19:27:39.720 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:27:39 smithi139 ceph-mon[129293]: osdmap e66: 10 total, 10 up, 10 in 2023-07-14T19:27:39.781 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:27:39 smithi098 ceph-mon[127486]: from='client.24347 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi101:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2023-07-14T19:27:39.781 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:27:39 smithi098 ceph-mon[127486]: osdmap e66: 10 total, 10 up, 10 in 2023-07-14T19:27:40.691 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:27:40 smithi101 ceph-mon[128807]: pgmap v195: 1 pgs: 1 active+clean; 577 KiB data, 272 MiB used, 894 GiB / 894 GiB avail 2023-07-14T19:27:40.720 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:27:40 smithi139 ceph-mon[129293]: pgmap v195: 1 pgs: 1 active+clean; 577 KiB data, 272 MiB used, 894 GiB / 894 GiB avail 2023-07-14T19:27:40.781 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:27:40 smithi098 ceph-mon[127486]: pgmap v195: 1 pgs: 1 active+clean; 577 KiB data, 272 MiB used, 894 GiB / 894 GiB avail 2023-07-14T19:27:41.691 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:27:41 smithi101 ceph-mon[128807]: from='client.? 172.21.15.101:0/538320225' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "326f0209-ba9d-4b59-97cc-d61a758a8f29"}]: dispatch 2023-07-14T19:27:41.692 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:27:41 smithi101 ceph-mon[128807]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "326f0209-ba9d-4b59-97cc-d61a758a8f29"}]: dispatch 2023-07-14T19:27:41.692 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:27:41 smithi101 ceph-mon[128807]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "326f0209-ba9d-4b59-97cc-d61a758a8f29"}]': finished 2023-07-14T19:27:41.692 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:27:41 smithi101 ceph-mon[128807]: osdmap e67: 11 total, 10 up, 11 in 2023-07-14T19:27:41.693 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:27:41 smithi101 ceph-mon[128807]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2023-07-14T19:27:41.693 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:27:41 smithi101 ceph-mon[128807]: from='client.? 172.21.15.101:0/3977049761' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-07-14T19:27:41.720 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:27:41 smithi139 ceph-mon[129293]: from='client.? 172.21.15.101:0/538320225' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "326f0209-ba9d-4b59-97cc-d61a758a8f29"}]: dispatch 2023-07-14T19:27:41.721 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:27:41 smithi139 ceph-mon[129293]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "326f0209-ba9d-4b59-97cc-d61a758a8f29"}]: dispatch 2023-07-14T19:27:41.721 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:27:41 smithi139 ceph-mon[129293]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "326f0209-ba9d-4b59-97cc-d61a758a8f29"}]': finished 2023-07-14T19:27:41.722 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:27:41 smithi139 ceph-mon[129293]: osdmap e67: 11 total, 10 up, 11 in 2023-07-14T19:27:41.722 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:27:41 smithi139 ceph-mon[129293]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2023-07-14T19:27:41.722 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:27:41 smithi139 ceph-mon[129293]: from='client.? 172.21.15.101:0/3977049761' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-07-14T19:27:41.781 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:27:41 smithi098 ceph-mon[127486]: from='client.? 172.21.15.101:0/538320225' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "326f0209-ba9d-4b59-97cc-d61a758a8f29"}]: dispatch 2023-07-14T19:27:41.781 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:27:41 smithi098 ceph-mon[127486]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "326f0209-ba9d-4b59-97cc-d61a758a8f29"}]: dispatch 2023-07-14T19:27:41.781 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:27:41 smithi098 ceph-mon[127486]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "326f0209-ba9d-4b59-97cc-d61a758a8f29"}]': finished 2023-07-14T19:27:41.782 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:27:41 smithi098 ceph-mon[127486]: osdmap e67: 11 total, 10 up, 11 in 2023-07-14T19:27:41.782 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:27:41 smithi098 ceph-mon[127486]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2023-07-14T19:27:41.782 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:27:41 smithi098 ceph-mon[127486]: from='client.? 172.21.15.101:0/3977049761' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-07-14T19:27:42.691 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:27:42 smithi101 ceph-mon[128807]: pgmap v197: 1 pgs: 1 active+clean; 577 KiB data, 272 MiB used, 894 GiB / 894 GiB avail 2023-07-14T19:27:42.720 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:27:42 smithi139 ceph-mon[129293]: pgmap v197: 1 pgs: 1 active+clean; 577 KiB data, 272 MiB used, 894 GiB / 894 GiB avail 2023-07-14T19:27:42.781 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:27:42 smithi098 ceph-mon[127486]: pgmap v197: 1 pgs: 1 active+clean; 577 KiB data, 272 MiB used, 894 GiB / 894 GiB avail 2023-07-14T19:27:44.893 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:27:44 smithi101 ceph-mon[128807]: pgmap v198: 1 pgs: 1 active+clean; 577 KiB data, 272 MiB used, 894 GiB / 894 GiB avail 2023-07-14T19:27:44.970 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:27:44 smithi139 ceph-mon[129293]: pgmap v198: 1 pgs: 1 active+clean; 577 KiB data, 272 MiB used, 894 GiB / 894 GiB avail 2023-07-14T19:27:45.032 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:27:44 smithi098 ceph-mon[127486]: pgmap v198: 1 pgs: 1 active+clean; 577 KiB data, 272 MiB used, 894 GiB / 894 GiB avail 2023-07-14T19:27:46.942 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:27:46 smithi101 ceph-mon[128807]: pgmap v199: 1 pgs: 1 active+clean; 577 KiB data, 272 MiB used, 894 GiB / 894 GiB avail 2023-07-14T19:27:46.971 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:27:46 smithi139 ceph-mon[129293]: pgmap v199: 1 pgs: 1 active+clean; 577 KiB data, 272 MiB used, 894 GiB / 894 GiB avail 2023-07-14T19:27:47.031 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:27:46 smithi098 ceph-mon[127486]: pgmap v199: 1 pgs: 1 active+clean; 577 KiB data, 272 MiB used, 894 GiB / 894 GiB avail 2023-07-14T19:27:47.941 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:27:47 smithi101 ceph-mon[128807]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.10"}]: dispatch 2023-07-14T19:27:47.941 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:27:47 smithi101 ceph-mon[128807]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T19:27:47.970 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:27:47 smithi139 ceph-mon[129293]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.10"}]: dispatch 2023-07-14T19:27:47.970 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:27:47 smithi139 ceph-mon[129293]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T19:27:48.031 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:27:47 smithi098 ceph-mon[127486]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.10"}]: dispatch 2023-07-14T19:27:48.031 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:27:47 smithi098 ceph-mon[127486]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T19:27:48.942 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:27:48 smithi101 ceph-mon[128807]: Deploying daemon osd.10 on smithi101 2023-07-14T19:27:48.943 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:27:48 smithi101 ceph-mon[128807]: pgmap v200: 1 pgs: 1 active+clean; 577 KiB data, 272 MiB used, 894 GiB / 894 GiB avail 2023-07-14T19:27:48.970 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:27:48 smithi139 ceph-mon[129293]: Deploying daemon osd.10 on smithi101 2023-07-14T19:27:48.970 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:27:48 smithi139 ceph-mon[129293]: pgmap v200: 1 pgs: 1 active+clean; 577 KiB data, 272 MiB used, 894 GiB / 894 GiB avail 2023-07-14T19:27:49.031 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:27:48 smithi098 ceph-mon[127486]: Deploying daemon osd.10 on smithi101 2023-07-14T19:27:49.031 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:27:48 smithi098 ceph-mon[127486]: pgmap v200: 1 pgs: 1 active+clean; 577 KiB data, 272 MiB used, 894 GiB / 894 GiB avail 2023-07-14T19:27:50.922 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:27:50 smithi101 ceph-mon[128807]: pgmap v201: 1 pgs: 1 active+clean; 577 KiB data, 272 MiB used, 894 GiB / 894 GiB avail 2023-07-14T19:27:50.970 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:27:50 smithi139 ceph-mon[129293]: pgmap v201: 1 pgs: 1 active+clean; 577 KiB data, 272 MiB used, 894 GiB / 894 GiB avail 2023-07-14T19:27:51.031 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:27:50 smithi098 ceph-mon[127486]: pgmap v201: 1 pgs: 1 active+clean; 577 KiB data, 272 MiB used, 894 GiB / 894 GiB avail 2023-07-14T19:27:53.070 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:27:52 smithi101 ceph-mon[128807]: pgmap v202: 1 pgs: 1 active+clean; 577 KiB data, 272 MiB used, 894 GiB / 894 GiB avail 2023-07-14T19:27:53.071 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:27:52 smithi101 ceph-mon[128807]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' 2023-07-14T19:27:53.071 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:27:52 smithi101 ceph-mon[128807]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' 2023-07-14T19:27:53.071 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:27:52 smithi101 ceph-mon[128807]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-14T19:27:53.220 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:27:52 smithi139 ceph-mon[129293]: pgmap v202: 1 pgs: 1 active+clean; 577 KiB data, 272 MiB used, 894 GiB / 894 GiB avail 2023-07-14T19:27:53.220 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:27:52 smithi139 ceph-mon[129293]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' 2023-07-14T19:27:53.221 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:27:52 smithi139 ceph-mon[129293]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' 2023-07-14T19:27:53.221 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:27:52 smithi139 ceph-mon[129293]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-14T19:27:53.281 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:27:52 smithi098 ceph-mon[127486]: pgmap v202: 1 pgs: 1 active+clean; 577 KiB data, 272 MiB used, 894 GiB / 894 GiB avail 2023-07-14T19:27:53.281 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:27:52 smithi098 ceph-mon[127486]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' 2023-07-14T19:27:53.282 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:27:52 smithi098 ceph-mon[127486]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' 2023-07-14T19:27:53.282 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:27:52 smithi098 ceph-mon[127486]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-14T19:27:54.779 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-14T19:27:54.776+0000 7f59a2ffd700 1 -- 172.21.15.101:0/2330330211 <== mgr.14152 v2:172.21.15.98:6800/452042164 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+38 (secure 0 0 0) 0x7f59b0069190 con 0x7f599c0a1150 2023-07-14T19:27:54.779 INFO:teuthology.orchestra.run.smithi101.stdout:Created osd(s) 10 on host 'smithi101' 2023-07-14T19:27:54.780 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-14T19:27:54.779+0000 7f59b8190700 1 -- 172.21.15.101:0/2330330211 >> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] conn(0x7f599c0a1150 msgr2=0x7f599c0a3600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:27:54.780 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-14T19:27:54.779+0000 7f59b8190700 1 --2- 172.21.15.101:0/2330330211 >> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] conn(0x7f599c0a1150 0x7f599c0a3600 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f59ac006010 tx=0x7f59ac00b540 comp rx=0 tx=0).stop 2023-07-14T19:27:54.780 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-14T19:27:54.779+0000 7f59b8190700 1 -- 172.21.15.101:0/2330330211 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f59b00fec10 msgr2=0x7f59b01959f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:27:54.780 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-14T19:27:54.779+0000 7f59b8190700 1 --2- 172.21.15.101:0/2330330211 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f59b00fec10 0x7f59b01959f0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f59a400d980 tx=0x7f59a400dd40 comp rx=0 tx=0).stop 2023-07-14T19:27:54.780 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-14T19:27:54.779+0000 7f59b8190700 1 -- 172.21.15.101:0/2330330211 shutdown_connections 2023-07-14T19:27:54.781 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-14T19:27:54.779+0000 7f59b8190700 1 --2- 172.21.15.101:0/2330330211 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f59b01007a0 0x7f59b019cfb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:27:54.781 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-14T19:27:54.779+0000 7f59b8190700 1 --2- 172.21.15.101:0/2330330211 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f59b00fec10 0x7f59b01959f0 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:27:54.781 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-14T19:27:54.779+0000 7f59b8190700 1 --2- 172.21.15.101:0/2330330211 >> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] conn(0x7f599c0a1150 0x7f599c0a3600 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:27:54.781 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-14T19:27:54.779+0000 7f59b8190700 1 --2- 172.21.15.101:0/2330330211 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f59b00ffe10 0x7f59b0195f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:27:54.781 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-14T19:27:54.779+0000 7f59b8190700 1 -- 172.21.15.101:0/2330330211 >> 172.21.15.101:0/2330330211 conn(0x7f59b00fa260 msgr2=0x7f59b0068a80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:27:54.781 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-14T19:27:54.779+0000 7f59b8190700 1 -- 172.21.15.101:0/2330330211 shutdown_connections 2023-07-14T19:27:54.781 INFO:teuthology.orchestra.run.smithi101.stderr:2023-07-14T19:27:54.779+0000 7f59b8190700 1 -- 172.21.15.101:0/2330330211 wait complete. 2023-07-14T19:27:55.146 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:27:54 smithi101 ceph-mon[128807]: pgmap v203: 1 pgs: 1 active+clean; 577 KiB data, 272 MiB used, 894 GiB / 894 GiB avail 2023-07-14T19:27:55.146 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:27:54 smithi101 ceph-mon[128807]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' 2023-07-14T19:27:55.146 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:27:54 smithi101 ceph-mon[128807]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' 2023-07-14T19:27:55.147 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:27:54 smithi101 ceph-mon[128807]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T19:27:55.147 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:27:54 smithi101 ceph-mon[128807]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-14T19:27:55.147 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:27:54 smithi101 ceph-mon[128807]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' 2023-07-14T19:27:55.147 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:27:54 smithi101 ceph-mon[128807]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' 2023-07-14T19:27:55.147 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:27:54 smithi101 ceph-mon[128807]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' 2023-07-14T19:27:55.147 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:27:54 smithi101 ceph-mon[128807]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-14T19:27:55.148 DEBUG:teuthology.orchestra.run.smithi101:osd.10> sudo journalctl -f -n 0 -u ceph-9adea648-227b-11ee-9b33-001a4aab830c@osd.10.service 2023-07-14T19:27:55.151 INFO:tasks.cephadm:Deploying osd.11 on smithi139 with /dev/vg_nvme/lv_1... 2023-07-14T19:27:55.151 DEBUG:teuthology.orchestra.run.smithi139:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81c9387d2e7576da065747ec9c16d1c7d69016ec shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 9adea648-227b-11ee-9b33-001a4aab830c -- ceph-volume lvm zap /dev/vg_nvme/lv_1 2023-07-14T19:27:55.180 INFO:journalctl@ceph.osd.10.smithi101.stdout:-- Logs begin at Fri 2023-07-14 18:56:58 UTC. -- 2023-07-14T19:27:55.183 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:27:54 smithi139 ceph-mon[129293]: pgmap v203: 1 pgs: 1 active+clean; 577 KiB data, 272 MiB used, 894 GiB / 894 GiB avail 2023-07-14T19:27:55.183 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:27:54 smithi139 ceph-mon[129293]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' 2023-07-14T19:27:55.184 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:27:54 smithi139 ceph-mon[129293]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' 2023-07-14T19:27:55.184 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:27:54 smithi139 ceph-mon[129293]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T19:27:55.184 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:27:54 smithi139 ceph-mon[129293]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-14T19:27:55.185 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:27:54 smithi139 ceph-mon[129293]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' 2023-07-14T19:27:55.185 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:27:54 smithi139 ceph-mon[129293]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' 2023-07-14T19:27:55.185 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:27:54 smithi139 ceph-mon[129293]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' 2023-07-14T19:27:55.185 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:27:54 smithi139 ceph-mon[129293]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-14T19:27:55.281 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:27:54 smithi098 ceph-mon[127486]: pgmap v203: 1 pgs: 1 active+clean; 577 KiB data, 272 MiB used, 894 GiB / 894 GiB avail 2023-07-14T19:27:55.281 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:27:54 smithi098 ceph-mon[127486]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' 2023-07-14T19:27:55.281 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:27:54 smithi098 ceph-mon[127486]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' 2023-07-14T19:27:55.282 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:27:54 smithi098 ceph-mon[127486]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T19:27:55.282 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:27:54 smithi098 ceph-mon[127486]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-14T19:27:55.282 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:27:54 smithi098 ceph-mon[127486]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' 2023-07-14T19:27:55.282 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:27:54 smithi098 ceph-mon[127486]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' 2023-07-14T19:27:55.282 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:27:54 smithi098 ceph-mon[127486]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' 2023-07-14T19:27:55.283 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:27:54 smithi098 ceph-mon[127486]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-14T19:27:55.712 INFO:teuthology.orchestra.run.smithi139.stderr:Inferring config /var/lib/ceph/9adea648-227b-11ee-9b33-001a4aab830c/mon.c/config 2023-07-14T19:27:55.941 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:27:55 smithi101 ceph-mon[128807]: from='osd.10 [v2:172.21.15.101:6824/3545244225,v1:172.21.15.101:6825/3545244225]' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2023-07-14T19:27:55.942 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:27:55 smithi101 ceph-mon[128807]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2023-07-14T19:27:55.942 INFO:journalctl@ceph.osd.10.smithi101.stdout:Jul 14 19:27:55 smithi101 ceph-9adea648-227b-11ee-9b33-001a4aab830c-osd-10[150362]: 2023-07-14T19:27:55.842+0000 7f2e87233640 -1 osd.10 0 log_to_monitors true 2023-07-14T19:27:56.220 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:27:55 smithi139 ceph-mon[129293]: from='osd.10 [v2:172.21.15.101:6824/3545244225,v1:172.21.15.101:6825/3545244225]' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2023-07-14T19:27:56.221 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:27:55 smithi139 ceph-mon[129293]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2023-07-14T19:27:56.281 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:27:55 smithi098 ceph-mon[127486]: from='osd.10 [v2:172.21.15.101:6824/3545244225,v1:172.21.15.101:6825/3545244225]' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2023-07-14T19:27:56.282 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:27:55 smithi098 ceph-mon[127486]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2023-07-14T19:27:56.976 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:27:56 smithi139 ceph-mon[129293]: pgmap v204: 1 pgs: 1 active+clean; 577 KiB data, 272 MiB used, 894 GiB / 894 GiB avail 2023-07-14T19:27:56.977 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:27:56 smithi139 ceph-mon[129293]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]': finished 2023-07-14T19:27:56.977 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:27:56 smithi139 ceph-mon[129293]: osdmap e68: 11 total, 10 up, 11 in 2023-07-14T19:27:56.977 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:27:56 smithi139 ceph-mon[129293]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2023-07-14T19:27:56.977 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:27:56 smithi139 ceph-mon[129293]: from='osd.10 [v2:172.21.15.101:6824/3545244225,v1:172.21.15.101:6825/3545244225]' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi101", "root=default"]}]: dispatch 2023-07-14T19:27:56.977 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:27:56 smithi139 ceph-mon[129293]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi101", "root=default"]}]: dispatch 2023-07-14T19:27:57.191 INFO:teuthology.orchestra.run.smithi139.stderr:--> Zapping: /dev/vg_nvme/lv_1 2023-07-14T19:27:57.192 INFO:teuthology.orchestra.run.smithi139.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_1 bs=1M count=10 conv=fsync 2023-07-14T19:27:57.192 INFO:teuthology.orchestra.run.smithi139.stderr: stderr: 10+0 records in 2023-07-14T19:27:57.192 INFO:teuthology.orchestra.run.smithi139.stderr:10+0 records out 2023-07-14T19:27:57.193 INFO:teuthology.orchestra.run.smithi139.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0109529 s, 957 MB/s 2023-07-14T19:27:57.193 INFO:teuthology.orchestra.run.smithi139.stderr:--> Zapping successful for: 2023-07-14T19:27:57.194 INFO:journalctl@ceph.osd.10.smithi101.stdout:Jul 14 19:27:56 smithi101 ceph-9adea648-227b-11ee-9b33-001a4aab830c-osd-10[150362]: 2023-07-14T19:27:56.928+0000 7f2e7c0aa700 -1 osd.10 0 waiting for initial osdmap 2023-07-14T19:27:57.194 INFO:journalctl@ceph.osd.10.smithi101.stdout:Jul 14 19:27:56 smithi101 ceph-9adea648-227b-11ee-9b33-001a4aab830c-osd-10[150362]: 2023-07-14T19:27:56.943+0000 7f2e75e9c700 -1 osd.10 69 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-07-14T19:27:57.194 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:27:56 smithi101 ceph-mon[128807]: pgmap v204: 1 pgs: 1 active+clean; 577 KiB data, 272 MiB used, 894 GiB / 894 GiB avail 2023-07-14T19:27:57.195 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:27:56 smithi101 ceph-mon[128807]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]': finished 2023-07-14T19:27:57.195 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:27:56 smithi101 ceph-mon[128807]: osdmap e68: 11 total, 10 up, 11 in 2023-07-14T19:27:57.195 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:27:56 smithi101 ceph-mon[128807]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2023-07-14T19:27:57.196 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:27:56 smithi101 ceph-mon[128807]: from='osd.10 [v2:172.21.15.101:6824/3545244225,v1:172.21.15.101:6825/3545244225]' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi101", "root=default"]}]: dispatch 2023-07-14T19:27:57.196 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:27:56 smithi101 ceph-mon[128807]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi101", "root=default"]}]: dispatch 2023-07-14T19:27:57.283 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:27:56 smithi098 ceph-mon[127486]: pgmap v204: 1 pgs: 1 active+clean; 577 KiB data, 272 MiB used, 894 GiB / 894 GiB avail 2023-07-14T19:27:57.283 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:27:56 smithi098 ceph-mon[127486]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]': finished 2023-07-14T19:27:57.283 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:27:56 smithi098 ceph-mon[127486]: osdmap e68: 11 total, 10 up, 11 in 2023-07-14T19:27:57.284 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:27:56 smithi098 ceph-mon[127486]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2023-07-14T19:27:57.284 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:27:56 smithi098 ceph-mon[127486]: from='osd.10 [v2:172.21.15.101:6824/3545244225,v1:172.21.15.101:6825/3545244225]' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi101", "root=default"]}]: dispatch 2023-07-14T19:27:57.284 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:27:56 smithi098 ceph-mon[127486]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi101", "root=default"]}]: dispatch 2023-07-14T19:27:57.567 DEBUG:teuthology.orchestra.run.smithi139:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81c9387d2e7576da065747ec9c16d1c7d69016ec shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 9adea648-227b-11ee-9b33-001a4aab830c -- ceph orch daemon add osd smithi139:vg_nvme/lv_1 2023-07-14T19:27:58.158 INFO:teuthology.orchestra.run.smithi139.stderr:Inferring config /var/lib/ceph/9adea648-227b-11ee-9b33-001a4aab830c/mon.c/config 2023-07-14T19:27:58.191 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:27:57 smithi101 ceph-mon[128807]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi101", "root=default"]}]': finished 2023-07-14T19:27:58.192 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:27:57 smithi101 ceph-mon[128807]: osdmap e69: 11 total, 10 up, 11 in 2023-07-14T19:27:58.192 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:27:57 smithi101 ceph-mon[128807]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2023-07-14T19:27:58.192 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:27:57 smithi101 ceph-mon[128807]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2023-07-14T19:27:58.192 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:27:57 smithi101 ceph-mon[128807]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' 2023-07-14T19:27:58.192 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:27:57 smithi101 ceph-mon[128807]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' 2023-07-14T19:27:58.192 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:27:57 smithi101 ceph-mon[128807]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2023-07-14T19:27:58.193 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:27:57 smithi101 ceph-mon[128807]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2023-07-14T19:27:58.193 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:27:57 smithi101 ceph-mon[128807]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2023-07-14T19:27:58.193 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:27:57 smithi101 ceph-mon[128807]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2023-07-14T19:27:58.193 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:27:57 smithi101 ceph-mon[128807]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' 2023-07-14T19:27:58.193 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:27:57 smithi101 ceph-mon[128807]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T19:27:58.193 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:27:57 smithi101 ceph-mon[128807]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-14T19:27:58.194 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:27:57 smithi101 ceph-mon[128807]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' 2023-07-14T19:27:58.220 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:27:57 smithi139 ceph-mon[129293]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi101", "root=default"]}]': finished 2023-07-14T19:27:58.221 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:27:57 smithi139 ceph-mon[129293]: osdmap e69: 11 total, 10 up, 11 in 2023-07-14T19:27:58.221 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:27:57 smithi139 ceph-mon[129293]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2023-07-14T19:27:58.221 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:27:57 smithi139 ceph-mon[129293]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2023-07-14T19:27:58.221 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:27:57 smithi139 ceph-mon[129293]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' 2023-07-14T19:27:58.221 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:27:57 smithi139 ceph-mon[129293]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' 2023-07-14T19:27:58.221 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:27:57 smithi139 ceph-mon[129293]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2023-07-14T19:27:58.222 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:27:57 smithi139 ceph-mon[129293]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2023-07-14T19:27:58.222 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:27:57 smithi139 ceph-mon[129293]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2023-07-14T19:27:58.222 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:27:57 smithi139 ceph-mon[129293]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2023-07-14T19:27:58.222 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:27:57 smithi139 ceph-mon[129293]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' 2023-07-14T19:27:58.222 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:27:57 smithi139 ceph-mon[129293]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T19:27:58.222 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:27:57 smithi139 ceph-mon[129293]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-14T19:27:58.222 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:27:57 smithi139 ceph-mon[129293]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' 2023-07-14T19:27:58.281 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:27:57 smithi098 ceph-mon[127486]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi101", "root=default"]}]': finished 2023-07-14T19:27:58.281 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:27:57 smithi098 ceph-mon[127486]: osdmap e69: 11 total, 10 up, 11 in 2023-07-14T19:27:58.281 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:27:57 smithi098 ceph-mon[127486]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2023-07-14T19:27:58.282 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:27:57 smithi098 ceph-mon[127486]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2023-07-14T19:27:58.282 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:27:57 smithi098 ceph-mon[127486]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' 2023-07-14T19:27:58.282 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:27:57 smithi098 ceph-mon[127486]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' 2023-07-14T19:27:58.282 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:27:57 smithi098 ceph-mon[127486]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2023-07-14T19:27:58.282 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:27:57 smithi098 ceph-mon[127486]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2023-07-14T19:27:58.282 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:27:57 smithi098 ceph-mon[127486]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2023-07-14T19:27:58.282 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:27:57 smithi098 ceph-mon[127486]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2023-07-14T19:27:58.283 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:27:57 smithi098 ceph-mon[127486]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' 2023-07-14T19:27:58.283 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:27:57 smithi098 ceph-mon[127486]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T19:27:58.283 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:27:57 smithi098 ceph-mon[127486]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-14T19:27:58.283 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:27:57 smithi098 ceph-mon[127486]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' 2023-07-14T19:27:59.191 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:27:58 smithi101 ceph-mon[128807]: purged_snaps scrub starts 2023-07-14T19:27:59.192 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:27:58 smithi101 ceph-mon[128807]: purged_snaps scrub ok 2023-07-14T19:27:59.192 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:27:58 smithi101 ceph-mon[128807]: Detected new or changed devices on smithi101 2023-07-14T19:27:59.192 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:27:58 smithi101 ceph-mon[128807]: Adjusting osd_memory_target on smithi101 to 4292M 2023-07-14T19:27:59.192 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:27:58 smithi101 ceph-mon[128807]: pgmap v207: 1 pgs: 1 active+clean; 577 KiB data, 272 MiB used, 894 GiB / 894 GiB avail 2023-07-14T19:27:59.192 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:27:58 smithi101 ceph-mon[128807]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2023-07-14T19:27:59.192 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:27:58 smithi101 ceph-mon[128807]: osd.10 [v2:172.21.15.101:6824/3545244225,v1:172.21.15.101:6825/3545244225] boot 2023-07-14T19:27:59.193 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:27:58 smithi101 ceph-mon[128807]: osdmap e70: 11 total, 11 up, 11 in 2023-07-14T19:27:59.193 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:27:58 smithi101 ceph-mon[128807]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2023-07-14T19:27:59.220 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:27:58 smithi139 ceph-mon[129293]: purged_snaps scrub starts 2023-07-14T19:27:59.221 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:27:58 smithi139 ceph-mon[129293]: purged_snaps scrub ok 2023-07-14T19:27:59.221 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:27:58 smithi139 ceph-mon[129293]: Detected new or changed devices on smithi101 2023-07-14T19:27:59.221 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:27:58 smithi139 ceph-mon[129293]: Adjusting osd_memory_target on smithi101 to 4292M 2023-07-14T19:27:59.221 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:27:58 smithi139 ceph-mon[129293]: pgmap v207: 1 pgs: 1 active+clean; 577 KiB data, 272 MiB used, 894 GiB / 894 GiB avail 2023-07-14T19:27:59.221 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:27:58 smithi139 ceph-mon[129293]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2023-07-14T19:27:59.221 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:27:58 smithi139 ceph-mon[129293]: osd.10 [v2:172.21.15.101:6824/3545244225,v1:172.21.15.101:6825/3545244225] boot 2023-07-14T19:27:59.222 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:27:58 smithi139 ceph-mon[129293]: osdmap e70: 11 total, 11 up, 11 in 2023-07-14T19:27:59.222 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:27:58 smithi139 ceph-mon[129293]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2023-07-14T19:27:59.281 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:27:58 smithi098 ceph-mon[127486]: purged_snaps scrub starts 2023-07-14T19:27:59.281 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:27:58 smithi098 ceph-mon[127486]: purged_snaps scrub ok 2023-07-14T19:27:59.282 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:27:58 smithi098 ceph-mon[127486]: Detected new or changed devices on smithi101 2023-07-14T19:27:59.282 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:27:58 smithi098 ceph-mon[127486]: Adjusting osd_memory_target on smithi101 to 4292M 2023-07-14T19:27:59.282 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:27:58 smithi098 ceph-mon[127486]: pgmap v207: 1 pgs: 1 active+clean; 577 KiB data, 272 MiB used, 894 GiB / 894 GiB avail 2023-07-14T19:27:59.282 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:27:58 smithi098 ceph-mon[127486]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2023-07-14T19:27:59.282 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:27:58 smithi098 ceph-mon[127486]: osd.10 [v2:172.21.15.101:6824/3545244225,v1:172.21.15.101:6825/3545244225] boot 2023-07-14T19:27:59.282 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:27:58 smithi098 ceph-mon[127486]: osdmap e70: 11 total, 11 up, 11 in 2023-07-14T19:27:59.282 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:27:58 smithi098 ceph-mon[127486]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2023-07-14T19:27:59.431 INFO:teuthology.orchestra.run.smithi139.stderr:2023-07-14T19:27:59.430+0000 7f86f3996700 1 -- 172.21.15.139:0/3392183940 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f86ec0ff500 msgr2=0x7f86ec0ff950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:27:59.431 INFO:teuthology.orchestra.run.smithi139.stderr:2023-07-14T19:27:59.430+0000 7f86f3996700 1 --2- 172.21.15.139:0/3392183940 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f86ec0ff500 0x7f86ec0ff950 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f86e8009990 tx=0x7f86e8009ca0 comp rx=0 tx=0).stop 2023-07-14T19:27:59.431 INFO:teuthology.orchestra.run.smithi139.stderr:2023-07-14T19:27:59.430+0000 7f86f3996700 1 -- 172.21.15.139:0/3392183940 shutdown_connections 2023-07-14T19:27:59.432 INFO:teuthology.orchestra.run.smithi139.stderr:2023-07-14T19:27:59.430+0000 7f86f3996700 1 --2- 172.21.15.139:0/3392183940 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f86ec0ffe90 0x7f86ec106750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:27:59.432 INFO:teuthology.orchestra.run.smithi139.stderr:2023-07-14T19:27:59.430+0000 7f86f3996700 1 --2- 172.21.15.139:0/3392183940 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f86ec0fe300 0x7f86ec0fe710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:27:59.432 INFO:teuthology.orchestra.run.smithi139.stderr:2023-07-14T19:27:59.430+0000 7f86f3996700 1 --2- 172.21.15.139:0/3392183940 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f86ec0ff500 0x7f86ec0ff950 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:27:59.432 INFO:teuthology.orchestra.run.smithi139.stderr:2023-07-14T19:27:59.430+0000 7f86f3996700 1 -- 172.21.15.139:0/3392183940 >> 172.21.15.139:0/3392183940 conn(0x7f86ec0f9950 msgr2=0x7f86ec0fbd80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:27:59.432 INFO:teuthology.orchestra.run.smithi139.stderr:2023-07-14T19:27:59.431+0000 7f86f3996700 1 -- 172.21.15.139:0/3392183940 shutdown_connections 2023-07-14T19:27:59.432 INFO:teuthology.orchestra.run.smithi139.stderr:2023-07-14T19:27:59.431+0000 7f86f3996700 1 -- 172.21.15.139:0/3392183940 wait complete. 2023-07-14T19:27:59.434 INFO:teuthology.orchestra.run.smithi139.stderr:2023-07-14T19:27:59.431+0000 7f86f3996700 1 Processor -- start 2023-07-14T19:27:59.434 INFO:teuthology.orchestra.run.smithi139.stderr:2023-07-14T19:27:59.431+0000 7f86f3996700 1 -- start start 2023-07-14T19:27:59.434 INFO:teuthology.orchestra.run.smithi139.stderr:2023-07-14T19:27:59.431+0000 7f86f3996700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f86ec0fe300 0x7f86ec195930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:27:59.434 INFO:teuthology.orchestra.run.smithi139.stderr:2023-07-14T19:27:59.431+0000 7f86f3996700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f86ec0ff500 0x7f86ec195e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:27:59.434 INFO:teuthology.orchestra.run.smithi139.stderr:2023-07-14T19:27:59.431+0000 7f86f3996700 1 --2- >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f86ec0ffe90 0x7f86ec19cef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:27:59.435 INFO:teuthology.orchestra.run.smithi139.stderr:2023-07-14T19:27:59.431+0000 7f86f3996700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f86ec19d4c0 con 0x7f86ec0ff500 2023-07-14T19:27:59.435 INFO:teuthology.orchestra.run.smithi139.stderr:2023-07-14T19:27:59.431+0000 7f86f3996700 1 -- --> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f86ec19d600 con 0x7f86ec0ffe90 2023-07-14T19:27:59.435 INFO:teuthology.orchestra.run.smithi139.stderr:2023-07-14T19:27:59.431+0000 7f86f3996700 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f86ec19d790 con 0x7f86ec0fe300 2023-07-14T19:27:59.435 INFO:teuthology.orchestra.run.smithi139.stderr:2023-07-14T19:27:59.431+0000 7f86f1732700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f86ec0fe300 0x7f86ec195930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:27:59.435 INFO:teuthology.orchestra.run.smithi139.stderr:2023-07-14T19:27:59.431+0000 7f86f1732700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f86ec0fe300 0x7f86ec195930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.139:3300/0 says I am v2:172.21.15.139:37438/0 (socket says 172.21.15.139:37438) 2023-07-14T19:27:59.435 INFO:teuthology.orchestra.run.smithi139.stderr:2023-07-14T19:27:59.431+0000 7f86f1732700 1 -- 172.21.15.139:0/3424028319 learned_addr learned my addr 172.21.15.139:0/3424028319 (peer_addr_for_me v2:172.21.15.139:0/0) 2023-07-14T19:27:59.435 INFO:teuthology.orchestra.run.smithi139.stderr:2023-07-14T19:27:59.432+0000 7f86f1f33700 1 --2- 172.21.15.139:0/3424028319 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f86ec0ffe90 0x7f86ec19cef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:27:59.435 INFO:teuthology.orchestra.run.smithi139.stderr:2023-07-14T19:27:59.432+0000 7f86f0f31700 1 --2- 172.21.15.139:0/3424028319 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f86ec0ff500 0x7f86ec195e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:27:59.436 INFO:teuthology.orchestra.run.smithi139.stderr:2023-07-14T19:27:59.432+0000 7f86f1732700 1 -- 172.21.15.139:0/3424028319 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f86ec0ffe90 msgr2=0x7f86ec19cef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:27:59.436 INFO:teuthology.orchestra.run.smithi139.stderr:2023-07-14T19:27:59.432+0000 7f86f1732700 1 --2- 172.21.15.139:0/3424028319 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f86ec0ffe90 0x7f86ec19cef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:27:59.436 INFO:teuthology.orchestra.run.smithi139.stderr:2023-07-14T19:27:59.432+0000 7f86f1732700 1 -- 172.21.15.139:0/3424028319 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f86ec0ff500 msgr2=0x7f86ec195e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:27:59.436 INFO:teuthology.orchestra.run.smithi139.stderr:2023-07-14T19:27:59.432+0000 7f86f1732700 1 --2- 172.21.15.139:0/3424028319 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f86ec0ff500 0x7f86ec195e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:27:59.436 INFO:teuthology.orchestra.run.smithi139.stderr:2023-07-14T19:27:59.432+0000 7f86f1732700 1 -- 172.21.15.139:0/3424028319 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f86e80095f0 con 0x7f86ec0fe300 2023-07-14T19:27:59.436 INFO:teuthology.orchestra.run.smithi139.stderr:2023-07-14T19:27:59.432+0000 7f86f0f31700 1 --2- 172.21.15.139:0/3424028319 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f86ec0ff500 0x7f86ec195e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-07-14T19:27:59.437 INFO:teuthology.orchestra.run.smithi139.stderr:2023-07-14T19:27:59.432+0000 7f86f1732700 1 --2- 172.21.15.139:0/3424028319 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f86ec0fe300 0x7f86ec195930 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f86dc00d9b0 tx=0x7f86dc00dd70 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:27:59.437 INFO:teuthology.orchestra.run.smithi139.stderr:2023-07-14T19:27:59.432+0000 7f86e27fc700 1 -- 172.21.15.139:0/3424028319 <== mon.1 v2:172.21.15.139:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f86dc00bb60 con 0x7f86ec0fe300 2023-07-14T19:27:59.437 INFO:teuthology.orchestra.run.smithi139.stderr:2023-07-14T19:27:59.432+0000 7f86f3996700 1 -- 172.21.15.139:0/3424028319 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f86ec19da70 con 0x7f86ec0fe300 2023-07-14T19:27:59.437 INFO:teuthology.orchestra.run.smithi139.stderr:2023-07-14T19:27:59.432+0000 7f86e27fc700 1 -- 172.21.15.139:0/3424028319 <== mon.1 v2:172.21.15.139:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f86dc007620 con 0x7f86ec0fe300 2023-07-14T19:27:59.437 INFO:teuthology.orchestra.run.smithi139.stderr:2023-07-14T19:27:59.432+0000 7f86e27fc700 1 -- 172.21.15.139:0/3424028319 <== mon.1 v2:172.21.15.139:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f86dc00f620 con 0x7f86ec0fe300 2023-07-14T19:27:59.437 INFO:teuthology.orchestra.run.smithi139.stderr:2023-07-14T19:27:59.432+0000 7f86f3996700 1 -- 172.21.15.139:0/3424028319 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f86ec19df70 con 0x7f86ec0fe300 2023-07-14T19:27:59.437 INFO:teuthology.orchestra.run.smithi139.stderr:2023-07-14T19:27:59.433+0000 7f86f3996700 1 -- 172.21.15.139:0/3424028319 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f86ec105920 con 0x7f86ec0fe300 2023-07-14T19:27:59.439 INFO:teuthology.orchestra.run.smithi139.stderr:2023-07-14T19:27:59.434+0000 7f86e27fc700 1 -- 172.21.15.139:0/3424028319 <== mon.1 v2:172.21.15.139:3300/0 4 ==== mgrmap(e 16) v1 ==== 135715+0+0 (secure 0 0 0) 0x7f86dc010460 con 0x7f86ec0fe300 2023-07-14T19:27:59.439 INFO:teuthology.orchestra.run.smithi139.stderr:2023-07-14T19:27:59.434+0000 7f86e27fc700 1 --2- 172.21.15.139:0/3424028319 >> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] conn(0x7f86d80a1150 0x7f86d80a3600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:27:59.439 INFO:teuthology.orchestra.run.smithi139.stderr:2023-07-14T19:27:59.435+0000 7f86e27fc700 1 -- 172.21.15.139:0/3424028319 <== mon.1 v2:172.21.15.139:3300/0 5 ==== osd_map(71..71 src has 1..71) v4 ==== 6723+0+0 (secure 0 0 0) 0x7f86dc0cca50 con 0x7f86ec0fe300 2023-07-14T19:27:59.439 INFO:teuthology.orchestra.run.smithi139.stderr:2023-07-14T19:27:59.435+0000 7f86f0f31700 1 --2- 172.21.15.139:0/3424028319 >> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] conn(0x7f86d80a1150 0x7f86d80a3600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:27:59.440 INFO:teuthology.orchestra.run.smithi139.stderr:2023-07-14T19:27:59.435+0000 7f86f0f31700 1 --2- 172.21.15.139:0/3424028319 >> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] conn(0x7f86d80a1150 0x7f86d80a3600 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f86e800b5c0 tx=0x7f86e8005fb0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:27:59.440 INFO:teuthology.orchestra.run.smithi139.stderr:2023-07-14T19:27:59.438+0000 7f86e27fc700 1 -- 172.21.15.139:0/3424028319 <== mon.1 v2:172.21.15.139:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f86dc0967d0 con 0x7f86ec0fe300 2023-07-14T19:27:59.632 INFO:teuthology.orchestra.run.smithi139.stderr:2023-07-14T19:27:59.631+0000 7f86f3996700 1 -- 172.21.15.139:0/3424028319 --> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi139:vg_nvme/lv_1", "target": ["mon-mgr", ""]}) v1 -- 0x7f86ec060a80 con 0x7f86d80a1150 2023-07-14T19:28:00.191 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:27:59 smithi101 ceph-mon[128807]: osdmap e71: 11 total, 11 up, 11 in 2023-07-14T19:28:00.191 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:27:59 smithi101 ceph-mon[128807]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-07-14T19:28:00.191 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:27:59 smithi101 ceph-mon[128807]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-07-14T19:28:00.192 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:27:59 smithi101 ceph-mon[128807]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T19:28:00.220 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:27:59 smithi139 ceph-mon[129293]: osdmap e71: 11 total, 11 up, 11 in 2023-07-14T19:28:00.220 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:27:59 smithi139 ceph-mon[129293]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-07-14T19:28:00.221 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:27:59 smithi139 ceph-mon[129293]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-07-14T19:28:00.221 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:27:59 smithi139 ceph-mon[129293]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T19:28:00.281 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:27:59 smithi098 ceph-mon[127486]: osdmap e71: 11 total, 11 up, 11 in 2023-07-14T19:28:00.281 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:27:59 smithi098 ceph-mon[127486]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-07-14T19:28:00.281 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:27:59 smithi098 ceph-mon[127486]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-07-14T19:28:00.281 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:27:59 smithi098 ceph-mon[127486]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T19:28:01.220 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:28:00 smithi139 ceph-mon[129293]: pgmap v210: 1 pgs: 1 active+clean; 577 KiB data, 299 MiB used, 983 GiB / 983 GiB avail 2023-07-14T19:28:01.221 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:28:00 smithi139 ceph-mon[129293]: from='client.24358 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi139:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2023-07-14T19:28:01.221 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:28:00 smithi139 ceph-mon[129293]: osdmap e72: 11 total, 11 up, 11 in 2023-07-14T19:28:01.281 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:28:00 smithi098 ceph-mon[127486]: pgmap v210: 1 pgs: 1 active+clean; 577 KiB data, 299 MiB used, 983 GiB / 983 GiB avail 2023-07-14T19:28:01.281 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:28:00 smithi098 ceph-mon[127486]: from='client.24358 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi139:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2023-07-14T19:28:01.281 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:28:00 smithi098 ceph-mon[127486]: osdmap e72: 11 total, 11 up, 11 in 2023-07-14T19:28:01.441 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:28:00 smithi101 ceph-mon[128807]: pgmap v210: 1 pgs: 1 active+clean; 577 KiB data, 299 MiB used, 983 GiB / 983 GiB avail 2023-07-14T19:28:01.442 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:28:00 smithi101 ceph-mon[128807]: from='client.24358 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi139:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2023-07-14T19:28:01.442 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:28:00 smithi101 ceph-mon[128807]: osdmap e72: 11 total, 11 up, 11 in 2023-07-14T19:28:01.949 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Jul 14 19:28:01 smithi098 ceph-9adea648-227b-11ee-9b33-001a4aab830c-mgr-x[127734]: 2023-07-14T19:28:01.551+0000 7f24838a5700 -1 calc_pg_upmaps abort due to max <= 0 2023-07-14T19:28:02.220 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:28:01 smithi139 ceph-mon[129293]: osdmap e73: 11 total, 11 up, 11 in 2023-07-14T19:28:02.221 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:28:01 smithi139 ceph-mon[129293]: osdmap e74: 11 total, 11 up, 11 in 2023-07-14T19:28:02.281 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:28:01 smithi098 ceph-mon[127486]: osdmap e73: 11 total, 11 up, 11 in 2023-07-14T19:28:02.281 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:28:01 smithi098 ceph-mon[127486]: osdmap e74: 11 total, 11 up, 11 in 2023-07-14T19:28:02.441 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:28:01 smithi101 ceph-mon[128807]: osdmap e73: 11 total, 11 up, 11 in 2023-07-14T19:28:02.441 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:28:01 smithi101 ceph-mon[128807]: osdmap e74: 11 total, 11 up, 11 in 2023-07-14T19:28:03.281 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:28:02 smithi098 ceph-mon[127486]: pgmap v213: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 300 MiB used, 983 GiB / 983 GiB avail; 144 KiB/s, 0 objects/s recovering 2023-07-14T19:28:03.281 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:28:02 smithi098 ceph-mon[127486]: from='client.? 172.21.15.139:0/77879247' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "c61c0320-1057-4f17-9bf1-7ad306ed7ed3"}]: dispatch 2023-07-14T19:28:03.282 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:28:02 smithi098 ceph-mon[127486]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "c61c0320-1057-4f17-9bf1-7ad306ed7ed3"}]: dispatch 2023-07-14T19:28:03.282 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:28:02 smithi098 ceph-mon[127486]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "c61c0320-1057-4f17-9bf1-7ad306ed7ed3"}]': finished 2023-07-14T19:28:03.282 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:28:02 smithi098 ceph-mon[127486]: osdmap e75: 12 total, 11 up, 12 in 2023-07-14T19:28:03.282 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:28:02 smithi098 ceph-mon[127486]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2023-07-14T19:28:03.282 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:28:02 smithi098 ceph-mon[127486]: from='client.? 172.21.15.139:0/3058866774' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-07-14T19:28:03.441 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:28:02 smithi101 ceph-mon[128807]: pgmap v213: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 300 MiB used, 983 GiB / 983 GiB avail; 144 KiB/s, 0 objects/s recovering 2023-07-14T19:28:03.441 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:28:02 smithi101 ceph-mon[128807]: from='client.? 172.21.15.139:0/77879247' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "c61c0320-1057-4f17-9bf1-7ad306ed7ed3"}]: dispatch 2023-07-14T19:28:03.442 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:28:02 smithi101 ceph-mon[128807]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "c61c0320-1057-4f17-9bf1-7ad306ed7ed3"}]: dispatch 2023-07-14T19:28:03.442 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:28:02 smithi101 ceph-mon[128807]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "c61c0320-1057-4f17-9bf1-7ad306ed7ed3"}]': finished 2023-07-14T19:28:03.442 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:28:02 smithi101 ceph-mon[128807]: osdmap e75: 12 total, 11 up, 12 in 2023-07-14T19:28:03.442 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:28:02 smithi101 ceph-mon[128807]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2023-07-14T19:28:03.442 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:28:02 smithi101 ceph-mon[128807]: from='client.? 172.21.15.139:0/3058866774' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-07-14T19:28:03.470 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:28:03 smithi139 ceph-mon[129293]: pgmap v213: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 300 MiB used, 983 GiB / 983 GiB avail; 144 KiB/s, 0 objects/s recovering 2023-07-14T19:28:03.470 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:28:03 smithi139 ceph-mon[129293]: from='client.? 172.21.15.139:0/77879247' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "c61c0320-1057-4f17-9bf1-7ad306ed7ed3"}]: dispatch 2023-07-14T19:28:03.470 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:28:03 smithi139 ceph-mon[129293]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "c61c0320-1057-4f17-9bf1-7ad306ed7ed3"}]: dispatch 2023-07-14T19:28:03.471 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:28:03 smithi139 ceph-mon[129293]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "c61c0320-1057-4f17-9bf1-7ad306ed7ed3"}]': finished 2023-07-14T19:28:03.471 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:28:03 smithi139 ceph-mon[129293]: osdmap e75: 12 total, 11 up, 12 in 2023-07-14T19:28:03.471 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:28:03 smithi139 ceph-mon[129293]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2023-07-14T19:28:03.471 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:28:03 smithi139 ceph-mon[129293]: from='client.? 172.21.15.139:0/3058866774' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-07-14T19:28:05.281 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:28:04 smithi098 ceph-mon[127486]: pgmap v216: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 300 MiB used, 983 GiB / 983 GiB avail; 144 KiB/s, 0 objects/s recovering 2023-07-14T19:28:05.441 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:28:04 smithi101 ceph-mon[128807]: pgmap v216: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 300 MiB used, 983 GiB / 983 GiB avail; 144 KiB/s, 0 objects/s recovering 2023-07-14T19:28:05.470 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:28:04 smithi139 ceph-mon[129293]: pgmap v216: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 300 MiB used, 983 GiB / 983 GiB avail; 144 KiB/s, 0 objects/s recovering 2023-07-14T19:28:07.242 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:28:06 smithi139 ceph-mon[129293]: pgmap v217: 1 pgs: 1 active+clean; 577 KiB data, 301 MiB used, 983 GiB / 983 GiB avail; 102 KiB/s, 0 objects/s recovering 2023-07-14T19:28:07.281 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:28:06 smithi098 ceph-mon[127486]: pgmap v217: 1 pgs: 1 active+clean; 577 KiB data, 301 MiB used, 983 GiB / 983 GiB avail; 102 KiB/s, 0 objects/s recovering 2023-07-14T19:28:07.442 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:28:06 smithi101 ceph-mon[128807]: pgmap v217: 1 pgs: 1 active+clean; 577 KiB data, 301 MiB used, 983 GiB / 983 GiB avail; 102 KiB/s, 0 objects/s recovering 2023-07-14T19:28:09.220 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:28:08 smithi139 ceph-mon[129293]: pgmap v218: 1 pgs: 1 active+clean; 577 KiB data, 300 MiB used, 983 GiB / 983 GiB avail 2023-07-14T19:28:09.220 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:28:08 smithi139 ceph-mon[129293]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.11"}]: dispatch 2023-07-14T19:28:09.221 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:28:08 smithi139 ceph-mon[129293]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T19:28:09.281 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:28:08 smithi098 ceph-mon[127486]: pgmap v218: 1 pgs: 1 active+clean; 577 KiB data, 300 MiB used, 983 GiB / 983 GiB avail 2023-07-14T19:28:09.281 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:28:08 smithi098 ceph-mon[127486]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.11"}]: dispatch 2023-07-14T19:28:09.281 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:28:08 smithi098 ceph-mon[127486]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T19:28:09.441 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:28:08 smithi101 ceph-mon[128807]: pgmap v218: 1 pgs: 1 active+clean; 577 KiB data, 300 MiB used, 983 GiB / 983 GiB avail 2023-07-14T19:28:09.442 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:28:08 smithi101 ceph-mon[128807]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.11"}]: dispatch 2023-07-14T19:28:09.442 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:28:08 smithi101 ceph-mon[128807]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T19:28:10.220 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:28:09 smithi139 ceph-mon[129293]: Deploying daemon osd.11 on smithi139 2023-07-14T19:28:10.281 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:28:09 smithi098 ceph-mon[127486]: Deploying daemon osd.11 on smithi139 2023-07-14T19:28:10.441 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:28:09 smithi101 ceph-mon[128807]: Deploying daemon osd.11 on smithi139 2023-07-14T19:28:11.250 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:28:10 smithi139 ceph-mon[129293]: pgmap v219: 1 pgs: 1 active+clean; 577 KiB data, 300 MiB used, 983 GiB / 983 GiB avail 2023-07-14T19:28:11.281 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:28:10 smithi098 ceph-mon[127486]: pgmap v219: 1 pgs: 1 active+clean; 577 KiB data, 300 MiB used, 983 GiB / 983 GiB avail 2023-07-14T19:28:11.441 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:28:10 smithi101 ceph-mon[128807]: pgmap v219: 1 pgs: 1 active+clean; 577 KiB data, 300 MiB used, 983 GiB / 983 GiB avail 2023-07-14T19:28:12.691 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:28:12 smithi101 ceph-mon[128807]: pgmap v220: 1 pgs: 1 active+clean; 577 KiB data, 300 MiB used, 983 GiB / 983 GiB avail 2023-07-14T19:28:12.721 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:28:12 smithi139 ceph-mon[129293]: pgmap v220: 1 pgs: 1 active+clean; 577 KiB data, 300 MiB used, 983 GiB / 983 GiB avail 2023-07-14T19:28:12.781 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:28:12 smithi098 ceph-mon[127486]: pgmap v220: 1 pgs: 1 active+clean; 577 KiB data, 300 MiB used, 983 GiB / 983 GiB avail 2023-07-14T19:28:14.569 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:28:14 smithi139 ceph-mon[129293]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' 2023-07-14T19:28:14.569 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:28:14 smithi139 ceph-mon[129293]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' 2023-07-14T19:28:14.569 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:28:14 smithi139 ceph-mon[129293]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-14T19:28:14.569 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:28:14 smithi139 ceph-mon[129293]: pgmap v221: 1 pgs: 1 active+clean; 577 KiB data, 300 MiB used, 983 GiB / 983 GiB avail 2023-07-14T19:28:14.691 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:28:14 smithi101 ceph-mon[128807]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' 2023-07-14T19:28:14.691 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:28:14 smithi101 ceph-mon[128807]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' 2023-07-14T19:28:14.692 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:28:14 smithi101 ceph-mon[128807]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-14T19:28:14.692 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:28:14 smithi101 ceph-mon[128807]: pgmap v221: 1 pgs: 1 active+clean; 577 KiB data, 300 MiB used, 983 GiB / 983 GiB avail 2023-07-14T19:28:14.781 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:28:14 smithi098 ceph-mon[127486]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' 2023-07-14T19:28:14.781 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:28:14 smithi098 ceph-mon[127486]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' 2023-07-14T19:28:14.781 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:28:14 smithi098 ceph-mon[127486]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-14T19:28:14.782 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:28:14 smithi098 ceph-mon[127486]: pgmap v221: 1 pgs: 1 active+clean; 577 KiB data, 300 MiB used, 983 GiB / 983 GiB avail 2023-07-14T19:28:16.143 INFO:teuthology.orchestra.run.smithi139.stderr:2023-07-14T19:28:16.142+0000 7f86e27fc700 1 -- 172.21.15.139:0/3424028319 <== mgr.14152 v2:172.21.15.98:6800/452042164 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+38 (secure 0 0 0) 0x7f86ec060a80 con 0x7f86d80a1150 2023-07-14T19:28:16.144 INFO:teuthology.orchestra.run.smithi139.stdout:Created osd(s) 11 on host 'smithi139' 2023-07-14T19:28:16.146 INFO:teuthology.orchestra.run.smithi139.stderr:2023-07-14T19:28:16.144+0000 7f86f3996700 1 -- 172.21.15.139:0/3424028319 >> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] conn(0x7f86d80a1150 msgr2=0x7f86d80a3600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:28:16.146 INFO:teuthology.orchestra.run.smithi139.stderr:2023-07-14T19:28:16.144+0000 7f86f3996700 1 --2- 172.21.15.139:0/3424028319 >> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] conn(0x7f86d80a1150 0x7f86d80a3600 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f86e800b5c0 tx=0x7f86e8005fb0 comp rx=0 tx=0).stop 2023-07-14T19:28:16.146 INFO:teuthology.orchestra.run.smithi139.stderr:2023-07-14T19:28:16.144+0000 7f86f3996700 1 -- 172.21.15.139:0/3424028319 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f86ec0fe300 msgr2=0x7f86ec195930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:28:16.146 INFO:teuthology.orchestra.run.smithi139.stderr:2023-07-14T19:28:16.144+0000 7f86f3996700 1 --2- 172.21.15.139:0/3424028319 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f86ec0fe300 0x7f86ec195930 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f86dc00d9b0 tx=0x7f86dc00dd70 comp rx=0 tx=0).stop 2023-07-14T19:28:16.146 INFO:teuthology.orchestra.run.smithi139.stderr:2023-07-14T19:28:16.144+0000 7f86f3996700 1 -- 172.21.15.139:0/3424028319 shutdown_connections 2023-07-14T19:28:16.146 INFO:teuthology.orchestra.run.smithi139.stderr:2023-07-14T19:28:16.144+0000 7f86f3996700 1 --2- 172.21.15.139:0/3424028319 >> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] conn(0x7f86d80a1150 0x7f86d80a3600 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:28:16.147 INFO:teuthology.orchestra.run.smithi139.stderr:2023-07-14T19:28:16.144+0000 7f86f3996700 1 --2- 172.21.15.139:0/3424028319 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f86ec0fe300 0x7f86ec195930 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:28:16.147 INFO:teuthology.orchestra.run.smithi139.stderr:2023-07-14T19:28:16.144+0000 7f86f3996700 1 --2- 172.21.15.139:0/3424028319 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f86ec0ffe90 0x7f86ec19cef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:28:16.147 INFO:teuthology.orchestra.run.smithi139.stderr:2023-07-14T19:28:16.144+0000 7f86f3996700 1 --2- 172.21.15.139:0/3424028319 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f86ec0ff500 0x7f86ec195e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:28:16.147 INFO:teuthology.orchestra.run.smithi139.stderr:2023-07-14T19:28:16.144+0000 7f86f3996700 1 -- 172.21.15.139:0/3424028319 >> 172.21.15.139:0/3424028319 conn(0x7f86ec0f9950 msgr2=0x7f86ec102a60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:28:16.147 INFO:teuthology.orchestra.run.smithi139.stderr:2023-07-14T19:28:16.144+0000 7f86f3996700 1 -- 172.21.15.139:0/3424028319 shutdown_connections 2023-07-14T19:28:16.147 INFO:teuthology.orchestra.run.smithi139.stderr:2023-07-14T19:28:16.144+0000 7f86f3996700 1 -- 172.21.15.139:0/3424028319 wait complete. 2023-07-14T19:28:16.536 DEBUG:teuthology.orchestra.run.smithi139:osd.11> sudo journalctl -f -n 0 -u ceph-9adea648-227b-11ee-9b33-001a4aab830c@osd.11.service 2023-07-14T19:28:16.539 INFO:tasks.cephadm:Waiting for 12 OSDs to come up... 2023-07-14T19:28:16.539 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81c9387d2e7576da065747ec9c16d1c7d69016ec shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 9adea648-227b-11ee-9b33-001a4aab830c -- ceph osd stat -f json 2023-07-14T19:28:16.572 INFO:journalctl@ceph.osd.11.smithi139.stdout:-- Logs begin at Fri 2023-07-14 18:56:10 UTC. -- 2023-07-14T19:28:16.941 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:28:16 smithi101 ceph-mon[128807]: pgmap v222: 1 pgs: 1 active+clean; 577 KiB data, 300 MiB used, 983 GiB / 983 GiB avail 2023-07-14T19:28:16.942 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:28:16 smithi101 ceph-mon[128807]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' 2023-07-14T19:28:16.942 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:28:16 smithi101 ceph-mon[128807]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' 2023-07-14T19:28:16.942 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:28:16 smithi101 ceph-mon[128807]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' 2023-07-14T19:28:16.942 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:28:16 smithi101 ceph-mon[128807]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' 2023-07-14T19:28:16.970 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:28:16 smithi139 ceph-mon[129293]: pgmap v222: 1 pgs: 1 active+clean; 577 KiB data, 300 MiB used, 983 GiB / 983 GiB avail 2023-07-14T19:28:16.971 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:28:16 smithi139 ceph-mon[129293]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' 2023-07-14T19:28:16.971 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:28:16 smithi139 ceph-mon[129293]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' 2023-07-14T19:28:16.971 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:28:16 smithi139 ceph-mon[129293]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' 2023-07-14T19:28:16.971 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:28:16 smithi139 ceph-mon[129293]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' 2023-07-14T19:28:17.031 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:28:16 smithi098 ceph-mon[127486]: pgmap v222: 1 pgs: 1 active+clean; 577 KiB data, 300 MiB used, 983 GiB / 983 GiB avail 2023-07-14T19:28:17.031 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:28:16 smithi098 ceph-mon[127486]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' 2023-07-14T19:28:17.032 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:28:16 smithi098 ceph-mon[127486]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' 2023-07-14T19:28:17.032 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:28:16 smithi098 ceph-mon[127486]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' 2023-07-14T19:28:17.032 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:28:16 smithi098 ceph-mon[127486]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' 2023-07-14T19:28:17.108 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/9adea648-227b-11ee-9b33-001a4aab830c/mon.a/config 2023-07-14T19:28:17.637 INFO:journalctl@ceph.osd.11.smithi139.stdout:Jul 14 19:28:17 smithi139 ceph-9adea648-227b-11ee-9b33-001a4aab830c-osd-11[151655]: 2023-07-14T19:28:17.360+0000 7f50b8b2b640 -1 osd.11 0 log_to_monitors true 2023-07-14T19:28:17.918 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:28:17 smithi139 ceph-mon[129293]: from='osd.11 [v2:172.21.15.139:6824/2630023454,v1:172.21.15.139:6825/2630023454]' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2023-07-14T19:28:17.941 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:28:17 smithi101 ceph-mon[128807]: from='osd.11 [v2:172.21.15.139:6824/2630023454,v1:172.21.15.139:6825/2630023454]' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2023-07-14T19:28:18.031 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:28:17 smithi098 ceph-mon[127486]: from='osd.11 [v2:172.21.15.139:6824/2630023454,v1:172.21.15.139:6825/2630023454]' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2023-07-14T19:28:18.286 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:18.284+0000 7fe982835700 1 -- 172.21.15.98:0/1407406126 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe97c0ffdb0 msgr2=0x7fe97c100200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:28:18.286 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:18.284+0000 7fe982835700 1 --2- 172.21.15.98:0/1407406126 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe97c0ffdb0 0x7fe97c100200 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7fe970009990 tx=0x7fe970009ca0 comp rx=0 tx=0).stop 2023-07-14T19:28:18.287 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:18.284+0000 7fe982835700 1 -- 172.21.15.98:0/1407406126 shutdown_connections 2023-07-14T19:28:18.287 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:18.284+0000 7fe982835700 1 --2- 172.21.15.98:0/1407406126 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fe97c100740 0x7fe97c107000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:28:18.287 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:18.284+0000 7fe982835700 1 --2- 172.21.15.98:0/1407406126 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7fe97c0febb0 0x7fe97c0fefc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:28:18.287 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:18.284+0000 7fe982835700 1 --2- 172.21.15.98:0/1407406126 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe97c0ffdb0 0x7fe97c100200 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:28:18.287 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:18.284+0000 7fe982835700 1 -- 172.21.15.98:0/1407406126 >> 172.21.15.98:0/1407406126 conn(0x7fe97c0fa1e0 msgr2=0x7fe97c0fc630 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:28:18.287 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:18.284+0000 7fe982835700 1 -- 172.21.15.98:0/1407406126 shutdown_connections 2023-07-14T19:28:18.287 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:18.285+0000 7fe982835700 1 -- 172.21.15.98:0/1407406126 wait complete. 2023-07-14T19:28:18.289 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:18.285+0000 7fe982835700 1 Processor -- start 2023-07-14T19:28:18.289 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:18.285+0000 7fe982835700 1 -- start start 2023-07-14T19:28:18.289 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:18.285+0000 7fe982835700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe97c0febb0 0x7fe97c195920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:28:18.289 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:18.285+0000 7fe982835700 1 --2- >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7fe97c0ffdb0 0x7fe97c195e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:28:18.289 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:18.285+0000 7fe982835700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fe97c100740 0x7fe97c19cee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:28:18.290 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:18.285+0000 7fe982835700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe97c19d4b0 con 0x7fe97c0febb0 2023-07-14T19:28:18.290 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:18.285+0000 7fe982835700 1 -- --> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe97c19d5f0 con 0x7fe97c0ffdb0 2023-07-14T19:28:18.290 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:18.285+0000 7fe982835700 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe97c19d780 con 0x7fe97c100740 2023-07-14T19:28:18.290 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:18.285+0000 7fe97bfff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe97c0febb0 0x7fe97c195920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:28:18.290 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:18.286+0000 7fe97bfff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe97c0febb0 0x7fe97c195920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:45256/0 (socket says 172.21.15.98:45256) 2023-07-14T19:28:18.290 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:18.286+0000 7fe97bfff700 1 -- 172.21.15.98:0/2064620196 learned_addr learned my addr 172.21.15.98:0/2064620196 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-07-14T19:28:18.290 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:18.286+0000 7fe980dd2700 1 --2- 172.21.15.98:0/2064620196 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fe97c100740 0x7fe97c19cee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:28:18.291 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:18.286+0000 7fe97b7fe700 1 --2- 172.21.15.98:0/2064620196 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7fe97c0ffdb0 0x7fe97c195e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:28:18.291 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:18.286+0000 7fe97bfff700 1 -- 172.21.15.98:0/2064620196 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fe97c100740 msgr2=0x7fe97c19cee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:28:18.291 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:18.286+0000 7fe97bfff700 1 --2- 172.21.15.98:0/2064620196 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fe97c100740 0x7fe97c19cee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:28:18.291 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:18.286+0000 7fe97bfff700 1 -- 172.21.15.98:0/2064620196 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7fe97c0ffdb0 msgr2=0x7fe97c195e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:28:18.291 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:18.286+0000 7fe97bfff700 1 --2- 172.21.15.98:0/2064620196 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7fe97c0ffdb0 0x7fe97c195e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:28:18.291 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:18.286+0000 7fe97bfff700 1 -- 172.21.15.98:0/2064620196 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe9700095f0 con 0x7fe97c0febb0 2023-07-14T19:28:18.291 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:18.286+0000 7fe97bfff700 1 --2- 172.21.15.98:0/2064620196 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe97c0febb0 0x7fe97c195920 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7fe96c00c9c0 tx=0x7fe96c00ccd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:28:18.291 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:18.286+0000 7fe9797fa700 1 -- 172.21.15.98:0/2064620196 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe96c007ac0 con 0x7fe97c0febb0 2023-07-14T19:28:18.292 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:18.286+0000 7fe9797fa700 1 -- 172.21.15.98:0/2064620196 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fe96c007c20 con 0x7fe97c0febb0 2023-07-14T19:28:18.292 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:18.286+0000 7fe982835700 1 -- 172.21.15.98:0/2064620196 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe97c19da60 con 0x7fe97c0febb0 2023-07-14T19:28:18.292 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:18.286+0000 7fe9797fa700 1 -- 172.21.15.98:0/2064620196 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe96c018660 con 0x7fe97c0febb0 2023-07-14T19:28:18.292 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:18.286+0000 7fe982835700 1 -- 172.21.15.98:0/2064620196 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe97c19dfb0 con 0x7fe97c0febb0 2023-07-14T19:28:18.292 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:18.287+0000 7fe982835700 1 -- 172.21.15.98:0/2064620196 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe97c10a370 con 0x7fe97c0febb0 2023-07-14T19:28:18.293 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:18.288+0000 7fe9797fa700 1 -- 172.21.15.98:0/2064620196 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 16) v1 ==== 135715+0+0 (secure 0 0 0) 0x7fe96c004020 con 0x7fe97c0febb0 2023-07-14T19:28:18.293 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:18.289+0000 7fe9797fa700 1 --2- 172.21.15.98:0/2064620196 >> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] conn(0x7fe9640a11d0 0x7fe9640a3680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:28:18.293 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:18.289+0000 7fe97b7fe700 1 --2- 172.21.15.98:0/2064620196 >> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] conn(0x7fe9640a11d0 0x7fe9640a3680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:28:18.293 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:18.289+0000 7fe9797fa700 1 -- 172.21.15.98:0/2064620196 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(76..76 src has 1..76) v4 ==== 6831+0+0 (secure 0 0 0) 0x7fe96c0cc310 con 0x7fe97c0febb0 2023-07-14T19:28:18.294 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:18.289+0000 7fe97b7fe700 1 --2- 172.21.15.98:0/2064620196 >> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] conn(0x7fe9640a11d0 0x7fe9640a3680 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7fe970006010 tx=0x7fe97000b260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:28:18.294 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:18.293+0000 7fe9797fa700 1 -- 172.21.15.98:0/2064620196 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7fe96c096140 con 0x7fe97c0febb0 2023-07-14T19:28:18.480 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:18.478+0000 7fe982835700 1 -- 172.21.15.98:0/2064620196 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd stat", "format": "json"} v 0) v1 -- 0x7fe97c19e620 con 0x7fe97c0febb0 2023-07-14T19:28:18.481 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:18.479+0000 7fe9797fa700 1 -- 172.21.15.98:0/2064620196 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd stat", "format": "json"}]=0 v76) v1 ==== 74+0+133 (secure 0 0 0) 0x7fe96c099760 con 0x7fe97c0febb0 2023-07-14T19:28:18.481 INFO:teuthology.orchestra.run.smithi098.stdout: 2023-07-14T19:28:18.482 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:18.481+0000 7fe982835700 1 -- 172.21.15.98:0/2064620196 >> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] conn(0x7fe9640a11d0 msgr2=0x7fe9640a3680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:28:18.483 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:18.481+0000 7fe982835700 1 --2- 172.21.15.98:0/2064620196 >> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] conn(0x7fe9640a11d0 0x7fe9640a3680 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7fe970006010 tx=0x7fe97000b260 comp rx=0 tx=0).stop 2023-07-14T19:28:18.483 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:18.481+0000 7fe982835700 1 -- 172.21.15.98:0/2064620196 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe97c0febb0 msgr2=0x7fe97c195920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:28:18.483 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:18.481+0000 7fe982835700 1 --2- 172.21.15.98:0/2064620196 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe97c0febb0 0x7fe97c195920 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7fe96c00c9c0 tx=0x7fe96c00ccd0 comp rx=0 tx=0).stop 2023-07-14T19:28:18.484 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:18.481+0000 7fe982835700 1 -- 172.21.15.98:0/2064620196 shutdown_connections 2023-07-14T19:28:18.484 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:18.481+0000 7fe982835700 1 --2- 172.21.15.98:0/2064620196 >> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] conn(0x7fe9640a11d0 0x7fe9640a3680 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:28:18.484 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:18.481+0000 7fe982835700 1 --2- 172.21.15.98:0/2064620196 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fe97c100740 0x7fe97c19cee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:28:18.484 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:18.481+0000 7fe982835700 1 --2- 172.21.15.98:0/2064620196 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe97c0febb0 0x7fe97c195920 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:28:18.485 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:18.481+0000 7fe982835700 1 --2- 172.21.15.98:0/2064620196 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7fe97c0ffdb0 0x7fe97c195e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:28:18.485 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:18.481+0000 7fe982835700 1 -- 172.21.15.98:0/2064620196 >> 172.21.15.98:0/2064620196 conn(0x7fe97c0fa1e0 msgr2=0x7fe97c068180 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:28:18.485 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:18.481+0000 7fe982835700 1 -- 172.21.15.98:0/2064620196 shutdown_connections 2023-07-14T19:28:18.485 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:18.481+0000 7fe982835700 1 -- 172.21.15.98:0/2064620196 wait complete. 2023-07-14T19:28:18.781 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:28:18 smithi098 ceph-mon[127486]: pgmap v223: 1 pgs: 1 active+clean; 577 KiB data, 300 MiB used, 983 GiB / 983 GiB avail 2023-07-14T19:28:18.781 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:28:18 smithi098 ceph-mon[127486]: from='osd.11 [v2:172.21.15.139:6824/2630023454,v1:172.21.15.139:6825/2630023454]' entity='osd.11' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]': finished 2023-07-14T19:28:18.781 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:28:18 smithi098 ceph-mon[127486]: osdmap e76: 12 total, 11 up, 12 in 2023-07-14T19:28:18.781 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:28:18 smithi098 ceph-mon[127486]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2023-07-14T19:28:18.782 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:28:18 smithi098 ceph-mon[127486]: from='osd.11 [v2:172.21.15.139:6824/2630023454,v1:172.21.15.139:6825/2630023454]' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi139", "root=default"]}]: dispatch 2023-07-14T19:28:18.782 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:28:18 smithi098 ceph-mon[127486]: from='client.? 172.21.15.98:0/2064620196' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2023-07-14T19:28:18.941 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:28:18 smithi101 ceph-mon[128807]: pgmap v223: 1 pgs: 1 active+clean; 577 KiB data, 300 MiB used, 983 GiB / 983 GiB avail 2023-07-14T19:28:18.941 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:28:18 smithi101 ceph-mon[128807]: from='osd.11 [v2:172.21.15.139:6824/2630023454,v1:172.21.15.139:6825/2630023454]' entity='osd.11' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]': finished 2023-07-14T19:28:18.942 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:28:18 smithi101 ceph-mon[128807]: osdmap e76: 12 total, 11 up, 12 in 2023-07-14T19:28:18.942 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:28:18 smithi101 ceph-mon[128807]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2023-07-14T19:28:18.942 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:28:18 smithi101 ceph-mon[128807]: from='osd.11 [v2:172.21.15.139:6824/2630023454,v1:172.21.15.139:6825/2630023454]' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi139", "root=default"]}]: dispatch 2023-07-14T19:28:18.942 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:28:18 smithi101 ceph-mon[128807]: from='client.? 172.21.15.98:0/2064620196' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2023-07-14T19:28:18.958 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":76,"num_osds":12,"num_up_osds":11,"osd_up_since":1689362877,"num_in_osds":12,"osd_in_since":1689362882,"num_remapped_pgs":0} 2023-07-14T19:28:18.970 INFO:journalctl@ceph.osd.11.smithi139.stdout:Jul 14 19:28:18 smithi139 ceph-9adea648-227b-11ee-9b33-001a4aab830c-osd-11[151655]: 2023-07-14T19:28:18.647+0000 7f50ad9a2700 -1 osd.11 0 waiting for initial osdmap 2023-07-14T19:28:18.970 INFO:journalctl@ceph.osd.11.smithi139.stdout:Jul 14 19:28:18 smithi139 ceph-9adea648-227b-11ee-9b33-001a4aab830c-osd-11[151655]: 2023-07-14T19:28:18.663+0000 7f50a7794700 -1 osd.11 77 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-07-14T19:28:18.971 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:28:18 smithi139 ceph-mon[129293]: pgmap v223: 1 pgs: 1 active+clean; 577 KiB data, 300 MiB used, 983 GiB / 983 GiB avail 2023-07-14T19:28:18.971 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:28:18 smithi139 ceph-mon[129293]: from='osd.11 [v2:172.21.15.139:6824/2630023454,v1:172.21.15.139:6825/2630023454]' entity='osd.11' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]': finished 2023-07-14T19:28:18.971 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:28:18 smithi139 ceph-mon[129293]: osdmap e76: 12 total, 11 up, 12 in 2023-07-14T19:28:18.971 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:28:18 smithi139 ceph-mon[129293]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2023-07-14T19:28:18.971 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:28:18 smithi139 ceph-mon[129293]: from='osd.11 [v2:172.21.15.139:6824/2630023454,v1:172.21.15.139:6825/2630023454]' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi139", "root=default"]}]: dispatch 2023-07-14T19:28:18.971 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:28:18 smithi139 ceph-mon[129293]: from='client.? 172.21.15.98:0/2064620196' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2023-07-14T19:28:19.941 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:28:19 smithi101 ceph-mon[128807]: from='osd.11 [v2:172.21.15.139:6824/2630023454,v1:172.21.15.139:6825/2630023454]' entity='osd.11' cmd='[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi139", "root=default"]}]': finished 2023-07-14T19:28:19.941 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:28:19 smithi101 ceph-mon[128807]: osdmap e77: 12 total, 11 up, 12 in 2023-07-14T19:28:19.942 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:28:19 smithi101 ceph-mon[128807]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2023-07-14T19:28:19.942 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:28:19 smithi101 ceph-mon[128807]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2023-07-14T19:28:19.942 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:28:19 smithi101 ceph-mon[128807]: Detected new or changed devices on smithi139 2023-07-14T19:28:19.942 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:28:19 smithi101 ceph-mon[128807]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' 2023-07-14T19:28:19.942 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:28:19 smithi101 ceph-mon[128807]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' 2023-07-14T19:28:19.942 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:28:19 smithi101 ceph-mon[128807]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2023-07-14T19:28:19.942 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:28:19 smithi101 ceph-mon[128807]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2023-07-14T19:28:19.943 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:28:19 smithi101 ceph-mon[128807]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2023-07-14T19:28:19.943 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:28:19 smithi101 ceph-mon[128807]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2023-07-14T19:28:19.943 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:28:19 smithi101 ceph-mon[128807]: Adjusting osd_memory_target on smithi139 to 4292M 2023-07-14T19:28:19.943 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:28:19 smithi101 ceph-mon[128807]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' 2023-07-14T19:28:19.943 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:28:19 smithi101 ceph-mon[128807]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T19:28:19.943 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:28:19 smithi101 ceph-mon[128807]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-14T19:28:19.943 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:28:19 smithi101 ceph-mon[128807]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' 2023-07-14T19:28:19.944 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:28:19 smithi101 ceph-mon[128807]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-14T19:28:19.944 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:28:19 smithi101 ceph-mon[128807]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T19:28:19.944 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:28:19 smithi101 ceph-mon[128807]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-14T19:28:19.944 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:28:19 smithi101 ceph-mon[128807]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' 2023-07-14T19:28:19.959 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81c9387d2e7576da065747ec9c16d1c7d69016ec shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 9adea648-227b-11ee-9b33-001a4aab830c -- ceph osd stat -f json 2023-07-14T19:28:19.970 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:28:19 smithi139 ceph-mon[129293]: from='osd.11 [v2:172.21.15.139:6824/2630023454,v1:172.21.15.139:6825/2630023454]' entity='osd.11' cmd='[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi139", "root=default"]}]': finished 2023-07-14T19:28:19.971 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:28:19 smithi139 ceph-mon[129293]: osdmap e77: 12 total, 11 up, 12 in 2023-07-14T19:28:19.971 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:28:19 smithi139 ceph-mon[129293]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2023-07-14T19:28:19.971 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:28:19 smithi139 ceph-mon[129293]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2023-07-14T19:28:19.971 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:28:19 smithi139 ceph-mon[129293]: Detected new or changed devices on smithi139 2023-07-14T19:28:19.971 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:28:19 smithi139 ceph-mon[129293]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' 2023-07-14T19:28:19.971 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:28:19 smithi139 ceph-mon[129293]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' 2023-07-14T19:28:19.971 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:28:19 smithi139 ceph-mon[129293]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2023-07-14T19:28:19.972 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:28:19 smithi139 ceph-mon[129293]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2023-07-14T19:28:19.972 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:28:19 smithi139 ceph-mon[129293]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2023-07-14T19:28:19.972 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:28:19 smithi139 ceph-mon[129293]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2023-07-14T19:28:19.972 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:28:19 smithi139 ceph-mon[129293]: Adjusting osd_memory_target on smithi139 to 4292M 2023-07-14T19:28:19.972 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:28:19 smithi139 ceph-mon[129293]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' 2023-07-14T19:28:19.972 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:28:19 smithi139 ceph-mon[129293]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T19:28:19.973 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:28:19 smithi139 ceph-mon[129293]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-14T19:28:19.973 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:28:19 smithi139 ceph-mon[129293]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' 2023-07-14T19:28:19.973 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:28:19 smithi139 ceph-mon[129293]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-14T19:28:19.973 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:28:19 smithi139 ceph-mon[129293]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T19:28:19.973 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:28:19 smithi139 ceph-mon[129293]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-14T19:28:19.973 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:28:19 smithi139 ceph-mon[129293]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' 2023-07-14T19:28:19.990 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:28:19 smithi098 ceph-mon[127486]: from='osd.11 [v2:172.21.15.139:6824/2630023454,v1:172.21.15.139:6825/2630023454]' entity='osd.11' cmd='[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi139", "root=default"]}]': finished 2023-07-14T19:28:19.990 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:28:19 smithi098 ceph-mon[127486]: osdmap e77: 12 total, 11 up, 12 in 2023-07-14T19:28:19.990 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:28:19 smithi098 ceph-mon[127486]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2023-07-14T19:28:19.990 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:28:19 smithi098 ceph-mon[127486]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2023-07-14T19:28:19.991 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:28:19 smithi098 ceph-mon[127486]: Detected new or changed devices on smithi139 2023-07-14T19:28:19.991 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:28:19 smithi098 ceph-mon[127486]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' 2023-07-14T19:28:19.991 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:28:19 smithi098 ceph-mon[127486]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' 2023-07-14T19:28:19.991 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:28:19 smithi098 ceph-mon[127486]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2023-07-14T19:28:19.991 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:28:19 smithi098 ceph-mon[127486]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2023-07-14T19:28:19.991 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:28:19 smithi098 ceph-mon[127486]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2023-07-14T19:28:19.992 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:28:19 smithi098 ceph-mon[127486]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2023-07-14T19:28:19.992 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:28:19 smithi098 ceph-mon[127486]: Adjusting osd_memory_target on smithi139 to 4292M 2023-07-14T19:28:19.992 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:28:19 smithi098 ceph-mon[127486]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' 2023-07-14T19:28:19.992 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:28:19 smithi098 ceph-mon[127486]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T19:28:19.992 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:28:19 smithi098 ceph-mon[127486]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-14T19:28:19.992 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:28:19 smithi098 ceph-mon[127486]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' 2023-07-14T19:28:19.992 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:28:19 smithi098 ceph-mon[127486]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-14T19:28:19.993 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:28:19 smithi098 ceph-mon[127486]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T19:28:19.993 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:28:19 smithi098 ceph-mon[127486]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-14T19:28:19.993 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:28:19 smithi098 ceph-mon[127486]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' 2023-07-14T19:28:20.521 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/9adea648-227b-11ee-9b33-001a4aab830c/mon.a/config 2023-07-14T19:28:20.941 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:28:20 smithi101 ceph-mon[128807]: purged_snaps scrub starts 2023-07-14T19:28:20.942 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:28:20 smithi101 ceph-mon[128807]: purged_snaps scrub ok 2023-07-14T19:28:20.942 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:28:20 smithi101 ceph-mon[128807]: pgmap v226: 1 pgs: 1 active+clean; 577 KiB data, 300 MiB used, 983 GiB / 983 GiB avail 2023-07-14T19:28:20.942 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:28:20 smithi101 ceph-mon[128807]: osd.11 [v2:172.21.15.139:6824/2630023454,v1:172.21.15.139:6825/2630023454] boot 2023-07-14T19:28:20.942 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:28:20 smithi101 ceph-mon[128807]: osdmap e78: 12 total, 12 up, 12 in 2023-07-14T19:28:20.943 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:28:20 smithi101 ceph-mon[128807]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2023-07-14T19:28:20.970 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:28:20 smithi139 ceph-mon[129293]: purged_snaps scrub starts 2023-07-14T19:28:20.971 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:28:20 smithi139 ceph-mon[129293]: purged_snaps scrub ok 2023-07-14T19:28:20.971 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:28:20 smithi139 ceph-mon[129293]: pgmap v226: 1 pgs: 1 active+clean; 577 KiB data, 300 MiB used, 983 GiB / 983 GiB avail 2023-07-14T19:28:20.971 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:28:20 smithi139 ceph-mon[129293]: osd.11 [v2:172.21.15.139:6824/2630023454,v1:172.21.15.139:6825/2630023454] boot 2023-07-14T19:28:20.971 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:28:20 smithi139 ceph-mon[129293]: osdmap e78: 12 total, 12 up, 12 in 2023-07-14T19:28:20.972 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:28:20 smithi139 ceph-mon[129293]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2023-07-14T19:28:20.993 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:28:20 smithi098 ceph-mon[127486]: purged_snaps scrub starts 2023-07-14T19:28:20.993 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:28:20 smithi098 ceph-mon[127486]: purged_snaps scrub ok 2023-07-14T19:28:20.993 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:28:20 smithi098 ceph-mon[127486]: pgmap v226: 1 pgs: 1 active+clean; 577 KiB data, 300 MiB used, 983 GiB / 983 GiB avail 2023-07-14T19:28:20.993 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:28:20 smithi098 ceph-mon[127486]: osd.11 [v2:172.21.15.139:6824/2630023454,v1:172.21.15.139:6825/2630023454] boot 2023-07-14T19:28:20.994 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:28:20 smithi098 ceph-mon[127486]: osdmap e78: 12 total, 12 up, 12 in 2023-07-14T19:28:20.994 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:28:20 smithi098 ceph-mon[127486]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2023-07-14T19:28:21.720 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:21.719+0000 7f3ee20d7700 1 -- 172.21.15.98:0/3722850832 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3edc0febb0 msgr2=0x7f3edc0fefc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:28:21.721 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:21.719+0000 7f3ee20d7700 1 --2- 172.21.15.98:0/3722850832 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3edc0febb0 0x7f3edc0fefc0 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f3ecc009990 tx=0x7f3ecc009ca0 comp rx=0 tx=0).stop 2023-07-14T19:28:21.721 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:21.719+0000 7f3ee20d7700 1 -- 172.21.15.98:0/3722850832 shutdown_connections 2023-07-14T19:28:21.721 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:21.719+0000 7f3ee20d7700 1 --2- 172.21.15.98:0/3722850832 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f3edc100740 0x7f3edc107000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:28:21.721 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:21.719+0000 7f3ee20d7700 1 --2- 172.21.15.98:0/3722850832 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3edc0febb0 0x7f3edc0fefc0 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:28:21.721 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:21.719+0000 7f3ee20d7700 1 --2- 172.21.15.98:0/3722850832 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f3edc0ffdb0 0x7f3edc100200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:28:21.721 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:21.719+0000 7f3ee20d7700 1 -- 172.21.15.98:0/3722850832 >> 172.21.15.98:0/3722850832 conn(0x7f3edc0fa1e0 msgr2=0x7f3edc0fc630 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:28:21.721 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:21.719+0000 7f3ee20d7700 1 -- 172.21.15.98:0/3722850832 shutdown_connections 2023-07-14T19:28:21.722 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:21.719+0000 7f3ee20d7700 1 -- 172.21.15.98:0/3722850832 wait complete. 2023-07-14T19:28:21.723 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:21.719+0000 7f3ee20d7700 1 Processor -- start 2023-07-14T19:28:21.723 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:21.719+0000 7f3ee20d7700 1 -- start start 2023-07-14T19:28:21.724 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:21.720+0000 7f3ee20d7700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3edc0febb0 0x7f3edc195a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:28:21.724 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:21.720+0000 7f3ee20d7700 1 --2- >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f3edc0ffdb0 0x7f3edc195fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:28:21.724 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:21.720+0000 7f3ee20d7700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f3edc100740 0x7f3edc19d020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:28:21.724 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:21.720+0000 7f3ee20d7700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3edc19d5f0 con 0x7f3edc0febb0 2023-07-14T19:28:21.724 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:21.720+0000 7f3ee20d7700 1 -- --> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3edc19d730 con 0x7f3edc0ffdb0 2023-07-14T19:28:21.724 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:21.720+0000 7f3ee20d7700 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3edc19d8c0 con 0x7f3edc100740 2023-07-14T19:28:21.724 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:21.720+0000 7f3edb7fe700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3edc0febb0 0x7f3edc195a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:28:21.725 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:21.720+0000 7f3edb7fe700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3edc0febb0 0x7f3edc195a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:45284/0 (socket says 172.21.15.98:45284) 2023-07-14T19:28:21.725 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:21.720+0000 7f3edb7fe700 1 -- 172.21.15.98:0/2808312400 learned_addr learned my addr 172.21.15.98:0/2808312400 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-07-14T19:28:21.725 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:21.720+0000 7f3edaffd700 1 --2- 172.21.15.98:0/2808312400 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f3edc0ffdb0 0x7f3edc195fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:28:21.725 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:21.720+0000 7f3edbfff700 1 --2- 172.21.15.98:0/2808312400 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f3edc100740 0x7f3edc19d020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:28:21.725 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:21.720+0000 7f3edb7fe700 1 -- 172.21.15.98:0/2808312400 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f3edc100740 msgr2=0x7f3edc19d020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:28:21.725 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:21.720+0000 7f3edb7fe700 1 --2- 172.21.15.98:0/2808312400 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f3edc100740 0x7f3edc19d020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:28:21.725 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:21.720+0000 7f3edb7fe700 1 -- 172.21.15.98:0/2808312400 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f3edc0ffdb0 msgr2=0x7f3edc195fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:28:21.726 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:21.720+0000 7f3edb7fe700 1 --2- 172.21.15.98:0/2808312400 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f3edc0ffdb0 0x7f3edc195fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:28:21.726 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:21.720+0000 7f3edb7fe700 1 -- 172.21.15.98:0/2808312400 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3ecc0095f0 con 0x7f3edc0febb0 2023-07-14T19:28:21.726 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:21.720+0000 7f3edbfff700 1 --2- 172.21.15.98:0/2808312400 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f3edc100740 0x7f3edc19d020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-07-14T19:28:21.726 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:21.720+0000 7f3edb7fe700 1 --2- 172.21.15.98:0/2808312400 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3edc0febb0 0x7f3edc195a60 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f3ecc004730 tx=0x7f3ecc005ef0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:28:21.726 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:21.720+0000 7f3ed8ff9700 1 -- 172.21.15.98:0/2808312400 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ecc01d070 con 0x7f3edc0febb0 2023-07-14T19:28:21.726 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:21.720+0000 7f3ed8ff9700 1 -- 172.21.15.98:0/2808312400 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f3ecc004cd0 con 0x7f3edc0febb0 2023-07-14T19:28:21.726 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:21.720+0000 7f3ee20d7700 1 -- 172.21.15.98:0/2808312400 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3edc107c00 con 0x7f3edc0febb0 2023-07-14T19:28:21.726 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:21.720+0000 7f3ed8ff9700 1 -- 172.21.15.98:0/2808312400 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ecc005590 con 0x7f3edc0febb0 2023-07-14T19:28:21.727 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:21.720+0000 7f3ee20d7700 1 -- 172.21.15.98:0/2808312400 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3edc108040 con 0x7f3edc0febb0 2023-07-14T19:28:21.727 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:21.721+0000 7f3ee20d7700 1 -- 172.21.15.98:0/2808312400 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3edc106250 con 0x7f3edc0febb0 2023-07-14T19:28:21.727 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:21.723+0000 7f3ed8ff9700 1 -- 172.21.15.98:0/2808312400 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 16) v1 ==== 135715+0+0 (secure 0 0 0) 0x7f3ecc00fbb0 con 0x7f3edc0febb0 2023-07-14T19:28:21.728 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:21.723+0000 7f3ed8ff9700 1 --2- 172.21.15.98:0/2808312400 >> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] conn(0x7f3ec40a1150 0x7f3ec40a3600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:28:21.728 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:21.723+0000 7f3edaffd700 1 --2- 172.21.15.98:0/2808312400 >> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] conn(0x7f3ec40a1150 0x7f3ec40a3600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:28:21.728 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:21.723+0000 7f3ed8ff9700 1 -- 172.21.15.98:0/2808312400 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(79..79 src has 1..79) v4 ==== 7127+0+0 (secure 0 0 0) 0x7f3ecc0ce590 con 0x7f3edc0febb0 2023-07-14T19:28:21.728 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:21.724+0000 7f3edaffd700 1 --2- 172.21.15.98:0/2808312400 >> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] conn(0x7f3ec40a1150 0x7f3ec40a3600 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f3ed0009bf0 tx=0x7f3ed0009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:28:21.728 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:21.726+0000 7f3ed8ff9700 1 -- 172.21.15.98:0/2808312400 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f3ecc098170 con 0x7f3edc0febb0 2023-07-14T19:28:21.913 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:21.912+0000 7f3ee20d7700 1 -- 172.21.15.98:0/2808312400 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd stat", "format": "json"} v 0) v1 -- 0x7f3edc02c8e0 con 0x7f3edc0febb0 2023-07-14T19:28:21.914 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:21.912+0000 7f3ed8ff9700 1 -- 172.21.15.98:0/2808312400 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd stat", "format": "json"}]=0 v79) v1 ==== 74+0+133 (secure 0 0 0) 0x7f3ecc09b790 con 0x7f3edc0febb0 2023-07-14T19:28:21.914 INFO:teuthology.orchestra.run.smithi098.stdout: 2023-07-14T19:28:21.916 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:21.915+0000 7f3ee20d7700 1 -- 172.21.15.98:0/2808312400 >> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] conn(0x7f3ec40a1150 msgr2=0x7f3ec40a3600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:28:21.916 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:21.915+0000 7f3ee20d7700 1 --2- 172.21.15.98:0/2808312400 >> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] conn(0x7f3ec40a1150 0x7f3ec40a3600 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f3ed0009bf0 tx=0x7f3ed0009260 comp rx=0 tx=0).stop 2023-07-14T19:28:21.917 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:21.915+0000 7f3ee20d7700 1 -- 172.21.15.98:0/2808312400 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3edc0febb0 msgr2=0x7f3edc195a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:28:21.917 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:21.915+0000 7f3ee20d7700 1 --2- 172.21.15.98:0/2808312400 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3edc0febb0 0x7f3edc195a60 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f3ecc004730 tx=0x7f3ecc005ef0 comp rx=0 tx=0).stop 2023-07-14T19:28:21.917 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:21.915+0000 7f3ee20d7700 1 -- 172.21.15.98:0/2808312400 shutdown_connections 2023-07-14T19:28:21.917 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:21.915+0000 7f3ee20d7700 1 --2- 172.21.15.98:0/2808312400 >> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] conn(0x7f3ec40a1150 0x7f3ec40a3600 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:28:21.918 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:21.915+0000 7f3ee20d7700 1 --2- 172.21.15.98:0/2808312400 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f3edc100740 0x7f3edc19d020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:28:21.918 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:21.915+0000 7f3ee20d7700 1 --2- 172.21.15.98:0/2808312400 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3edc0febb0 0x7f3edc195a60 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:28:21.918 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:21.915+0000 7f3ee20d7700 1 --2- 172.21.15.98:0/2808312400 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f3edc0ffdb0 0x7f3edc195fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:28:21.919 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:21.915+0000 7f3ee20d7700 1 -- 172.21.15.98:0/2808312400 >> 172.21.15.98:0/2808312400 conn(0x7f3edc0fa1e0 msgr2=0x7f3edc103390 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:28:21.919 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:21.915+0000 7f3ee20d7700 1 -- 172.21.15.98:0/2808312400 shutdown_connections 2023-07-14T19:28:21.919 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:21.915+0000 7f3ee20d7700 1 -- 172.21.15.98:0/2808312400 wait complete. 2023-07-14T19:28:22.191 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:28:21 smithi101 ceph-mon[128807]: osdmap e79: 12 total, 12 up, 12 in 2023-07-14T19:28:22.191 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:28:21 smithi101 ceph-mon[128807]: from='client.? 172.21.15.98:0/2808312400' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2023-07-14T19:28:22.220 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:28:21 smithi139 ceph-mon[129293]: osdmap e79: 12 total, 12 up, 12 in 2023-07-14T19:28:22.220 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:28:21 smithi139 ceph-mon[129293]: from='client.? 172.21.15.98:0/2808312400' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2023-07-14T19:28:22.271 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:28:21 smithi098 ceph-mon[127486]: osdmap e79: 12 total, 12 up, 12 in 2023-07-14T19:28:22.271 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:28:21 smithi098 ceph-mon[127486]: from='client.? 172.21.15.98:0/2808312400' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2023-07-14T19:28:22.275 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":79,"num_osds":12,"num_up_osds":12,"osd_up_since":1689362899,"num_in_osds":12,"osd_in_since":1689362882,"num_remapped_pgs":0} 2023-07-14T19:28:22.275 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81c9387d2e7576da065747ec9c16d1c7d69016ec shell --fsid 9adea648-227b-11ee-9b33-001a4aab830c -- ceph osd dump --format=json 2023-07-14T19:28:22.868 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/9adea648-227b-11ee-9b33-001a4aab830c/mon.a/config 2023-07-14T19:28:23.191 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:28:22 smithi101 ceph-mon[128807]: pgmap v229: 1 pgs: 1 active+clean; 577 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-14T19:28:23.220 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:28:22 smithi139 ceph-mon[129293]: pgmap v229: 1 pgs: 1 active+clean; 577 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-14T19:28:23.255 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:28:22 smithi098 ceph-mon[127486]: pgmap v229: 1 pgs: 1 active+clean; 577 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-14T19:28:24.172 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:24.171+0000 7f178222f700 1 -- 172.21.15.98:0/3510518681 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f177c0fef40 msgr2=0x7f177c0ff310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:28:24.172 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:24.171+0000 7f178222f700 1 --2- 172.21.15.98:0/3510518681 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f177c0fef40 0x7f177c0ff310 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f17700099c0 tx=0x7f1770009cd0 comp rx=0 tx=0).stop 2023-07-14T19:28:24.172 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:24.171+0000 7f178222f700 1 -- 172.21.15.98:0/3510518681 shutdown_connections 2023-07-14T19:28:24.173 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:24.171+0000 7f178222f700 1 --2- 172.21.15.98:0/3510518681 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f177c101150 0x7f177c10bb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:28:24.173 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:24.171+0000 7f178222f700 1 --2- 172.21.15.98:0/3510518681 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f177c0fef40 0x7f177c0ff310 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:28:24.173 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:24.171+0000 7f178222f700 1 --2- 172.21.15.98:0/3510518681 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f177c1007a0 0x7f177c100c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:28:24.173 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:24.171+0000 7f178222f700 1 -- 172.21.15.98:0/3510518681 >> 172.21.15.98:0/3510518681 conn(0x7f177c075c90 msgr2=0x7f177c076090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:28:24.173 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:24.171+0000 7f178222f700 1 -- 172.21.15.98:0/3510518681 shutdown_connections 2023-07-14T19:28:24.173 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:24.171+0000 7f178222f700 1 -- 172.21.15.98:0/3510518681 wait complete. 2023-07-14T19:28:24.175 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:24.172+0000 7f178222f700 1 Processor -- start 2023-07-14T19:28:24.175 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:24.172+0000 7f178222f700 1 -- start start 2023-07-14T19:28:24.175 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:24.172+0000 7f178222f700 1 --2- >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f177c0fef40 0x7f177c101980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:28:24.175 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:24.172+0000 7f178222f700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f177c1007a0 0x7f177c101ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:28:24.175 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:24.172+0000 7f178222f700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f177c101150 0x7f177c102440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:28:24.176 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:24.172+0000 7f178222f700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f177c1a41c0 con 0x7f177c101150 2023-07-14T19:28:24.176 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:24.172+0000 7f178222f700 1 -- --> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f177c1a4330 con 0x7f177c0fef40 2023-07-14T19:28:24.176 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:24.172+0000 7f178222f700 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f177c1a44f0 con 0x7f177c1007a0 2023-07-14T19:28:24.176 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:24.173+0000 7f177bfff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f177c101150 0x7f177c102440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:28:24.176 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:24.173+0000 7f177bfff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f177c101150 0x7f177c102440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:45302/0 (socket says 172.21.15.98:45302) 2023-07-14T19:28:24.176 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:24.173+0000 7f177bfff700 1 -- 172.21.15.98:0/2160941470 learned_addr learned my addr 172.21.15.98:0/2160941470 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-07-14T19:28:24.176 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:24.173+0000 7f177affd700 1 --2- 172.21.15.98:0/2160941470 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f177c1007a0 0x7f177c101ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:28:24.177 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:24.173+0000 7f177b7fe700 1 --2- 172.21.15.98:0/2160941470 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f177c0fef40 0x7f177c101980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:28:24.177 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:24.173+0000 7f177bfff700 1 -- 172.21.15.98:0/2160941470 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f177c1007a0 msgr2=0x7f177c101ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:28:24.177 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:24.173+0000 7f177bfff700 1 --2- 172.21.15.98:0/2160941470 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f177c1007a0 0x7f177c101ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:28:24.177 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:24.173+0000 7f177bfff700 1 -- 172.21.15.98:0/2160941470 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f177c0fef40 msgr2=0x7f177c101980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:28:24.177 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:24.173+0000 7f177bfff700 1 --2- 172.21.15.98:0/2160941470 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f177c0fef40 0x7f177c101980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:28:24.177 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:24.173+0000 7f177bfff700 1 -- 172.21.15.98:0/2160941470 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f17700095f0 con 0x7f177c101150 2023-07-14T19:28:24.177 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:24.173+0000 7f177affd700 1 --2- 172.21.15.98:0/2160941470 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f177c1007a0 0x7f177c101ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-07-14T19:28:24.178 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:24.173+0000 7f177bfff700 1 --2- 172.21.15.98:0/2160941470 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f177c101150 0x7f177c102440 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f1774007ee0 tx=0x7f177400d3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:28:24.178 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:24.173+0000 7f1778ff9700 1 -- 172.21.15.98:0/2160941470 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1774017070 con 0x7f177c101150 2023-07-14T19:28:24.178 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:24.173+0000 7f178222f700 1 -- 172.21.15.98:0/2160941470 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f177c1a47d0 con 0x7f177c101150 2023-07-14T19:28:24.178 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:24.173+0000 7f1778ff9700 1 -- 172.21.15.98:0/2160941470 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f177400f040 con 0x7f177c101150 2023-07-14T19:28:24.178 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:24.173+0000 7f1778ff9700 1 -- 172.21.15.98:0/2160941470 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1774005740 con 0x7f177c101150 2023-07-14T19:28:24.178 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:24.173+0000 7f178222f700 1 -- 172.21.15.98:0/2160941470 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f177c1a4c80 con 0x7f177c101150 2023-07-14T19:28:24.178 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:24.174+0000 7f178222f700 1 -- 172.21.15.98:0/2160941470 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f177c04e340 con 0x7f177c101150 2023-07-14T19:28:24.179 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:24.175+0000 7f1778ff9700 1 -- 172.21.15.98:0/2160941470 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 16) v1 ==== 135715+0+0 (secure 0 0 0) 0x7f17740058a0 con 0x7f177c101150 2023-07-14T19:28:24.179 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:24.176+0000 7f1778ff9700 1 --2- 172.21.15.98:0/2160941470 >> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] conn(0x7f17640a1150 0x7f17640a3600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:28:24.179 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:24.176+0000 7f1778ff9700 1 -- 172.21.15.98:0/2160941470 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(79..79 src has 1..79) v4 ==== 7127+0+0 (secure 0 0 0) 0x7f17740cbd30 con 0x7f177c101150 2023-07-14T19:28:24.180 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:24.176+0000 7f177b7fe700 1 --2- 172.21.15.98:0/2160941470 >> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] conn(0x7f17640a1150 0x7f17640a3600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:28:24.180 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:24.176+0000 7f177b7fe700 1 --2- 172.21.15.98:0/2160941470 >> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] conn(0x7f17640a1150 0x7f17640a3600 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f1770000c00 tx=0x7f1770005fb0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:28:24.180 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:24.180+0000 7f1778ff9700 1 -- 172.21.15.98:0/2160941470 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f1774095920 con 0x7f177c101150 2023-07-14T19:28:24.364 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:24.364+0000 7f178222f700 1 -- 172.21.15.98:0/2160941470 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f177c1a4ed0 con 0x7f177c101150 2023-07-14T19:28:24.365 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:24.365+0000 7f1778ff9700 1 -- 172.21.15.98:0/2160941470 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v79) v1 ==== 74+0+18420 (secure 0 0 0) 0x7f1774098f40 con 0x7f177c101150 2023-07-14T19:28:24.366 INFO:teuthology.orchestra.run.smithi098.stdout: 2023-07-14T19:28:24.366 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":79,"fsid":"9adea648-227b-11ee-9b33-001a4aab830c","created":"2023-07-14T19:22:07.178854+0000","modified":"2023-07-14T19:28:20.933203+0000","last_up_change":"2023-07-14T19:28:19.646827+0000","last_in_change":"2023-07-14T19:28:02.018325+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-07-14T19:25:09.845827+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"a3797e89-1754-41cf-9517-227e06ffdd72","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6802","nonce":120507228},{"type":"v1","addr":"172.21.15.98:6803","nonce":120507228}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6804","nonce":120507228},{"type":"v1","addr":"172.21.15.98:6805","nonce":120507228}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6808","nonce":120507228},{"type":"v1","addr":"172.21.15.98:6809","nonce":120507228}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6806","nonce":120507228},{"type":"v1","addr":"172.21.15.98:6807","nonce":120507228}]},"public_addr":"172.21.15.98:6803/120507228","cluster_addr":"172.21.15.98:6805/120507228","heartbeat_back_addr":"172.21.15.98:6809/120507228","heartbeat_front_addr":"172.21.15.98:6807/120507228","state":["exists","up"]},{"osd":1,"uuid":"2b001add-1aaa-4496-a403-7ef9725b80a0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.101:6800","nonce":2450584584},{"type":"v1","addr":"172.21.15.101:6801","nonce":2450584584}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.101:6802","nonce":2450584584},{"type":"v1","addr":"172.21.15.101:6803","nonce":2450584584}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.101:6806","nonce":2450584584},{"type":"v1","addr":"172.21.15.101:6807","nonce":2450584584}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.101:6804","nonce":2450584584},{"type":"v1","addr":"172.21.15.101:6805","nonce":2450584584}]},"public_addr":"172.21.15.101:6801/2450584584","cluster_addr":"172.21.15.101:6803/2450584584","heartbeat_back_addr":"172.21.15.101:6807/2450584584","heartbeat_front_addr":"172.21.15.101:6805/2450584584","state":["exists","up"]},{"osd":2,"uuid":"48e6d499-3ca9-424b-b0fb-3ba5f33b9fc9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":945038995},{"type":"v1","addr":"172.21.15.139:6801","nonce":945038995}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":945038995},{"type":"v1","addr":"172.21.15.139:6803","nonce":945038995}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":945038995},{"type":"v1","addr":"172.21.15.139:6807","nonce":945038995}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":945038995},{"type":"v1","addr":"172.21.15.139:6805","nonce":945038995}]},"public_addr":"172.21.15.139:6801/945038995","cluster_addr":"172.21.15.139:6803/945038995","heartbeat_back_addr":"172.21.15.139:6807/945038995","heartbeat_front_addr":"172.21.15.139:6805/945038995","state":["exists","up"]},{"osd":3,"uuid":"5530b265-31a5-447b-a1ea-17ebae9cfddf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6810","nonce":3095249665},{"type":"v1","addr":"172.21.15.98:6811","nonce":3095249665}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6812","nonce":3095249665},{"type":"v1","addr":"172.21.15.98:6813","nonce":3095249665}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6816","nonce":3095249665},{"type":"v1","addr":"172.21.15.98:6817","nonce":3095249665}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6814","nonce":3095249665},{"type":"v1","addr":"172.21.15.98:6815","nonce":3095249665}]},"public_addr":"172.21.15.98:6811/3095249665","cluster_addr":"172.21.15.98:6813/3095249665","heartbeat_back_addr":"172.21.15.98:6817/3095249665","heartbeat_front_addr":"172.21.15.98:6815/3095249665","state":["exists","up"]},{"osd":4,"uuid":"615d4642-a997-4f95-9e38-9fbdaa831400","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":32,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.101:6808","nonce":2844761406},{"type":"v1","addr":"172.21.15.101:6809","nonce":2844761406}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.101:6810","nonce":2844761406},{"type":"v1","addr":"172.21.15.101:6811","nonce":2844761406}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.101:6814","nonce":2844761406},{"type":"v1","addr":"172.21.15.101:6815","nonce":2844761406}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.101:6812","nonce":2844761406},{"type":"v1","addr":"172.21.15.101:6813","nonce":2844761406}]},"public_addr":"172.21.15.101:6809/2844761406","cluster_addr":"172.21.15.101:6811/2844761406","heartbeat_back_addr":"172.21.15.101:6815/2844761406","heartbeat_front_addr":"172.21.15.101:6813/2844761406","state":["exists","up"]},{"osd":5,"uuid":"415de5fa-607a-45a8-b732-28daf3be6dd3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":37,"up_thru":52,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":3532263042},{"type":"v1","addr":"172.21.15.139:6809","nonce":3532263042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6810","nonce":3532263042},{"type":"v1","addr":"172.21.15.139:6811","nonce":3532263042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6814","nonce":3532263042},{"type":"v1","addr":"172.21.15.139:6815","nonce":3532263042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":3532263042},{"type":"v1","addr":"172.21.15.139:6813","nonce":3532263042}]},"public_addr":"172.21.15.139:6809/3532263042","cluster_addr":"172.21.15.139:6811/3532263042","heartbeat_back_addr":"172.21.15.139:6815/3532263042","heartbeat_front_addr":"172.21.15.139:6813/3532263042","state":["exists","up"]},{"osd":6,"uuid":"d7f9d4ba-c309-4b75-a836-5a36384b528e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":45,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6818","nonce":658948743},{"type":"v1","addr":"172.21.15.98:6819","nonce":658948743}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6820","nonce":658948743},{"type":"v1","addr":"172.21.15.98:6821","nonce":658948743}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6824","nonce":658948743},{"type":"v1","addr":"172.21.15.98:6825","nonce":658948743}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6822","nonce":658948743},{"type":"v1","addr":"172.21.15.98:6823","nonce":658948743}]},"public_addr":"172.21.15.98:6819/658948743","cluster_addr":"172.21.15.98:6821/658948743","heartbeat_back_addr":"172.21.15.98:6825/658948743","heartbeat_front_addr":"172.21.15.98:6823/658948743","state":["exists","up"]},{"osd":7,"uuid":"eedb94d9-c5cd-4a93-8a8c-2648804b7195","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":50,"up_thru":73,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.101:6816","nonce":2881214066},{"type":"v1","addr":"172.21.15.101:6817","nonce":2881214066}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.101:6818","nonce":2881214066},{"type":"v1","addr":"172.21.15.101:6819","nonce":2881214066}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.101:6822","nonce":2881214066},{"type":"v1","addr":"172.21.15.101:6823","nonce":2881214066}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.101:6820","nonce":2881214066},{"type":"v1","addr":"172.21.15.101:6821","nonce":2881214066}]},"public_addr":"172.21.15.101:6817/2881214066","cluster_addr":"172.21.15.101:6819/2881214066","heartbeat_back_addr":"172.21.15.101:6823/2881214066","heartbeat_front_addr":"172.21.15.101:6821/2881214066","state":["exists","up"]},{"osd":8,"uuid":"6bfd77c2-5770-466e-aa6f-84b50c74ffb6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":57,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":3669514524},{"type":"v1","addr":"172.21.15.139:6817","nonce":3669514524}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":3669514524},{"type":"v1","addr":"172.21.15.139:6819","nonce":3669514524}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":3669514524},{"type":"v1","addr":"172.21.15.139:6823","nonce":3669514524}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6820","nonce":3669514524},{"type":"v1","addr":"172.21.15.139:6821","nonce":3669514524}]},"public_addr":"172.21.15.139:6817/3669514524","cluster_addr":"172.21.15.139:6819/3669514524","heartbeat_back_addr":"172.21.15.139:6823/3669514524","heartbeat_front_addr":"172.21.15.139:6821/3669514524","state":["exists","up"]},{"osd":9,"uuid":"19c5b54a-2242-4b23-b721-63be9559653d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":65,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6826","nonce":2893994080},{"type":"v1","addr":"172.21.15.98:6827","nonce":2893994080}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6828","nonce":2893994080},{"type":"v1","addr":"172.21.15.98:6829","nonce":2893994080}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6832","nonce":2893994080},{"type":"v1","addr":"172.21.15.98:6833","nonce":2893994080}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6830","nonce":2893994080},{"type":"v1","addr":"172.21.15.98:6831","nonce":2893994080}]},"public_addr":"172.21.15.98:6827/2893994080","cluster_addr":"172.21.15.98:6829/2893994080","heartbeat_back_addr":"172.21.15.98:6833/2893994080","heartbeat_front_addr":"172.21.15.98:6831/2893994080","state":["exists","up"]},{"osd":10,"uuid":"326f0209-ba9d-4b59-97cc-d61a758a8f29","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":70,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.101:6824","nonce":3545244225},{"type":"v1","addr":"172.21.15.101:6825","nonce":3545244225}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.101:6826","nonce":3545244225},{"type":"v1","addr":"172.21.15.101:6827","nonce":3545244225}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.101:6830","nonce":3545244225},{"type":"v1","addr":"172.21.15.101:6831","nonce":3545244225}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.101:6828","nonce":3545244225},{"type":"v1","addr":"172.21.15.101:6829","nonce":3545244225}]},"public_addr":"172.21.15.101:6825/3545244225","cluster_addr":"172.21.15.101:6827/3545244225","heartbeat_back_addr":"172.21.15.101:6831/3545244225","heartbeat_front_addr":"172.21.15.101:6829/3545244225","state":["exists","up"]},{"osd":11,"uuid":"c61c0320-1057-4f17-9bf1-7ad306ed7ed3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":78,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":2630023454},{"type":"v1","addr":"172.21.15.139:6825","nonce":2630023454}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6826","nonce":2630023454},{"type":"v1","addr":"172.21.15.139:6827","nonce":2630023454}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":2630023454},{"type":"v1","addr":"172.21.15.139:6831","nonce":2630023454}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":2630023454},{"type":"v1","addr":"172.21.15.139:6829","nonce":2630023454}]},"public_addr":"172.21.15.139:6825/2630023454","cluster_addr":"172.21.15.139:6827/2630023454","heartbeat_back_addr":"172.21.15.139:6831/2630023454","heartbeat_front_addr":"172.21.15.139:6829/2630023454","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T19:24:23.620028+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T19:24:45.779846+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T19:25:06.877949+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T19:25:27.994199+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T19:25:49.461139+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T19:26:10.409282+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T19:26:32.116390+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T19:26:53.536474+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T19:27:14.384839+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T19:27:35.635921+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T19:27:56.858744+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T19:28:18.342519+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.98:0/2164590253":"2023-07-15T19:23:01.520302+0000","172.21.15.98:0/2094734896":"2023-07-15T19:23:01.520302+0000","172.21.15.98:0/784745009":"2023-07-15T19:22:39.481502+0000","172.21.15.98:0/3657190818":"2023-07-15T19:22:39.481502+0000","172.21.15.98:6800/1732769087":"2023-07-15T19:23:01.520302+0000","172.21.15.98:0/3393542419":"2023-07-15T19:22:39.481502+0000","172.21.15.98:6800/749234665":"2023-07-15T19:22:39.481502+0000","172.21.15.98:6801/1732769087":"2023-07-15T19:23:01.520302+0000","172.21.15.98:6801/749234665":"2023-07-15T19:22:39.481502+0000","172.21.15.98:0/187822873":"2023-07-15T19:23:01.520302+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-07-14T19:28:24.367 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:24.367+0000 7f178222f700 1 -- 172.21.15.98:0/2160941470 >> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] conn(0x7f17640a1150 msgr2=0x7f17640a3600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:28:24.368 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:24.367+0000 7f178222f700 1 --2- 172.21.15.98:0/2160941470 >> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] conn(0x7f17640a1150 0x7f17640a3600 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f1770000c00 tx=0x7f1770005fb0 comp rx=0 tx=0).stop 2023-07-14T19:28:24.368 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:24.367+0000 7f178222f700 1 -- 172.21.15.98:0/2160941470 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f177c101150 msgr2=0x7f177c102440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:28:24.368 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:24.367+0000 7f178222f700 1 --2- 172.21.15.98:0/2160941470 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f177c101150 0x7f177c102440 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f1774007ee0 tx=0x7f177400d3b0 comp rx=0 tx=0).stop 2023-07-14T19:28:24.368 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:24.367+0000 7f178222f700 1 -- 172.21.15.98:0/2160941470 shutdown_connections 2023-07-14T19:28:24.368 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:24.367+0000 7f178222f700 1 --2- 172.21.15.98:0/2160941470 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f177c101150 0x7f177c102440 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:28:24.368 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:24.367+0000 7f178222f700 1 --2- 172.21.15.98:0/2160941470 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f177c0fef40 0x7f177c101980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:28:24.368 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:24.367+0000 7f178222f700 1 --2- 172.21.15.98:0/2160941470 >> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] conn(0x7f17640a1150 0x7f17640a3600 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:28:24.369 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:24.367+0000 7f178222f700 1 --2- 172.21.15.98:0/2160941470 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f177c1007a0 0x7f177c101ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:28:24.369 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:24.367+0000 7f178222f700 1 -- 172.21.15.98:0/2160941470 >> 172.21.15.98:0/2160941470 conn(0x7f177c075c90 msgr2=0x7f177c069260 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:28:24.369 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:24.367+0000 7f178222f700 1 -- 172.21.15.98:0/2160941470 shutdown_connections 2023-07-14T19:28:24.369 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:24.367+0000 7f178222f700 1 -- 172.21.15.98:0/2160941470 wait complete. 2023-07-14T19:28:24.823 INFO:tasks.cephadm.ceph_manager.ceph:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-07-14T19:25:09.845827+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-07-14T19:28:24.824 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81c9387d2e7576da065747ec9c16d1c7d69016ec shell --fsid 9adea648-227b-11ee-9b33-001a4aab830c -- ceph osd pool get .mgr pg_num 2023-07-14T19:28:25.031 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:28:24 smithi098 ceph-mon[127486]: pgmap v230: 1 pgs: 1 active+clean; 577 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-14T19:28:25.032 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:28:24 smithi098 ceph-mon[127486]: from='client.? 172.21.15.98:0/2160941470' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-07-14T19:28:25.191 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:28:24 smithi101 ceph-mon[128807]: pgmap v230: 1 pgs: 1 active+clean; 577 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-14T19:28:25.192 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:28:24 smithi101 ceph-mon[128807]: from='client.? 172.21.15.98:0/2160941470' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-07-14T19:28:25.220 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:28:24 smithi139 ceph-mon[129293]: pgmap v230: 1 pgs: 1 active+clean; 577 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-14T19:28:25.220 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:28:24 smithi139 ceph-mon[129293]: from='client.? 172.21.15.98:0/2160941470' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-07-14T19:28:25.375 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/9adea648-227b-11ee-9b33-001a4aab830c/mon.a/config 2023-07-14T19:28:26.589 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:26.588+0000 7f96ed1af700 1 -- 172.21.15.98:0/1801867550 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f96e80ff810 msgr2=0x7f96e81037e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:28:26.589 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:26.588+0000 7f96ed1af700 1 --2- 172.21.15.98:0/1801867550 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f96e80ff810 0x7f96e81037e0 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f96d80099c0 tx=0x7f96d8009cd0 comp rx=0 tx=0).stop 2023-07-14T19:28:26.589 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:26.588+0000 7f96ed1af700 1 -- 172.21.15.98:0/1801867550 shutdown_connections 2023-07-14T19:28:26.590 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:26.588+0000 7f96ed1af700 1 --2- 172.21.15.98:0/1801867550 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f96e8103f10 0x7f96e810ba90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:28:26.590 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:26.588+0000 7f96ed1af700 1 --2- 172.21.15.98:0/1801867550 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f96e80fee70 0x7f96e80ff240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:28:26.590 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:26.588+0000 7f96ed1af700 1 --2- 172.21.15.98:0/1801867550 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f96e80ff810 0x7f96e81037e0 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:28:26.590 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:26.588+0000 7f96ed1af700 1 -- 172.21.15.98:0/1801867550 >> 172.21.15.98:0/1801867550 conn(0x7f96e8075ae0 msgr2=0x7f96e8075ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:28:26.590 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:26.588+0000 7f96ed1af700 1 -- 172.21.15.98:0/1801867550 shutdown_connections 2023-07-14T19:28:26.590 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:26.588+0000 7f96ed1af700 1 -- 172.21.15.98:0/1801867550 wait complete. 2023-07-14T19:28:26.592 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:26.589+0000 7f96ed1af700 1 Processor -- start 2023-07-14T19:28:26.593 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:26.589+0000 7f96ed1af700 1 -- start start 2023-07-14T19:28:26.593 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:26.589+0000 7f96ed1af700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f96e80fee70 0x7f96e8196ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:28:26.593 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:26.589+0000 7f96ed1af700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f96e80ff810 0x7f96e81970e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:28:26.593 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:26.589+0000 7f96ed1af700 1 --2- >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f96e8103f10 0x7f96e81a1c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:28:26.593 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:26.589+0000 7f96ed1af700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f96e81a2160 con 0x7f96e80fee70 2023-07-14T19:28:26.593 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:26.589+0000 7f96ed1af700 1 -- --> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f96e81a22a0 con 0x7f96e8103f10 2023-07-14T19:28:26.594 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:26.589+0000 7f96ed1af700 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f96e81a23e0 con 0x7f96e80ff810 2023-07-14T19:28:26.594 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:26.590+0000 7f96e6d9d700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f96e80fee70 0x7f96e8196ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:28:26.594 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:26.590+0000 7f96e6d9d700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f96e80fee70 0x7f96e8196ba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:45312/0 (socket says 172.21.15.98:45312) 2023-07-14T19:28:26.594 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:26.590+0000 7f96e6d9d700 1 -- 172.21.15.98:0/245826581 learned_addr learned my addr 172.21.15.98:0/245826581 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-07-14T19:28:26.594 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:26.590+0000 7f96e659c700 1 --2- 172.21.15.98:0/245826581 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f96e80ff810 0x7f96e81970e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:28:26.594 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:26.590+0000 7f96e759e700 1 --2- 172.21.15.98:0/245826581 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f96e8103f10 0x7f96e81a1c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:28:26.595 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:26.590+0000 7f96e6d9d700 1 -- 172.21.15.98:0/245826581 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f96e80ff810 msgr2=0x7f96e81970e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:28:26.595 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:26.590+0000 7f96e6d9d700 1 --2- 172.21.15.98:0/245826581 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f96e80ff810 0x7f96e81970e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:28:26.595 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:26.590+0000 7f96e6d9d700 1 -- 172.21.15.98:0/245826581 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f96e8103f10 msgr2=0x7f96e81a1c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:28:26.595 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:26.590+0000 7f96e6d9d700 1 --2- 172.21.15.98:0/245826581 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f96e8103f10 0x7f96e81a1c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:28:26.595 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:26.590+0000 7f96e6d9d700 1 -- 172.21.15.98:0/245826581 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f96d80095f0 con 0x7f96e80fee70 2023-07-14T19:28:26.595 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:26.590+0000 7f96e759e700 1 --2- 172.21.15.98:0/245826581 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f96e8103f10 0x7f96e81a1c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-07-14T19:28:26.595 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:26.590+0000 7f96e6d9d700 1 --2- 172.21.15.98:0/245826581 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f96e80fee70 0x7f96e8196ba0 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f96d000e9a0 tx=0x7f96d000ecb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:28:26.596 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:26.590+0000 7f96dffff700 1 -- 172.21.15.98:0/245826581 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f96d0009790 con 0x7f96e80fee70 2023-07-14T19:28:26.596 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:26.590+0000 7f96dffff700 1 -- 172.21.15.98:0/245826581 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f96d000cbd0 con 0x7f96e80fee70 2023-07-14T19:28:26.596 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:26.590+0000 7f96ed1af700 1 -- 172.21.15.98:0/245826581 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f96e81a2640 con 0x7f96e80fee70 2023-07-14T19:28:26.596 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:26.590+0000 7f96dffff700 1 -- 172.21.15.98:0/245826581 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f96d0018680 con 0x7f96e80fee70 2023-07-14T19:28:26.596 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:26.590+0000 7f96ed1af700 1 -- 172.21.15.98:0/245826581 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f96e81a2b10 con 0x7f96e80fee70 2023-07-14T19:28:26.596 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:26.591+0000 7f96ed1af700 1 -- 172.21.15.98:0/245826581 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f96e804e340 con 0x7f96e80fee70 2023-07-14T19:28:26.597 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:26.592+0000 7f96dffff700 1 -- 172.21.15.98:0/245826581 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 16) v1 ==== 135715+0+0 (secure 0 0 0) 0x7f96d000cd40 con 0x7f96e80fee70 2023-07-14T19:28:26.597 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:26.593+0000 7f96dffff700 1 --2- 172.21.15.98:0/245826581 >> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] conn(0x7f96d40a1150 0x7f96d40a3600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:28:26.598 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:26.593+0000 7f96e659c700 1 --2- 172.21.15.98:0/245826581 >> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] conn(0x7f96d40a1150 0x7f96d40a3600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:28:26.598 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:26.593+0000 7f96dffff700 1 -- 172.21.15.98:0/245826581 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(79..79 src has 1..79) v4 ==== 7127+0+0 (secure 0 0 0) 0x7f96d0014070 con 0x7f96e80fee70 2023-07-14T19:28:26.598 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:26.593+0000 7f96e659c700 1 --2- 172.21.15.98:0/245826581 >> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] conn(0x7f96d40a1150 0x7f96d40a3600 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f96d8009fd0 tx=0x7f96d8005fb0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:28:26.598 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:26.596+0000 7f96dffff700 1 -- 172.21.15.98:0/245826581 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f96d0096590 con 0x7f96e80fee70 2023-07-14T19:28:26.779 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:26.779+0000 7f96ed1af700 1 -- 172.21.15.98:0/245826581 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f96e80690f0 con 0x7f96e80fee70 2023-07-14T19:28:26.780 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:26.779+0000 7f96dffff700 1 -- 172.21.15.98:0/245826581 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v79) v1 ==== 93+0+10 (secure 0 0 0) 0x7f96d0094e60 con 0x7f96e80fee70 2023-07-14T19:28:26.780 INFO:teuthology.orchestra.run.smithi098.stdout:pg_num: 1 2023-07-14T19:28:26.782 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:26.781+0000 7f96ed1af700 1 -- 172.21.15.98:0/245826581 >> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] conn(0x7f96d40a1150 msgr2=0x7f96d40a3600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:28:26.782 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:26.781+0000 7f96ed1af700 1 --2- 172.21.15.98:0/245826581 >> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] conn(0x7f96d40a1150 0x7f96d40a3600 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f96d8009fd0 tx=0x7f96d8005fb0 comp rx=0 tx=0).stop 2023-07-14T19:28:26.782 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:26.781+0000 7f96ed1af700 1 -- 172.21.15.98:0/245826581 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f96e80fee70 msgr2=0x7f96e8196ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:28:26.782 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:26.781+0000 7f96ed1af700 1 --2- 172.21.15.98:0/245826581 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f96e80fee70 0x7f96e8196ba0 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f96d000e9a0 tx=0x7f96d000ecb0 comp rx=0 tx=0).stop 2023-07-14T19:28:26.782 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:26.781+0000 7f96ed1af700 1 -- 172.21.15.98:0/245826581 shutdown_connections 2023-07-14T19:28:26.783 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:26.781+0000 7f96ed1af700 1 --2- 172.21.15.98:0/245826581 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f96e8103f10 0x7f96e81a1c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:28:26.783 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:26.781+0000 7f96ed1af700 1 --2- 172.21.15.98:0/245826581 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f96e80fee70 0x7f96e8196ba0 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:28:26.783 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:26.781+0000 7f96ed1af700 1 --2- 172.21.15.98:0/245826581 >> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] conn(0x7f96d40a1150 0x7f96d40a3600 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:28:26.783 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:26.781+0000 7f96ed1af700 1 --2- 172.21.15.98:0/245826581 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f96e80ff810 0x7f96e81970e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:28:26.783 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:26.781+0000 7f96ed1af700 1 -- 172.21.15.98:0/245826581 >> 172.21.15.98:0/245826581 conn(0x7f96e8075ae0 msgr2=0x7f96e80fc790 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:28:26.783 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:26.781+0000 7f96ed1af700 1 -- 172.21.15.98:0/245826581 shutdown_connections 2023-07-14T19:28:26.783 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:26.781+0000 7f96ed1af700 1 -- 172.21.15.98:0/245826581 wait complete. 2023-07-14T19:28:27.126 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:28:26 smithi098 ceph-mon[127486]: pgmap v231: 1 pgs: 1 active+clean; 577 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-14T19:28:27.127 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:28:26 smithi098 ceph-mon[127486]: from='client.? 172.21.15.98:0/245826581' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2023-07-14T19:28:27.128 INFO:tasks.cephadm:Adding mds.a on smithi098 2023-07-14T19:28:27.128 INFO:tasks.cephadm:Adding mds.d on smithi098 2023-07-14T19:28:27.129 INFO:tasks.cephadm:Adding mds.g on smithi098 2023-07-14T19:28:27.129 INFO:tasks.cephadm:Adding mds.j on smithi098 2023-07-14T19:28:27.130 INFO:tasks.cephadm:Adding mds.b on smithi101 2023-07-14T19:28:27.130 INFO:tasks.cephadm:Adding mds.e on smithi101 2023-07-14T19:28:27.130 INFO:tasks.cephadm:Adding mds.h on smithi101 2023-07-14T19:28:27.131 INFO:tasks.cephadm:Adding mds.k on smithi101 2023-07-14T19:28:27.131 INFO:tasks.cephadm:Adding mds.c on smithi139 2023-07-14T19:28:27.132 INFO:tasks.cephadm:Adding mds.f on smithi139 2023-07-14T19:28:27.132 INFO:tasks.cephadm:Adding mds.i on smithi139 2023-07-14T19:28:27.132 INFO:tasks.cephadm:Adding mds.l on smithi139 2023-07-14T19:28:27.133 DEBUG:teuthology.orchestra.run.smithi139:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81c9387d2e7576da065747ec9c16d1c7d69016ec shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 9adea648-227b-11ee-9b33-001a4aab830c -- ceph orch apply mds all '12;smithi098=a;smithi098=d;smithi098=g;smithi098=j;smithi101=b;smithi101=e;smithi101=h;smithi101=k;smithi139=c;smithi139=f;smithi139=i;smithi139=l' 2023-07-14T19:28:27.221 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:28:26 smithi139 ceph-mon[129293]: pgmap v231: 1 pgs: 1 active+clean; 577 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-14T19:28:27.221 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:28:26 smithi139 ceph-mon[129293]: from='client.? 172.21.15.98:0/245826581' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2023-07-14T19:28:27.441 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:28:26 smithi101 ceph-mon[128807]: pgmap v231: 1 pgs: 1 active+clean; 577 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-14T19:28:27.441 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:28:26 smithi101 ceph-mon[128807]: from='client.? 172.21.15.98:0/245826581' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2023-07-14T19:28:27.701 INFO:teuthology.orchestra.run.smithi139.stderr:Inferring config /var/lib/ceph/9adea648-227b-11ee-9b33-001a4aab830c/mon.c/config 2023-07-14T19:28:28.896 INFO:teuthology.orchestra.run.smithi139.stderr:2023-07-14T19:28:28.895+0000 7f910101a700 1 -- 172.21.15.139:0/3500070777 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f90fc100740 msgr2=0x7f90fc107000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:28:28.897 INFO:teuthology.orchestra.run.smithi139.stderr:2023-07-14T19:28:28.895+0000 7f910101a700 1 --2- 172.21.15.139:0/3500070777 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f90fc100740 0x7f90fc107000 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f90f400b0a0 tx=0x7f90f400b3b0 comp rx=0 tx=0).stop 2023-07-14T19:28:28.897 INFO:teuthology.orchestra.run.smithi139.stderr:2023-07-14T19:28:28.895+0000 7f910101a700 1 -- 172.21.15.139:0/3500070777 shutdown_connections 2023-07-14T19:28:28.897 INFO:teuthology.orchestra.run.smithi139.stderr:2023-07-14T19:28:28.895+0000 7f910101a700 1 --2- 172.21.15.139:0/3500070777 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f90fc100740 0x7f90fc107000 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:28:28.897 INFO:teuthology.orchestra.run.smithi139.stderr:2023-07-14T19:28:28.895+0000 7f910101a700 1 --2- 172.21.15.139:0/3500070777 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f90fc0febb0 0x7f90fc0fefc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:28:28.897 INFO:teuthology.orchestra.run.smithi139.stderr:2023-07-14T19:28:28.895+0000 7f910101a700 1 --2- 172.21.15.139:0/3500070777 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f90fc0ffdb0 0x7f90fc100200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:28:28.897 INFO:teuthology.orchestra.run.smithi139.stderr:2023-07-14T19:28:28.895+0000 7f910101a700 1 -- 172.21.15.139:0/3500070777 >> 172.21.15.139:0/3500070777 conn(0x7f90fc0fa1e0 msgr2=0x7f90fc0fc630 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:28:28.897 INFO:teuthology.orchestra.run.smithi139.stderr:2023-07-14T19:28:28.895+0000 7f910101a700 1 -- 172.21.15.139:0/3500070777 shutdown_connections 2023-07-14T19:28:28.898 INFO:teuthology.orchestra.run.smithi139.stderr:2023-07-14T19:28:28.895+0000 7f910101a700 1 -- 172.21.15.139:0/3500070777 wait complete. 2023-07-14T19:28:28.899 INFO:teuthology.orchestra.run.smithi139.stderr:2023-07-14T19:28:28.895+0000 7f910101a700 1 Processor -- start 2023-07-14T19:28:28.899 INFO:teuthology.orchestra.run.smithi139.stderr:2023-07-14T19:28:28.896+0000 7f910101a700 1 -- start start 2023-07-14T19:28:28.899 INFO:teuthology.orchestra.run.smithi139.stderr:2023-07-14T19:28:28.896+0000 7f910101a700 1 --2- >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f90fc0febb0 0x7f90fc195b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:28:28.899 INFO:teuthology.orchestra.run.smithi139.stderr:2023-07-14T19:28:28.896+0000 7f910101a700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f90fc0ffdb0 0x7f90fc196060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:28:28.900 INFO:teuthology.orchestra.run.smithi139.stderr:2023-07-14T19:28:28.896+0000 7f910101a700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f90fc100740 0x7f90fc19d0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:28:28.900 INFO:teuthology.orchestra.run.smithi139.stderr:2023-07-14T19:28:28.896+0000 7f910101a700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f90fc19d6b0 con 0x7f90fc0ffdb0 2023-07-14T19:28:28.900 INFO:teuthology.orchestra.run.smithi139.stderr:2023-07-14T19:28:28.896+0000 7f910101a700 1 -- --> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f90fc19d7f0 con 0x7f90fc0febb0 2023-07-14T19:28:28.900 INFO:teuthology.orchestra.run.smithi139.stderr:2023-07-14T19:28:28.896+0000 7f910101a700 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f90fc19d980 con 0x7f90fc100740 2023-07-14T19:28:28.900 INFO:teuthology.orchestra.run.smithi139.stderr:2023-07-14T19:28:28.896+0000 7f90fb59e700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f90fc100740 0x7f90fc19d0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:28:28.900 INFO:teuthology.orchestra.run.smithi139.stderr:2023-07-14T19:28:28.896+0000 7f90fb59e700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f90fc100740 0x7f90fc19d0e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.139:3300/0 says I am v2:172.21.15.139:37572/0 (socket says 172.21.15.139:37572) 2023-07-14T19:28:28.900 INFO:teuthology.orchestra.run.smithi139.stderr:2023-07-14T19:28:28.896+0000 7f90fb59e700 1 -- 172.21.15.139:0/1871033866 learned_addr learned my addr 172.21.15.139:0/1871033866 (peer_addr_for_me v2:172.21.15.139:0/0) 2023-07-14T19:28:28.901 INFO:teuthology.orchestra.run.smithi139.stderr:2023-07-14T19:28:28.896+0000 7f90fa59c700 1 --2- 172.21.15.139:0/1871033866 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f90fc0ffdb0 0x7f90fc196060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:28:28.901 INFO:teuthology.orchestra.run.smithi139.stderr:2023-07-14T19:28:28.896+0000 7f90fad9d700 1 --2- 172.21.15.139:0/1871033866 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f90fc0febb0 0x7f90fc195b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:28:28.901 INFO:teuthology.orchestra.run.smithi139.stderr:2023-07-14T19:28:28.896+0000 7f90fb59e700 1 -- 172.21.15.139:0/1871033866 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f90fc0febb0 msgr2=0x7f90fc195b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:28:28.901 INFO:teuthology.orchestra.run.smithi139.stderr:2023-07-14T19:28:28.896+0000 7f90fb59e700 1 --2- 172.21.15.139:0/1871033866 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f90fc0febb0 0x7f90fc195b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:28:28.901 INFO:teuthology.orchestra.run.smithi139.stderr:2023-07-14T19:28:28.896+0000 7f90fb59e700 1 -- 172.21.15.139:0/1871033866 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f90fc0ffdb0 msgr2=0x7f90fc196060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:28:28.901 INFO:teuthology.orchestra.run.smithi139.stderr:2023-07-14T19:28:28.896+0000 7f90fb59e700 1 --2- 172.21.15.139:0/1871033866 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f90fc0ffdb0 0x7f90fc196060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:28:28.901 INFO:teuthology.orchestra.run.smithi139.stderr:2023-07-14T19:28:28.896+0000 7f90fb59e700 1 -- 172.21.15.139:0/1871033866 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f90f4009d00 con 0x7f90fc100740 2023-07-14T19:28:28.902 INFO:teuthology.orchestra.run.smithi139.stderr:2023-07-14T19:28:28.896+0000 7f90fb59e700 1 --2- 172.21.15.139:0/1871033866 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f90fc100740 0x7f90fc19d0e0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f90f4004d00 tx=0x7f90f4004d30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:28:28.902 INFO:teuthology.orchestra.run.smithi139.stderr:2023-07-14T19:28:28.897+0000 7f90e3fff700 1 -- 172.21.15.139:0/1871033866 <== mon.1 v2:172.21.15.139:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f90f400e040 con 0x7f90fc100740 2023-07-14T19:28:28.902 INFO:teuthology.orchestra.run.smithi139.stderr:2023-07-14T19:28:28.897+0000 7f910101a700 1 -- 172.21.15.139:0/1871033866 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f90fc19dc00 con 0x7f90fc100740 2023-07-14T19:28:28.902 INFO:teuthology.orchestra.run.smithi139.stderr:2023-07-14T19:28:28.897+0000 7f90e3fff700 1 -- 172.21.15.139:0/1871033866 <== mon.1 v2:172.21.15.139:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f90f401bb20 con 0x7f90fc100740 2023-07-14T19:28:28.902 INFO:teuthology.orchestra.run.smithi139.stderr:2023-07-14T19:28:28.897+0000 7f90e3fff700 1 -- 172.21.15.139:0/1871033866 <== mon.1 v2:172.21.15.139:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f90f40057f0 con 0x7f90fc100740 2023-07-14T19:28:28.902 INFO:teuthology.orchestra.run.smithi139.stderr:2023-07-14T19:28:28.897+0000 7f910101a700 1 -- 172.21.15.139:0/1871033866 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f90fc19e070 con 0x7f90fc100740 2023-07-14T19:28:28.902 INFO:teuthology.orchestra.run.smithi139.stderr:2023-07-14T19:28:28.898+0000 7f910101a700 1 -- 172.21.15.139:0/1871033866 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f90fc10a410 con 0x7f90fc100740 2023-07-14T19:28:28.904 INFO:teuthology.orchestra.run.smithi139.stderr:2023-07-14T19:28:28.899+0000 7f90e3fff700 1 -- 172.21.15.139:0/1871033866 <== mon.1 v2:172.21.15.139:3300/0 4 ==== mgrmap(e 16) v1 ==== 135715+0+0 (secure 0 0 0) 0x7f90f4005950 con 0x7f90fc100740 2023-07-14T19:28:28.904 INFO:teuthology.orchestra.run.smithi139.stderr:2023-07-14T19:28:28.900+0000 7f90e3fff700 1 --2- 172.21.15.139:0/1871033866 >> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] conn(0x7f90e40a1150 0x7f90e40a3600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:28:28.904 INFO:teuthology.orchestra.run.smithi139.stderr:2023-07-14T19:28:28.900+0000 7f90e3fff700 1 -- 172.21.15.139:0/1871033866 <== mon.1 v2:172.21.15.139:3300/0 5 ==== osd_map(79..79 src has 1..79) v4 ==== 7127+0+0 (secure 0 0 0) 0x7f90f40ce690 con 0x7f90fc100740 2023-07-14T19:28:28.904 INFO:teuthology.orchestra.run.smithi139.stderr:2023-07-14T19:28:28.900+0000 7f90fad9d700 1 --2- 172.21.15.139:0/1871033866 >> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] conn(0x7f90e40a1150 0x7f90e40a3600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:28:28.904 INFO:teuthology.orchestra.run.smithi139.stderr:2023-07-14T19:28:28.900+0000 7f90fad9d700 1 --2- 172.21.15.139:0/1871033866 >> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] conn(0x7f90e40a1150 0x7f90e40a3600 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f90f0006fd0 tx=0x7f90f0008040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:28:28.905 INFO:teuthology.orchestra.run.smithi139.stderr:2023-07-14T19:28:28.902+0000 7f90e3fff700 1 -- 172.21.15.139:0/1871033866 <== mon.1 v2:172.21.15.139:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f90f4098310 con 0x7f90fc100740 2023-07-14T19:28:28.970 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:28:28 smithi139 ceph-mon[129293]: pgmap v232: 1 pgs: 1 active+clean; 577 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-14T19:28:29.101 INFO:teuthology.orchestra.run.smithi139.stderr:2023-07-14T19:28:29.100+0000 7f910101a700 1 -- 172.21.15.139:0/1871033866 --> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] -- mgr_command(tid 0: {"prefix": "orch apply mds", "fs_name": "all", "placement": "12;smithi098=a;smithi098=d;smithi098=g;smithi098=j;smithi101=b;smithi101=e;smithi101=h;smithi101=k;smithi139=c;smithi139=f;smithi139=i;smithi139=l", "target": ["mon-mgr", ""]}) v1 -- 0x7f90fc069240 con 0x7f90e40a1150 2023-07-14T19:28:29.109 INFO:teuthology.orchestra.run.smithi139.stderr:2023-07-14T19:28:29.108+0000 7f90e3fff700 1 -- 172.21.15.139:0/1871033866 <== mgr.14152 v2:172.21.15.98:6800/452042164 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+28 (secure 0 0 0) 0x7f90fc069240 con 0x7f90e40a1150 2023-07-14T19:28:29.110 INFO:teuthology.orchestra.run.smithi139.stdout:Scheduled mds.all update... 2023-07-14T19:28:29.112 INFO:teuthology.orchestra.run.smithi139.stderr:2023-07-14T19:28:29.110+0000 7f910101a700 1 -- 172.21.15.139:0/1871033866 >> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] conn(0x7f90e40a1150 msgr2=0x7f90e40a3600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:28:29.112 INFO:teuthology.orchestra.run.smithi139.stderr:2023-07-14T19:28:29.110+0000 7f910101a700 1 --2- 172.21.15.139:0/1871033866 >> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] conn(0x7f90e40a1150 0x7f90e40a3600 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f90f0006fd0 tx=0x7f90f0008040 comp rx=0 tx=0).stop 2023-07-14T19:28:29.113 INFO:teuthology.orchestra.run.smithi139.stderr:2023-07-14T19:28:29.110+0000 7f910101a700 1 -- 172.21.15.139:0/1871033866 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f90fc100740 msgr2=0x7f90fc19d0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:28:29.113 INFO:teuthology.orchestra.run.smithi139.stderr:2023-07-14T19:28:29.110+0000 7f910101a700 1 --2- 172.21.15.139:0/1871033866 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f90fc100740 0x7f90fc19d0e0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f90f4004d00 tx=0x7f90f4004d30 comp rx=0 tx=0).stop 2023-07-14T19:28:29.113 INFO:teuthology.orchestra.run.smithi139.stderr:2023-07-14T19:28:29.111+0000 7f910101a700 1 -- 172.21.15.139:0/1871033866 shutdown_connections 2023-07-14T19:28:29.113 INFO:teuthology.orchestra.run.smithi139.stderr:2023-07-14T19:28:29.111+0000 7f910101a700 1 --2- 172.21.15.139:0/1871033866 >> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] conn(0x7f90e40a1150 0x7f90e40a3600 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:28:29.114 INFO:teuthology.orchestra.run.smithi139.stderr:2023-07-14T19:28:29.111+0000 7f910101a700 1 --2- 172.21.15.139:0/1871033866 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f90fc100740 0x7f90fc19d0e0 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:28:29.114 INFO:teuthology.orchestra.run.smithi139.stderr:2023-07-14T19:28:29.111+0000 7f910101a700 1 --2- 172.21.15.139:0/1871033866 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f90fc0febb0 0x7f90fc195b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:28:29.114 INFO:teuthology.orchestra.run.smithi139.stderr:2023-07-14T19:28:29.111+0000 7f910101a700 1 --2- 172.21.15.139:0/1871033866 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f90fc0ffdb0 0x7f90fc196060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:28:29.115 INFO:teuthology.orchestra.run.smithi139.stderr:2023-07-14T19:28:29.111+0000 7f910101a700 1 -- 172.21.15.139:0/1871033866 >> 172.21.15.139:0/1871033866 conn(0x7f90fc0fa1e0 msgr2=0x7f90fc068a30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:28:29.115 INFO:teuthology.orchestra.run.smithi139.stderr:2023-07-14T19:28:29.111+0000 7f910101a700 1 -- 172.21.15.139:0/1871033866 shutdown_connections 2023-07-14T19:28:29.115 INFO:teuthology.orchestra.run.smithi139.stderr:2023-07-14T19:28:29.111+0000 7f910101a700 1 -- 172.21.15.139:0/1871033866 wait complete. 2023-07-14T19:28:29.281 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:28:28 smithi098 ceph-mon[127486]: pgmap v232: 1 pgs: 1 active+clean; 577 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-14T19:28:29.441 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:28:28 smithi101 ceph-mon[128807]: pgmap v232: 1 pgs: 1 active+clean; 577 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-14T19:28:29.542 DEBUG:teuthology.orchestra.run.smithi098:mds.a> sudo journalctl -f -n 0 -u ceph-9adea648-227b-11ee-9b33-001a4aab830c@mds.a.service 2023-07-14T19:28:29.546 DEBUG:teuthology.orchestra.run.smithi098:mds.d> sudo journalctl -f -n 0 -u ceph-9adea648-227b-11ee-9b33-001a4aab830c@mds.d.service 2023-07-14T19:28:29.583 INFO:journalctl@ceph.mds.a.smithi098.stdout:-- Logs begin at Fri 2023-07-14 18:57:10 UTC. -- 2023-07-14T19:28:29.584 DEBUG:teuthology.orchestra.run.smithi098:mds.g> sudo journalctl -f -n 0 -u ceph-9adea648-227b-11ee-9b33-001a4aab830c@mds.g.service 2023-07-14T19:28:29.613 INFO:journalctl@ceph.mds.d.smithi098.stdout:-- Logs begin at Fri 2023-07-14 18:57:10 UTC. -- 2023-07-14T19:28:29.614 DEBUG:teuthology.orchestra.run.smithi098:mds.j> sudo journalctl -f -n 0 -u ceph-9adea648-227b-11ee-9b33-001a4aab830c@mds.j.service 2023-07-14T19:28:29.642 INFO:journalctl@ceph.mds.g.smithi098.stdout:-- Logs begin at Fri 2023-07-14 18:57:10 UTC. -- 2023-07-14T19:28:29.643 DEBUG:teuthology.orchestra.run.smithi101:mds.b> sudo journalctl -f -n 0 -u ceph-9adea648-227b-11ee-9b33-001a4aab830c@mds.b.service 2023-07-14T19:28:29.646 DEBUG:teuthology.orchestra.run.smithi101:mds.e> sudo journalctl -f -n 0 -u ceph-9adea648-227b-11ee-9b33-001a4aab830c@mds.e.service 2023-07-14T19:28:29.669 INFO:journalctl@ceph.mds.j.smithi098.stdout:-- Logs begin at Fri 2023-07-14 18:57:10 UTC. -- 2023-07-14T19:28:29.682 INFO:journalctl@ceph.mds.b.smithi101.stdout:-- Logs begin at Fri 2023-07-14 18:56:58 UTC. -- 2023-07-14T19:28:29.684 DEBUG:teuthology.orchestra.run.smithi101:mds.h> sudo journalctl -f -n 0 -u ceph-9adea648-227b-11ee-9b33-001a4aab830c@mds.h.service 2023-07-14T19:28:29.712 INFO:journalctl@ceph.mds.e.smithi101.stdout:-- Logs begin at Fri 2023-07-14 18:56:58 UTC. -- 2023-07-14T19:28:29.713 DEBUG:teuthology.orchestra.run.smithi101:mds.k> sudo journalctl -f -n 0 -u ceph-9adea648-227b-11ee-9b33-001a4aab830c@mds.k.service 2023-07-14T19:28:29.740 INFO:journalctl@ceph.mds.h.smithi101.stdout:-- Logs begin at Fri 2023-07-14 18:56:58 UTC. -- 2023-07-14T19:28:29.741 DEBUG:teuthology.orchestra.run.smithi139:mds.c> sudo journalctl -f -n 0 -u ceph-9adea648-227b-11ee-9b33-001a4aab830c@mds.c.service 2023-07-14T19:28:29.744 DEBUG:teuthology.orchestra.run.smithi139:mds.f> sudo journalctl -f -n 0 -u ceph-9adea648-227b-11ee-9b33-001a4aab830c@mds.f.service 2023-07-14T19:28:29.767 INFO:journalctl@ceph.mds.k.smithi101.stdout:-- Logs begin at Fri 2023-07-14 18:56:58 UTC. -- 2023-07-14T19:28:29.774 INFO:journalctl@ceph.mds.c.smithi139.stdout:-- Logs begin at Fri 2023-07-14 18:56:10 UTC. -- 2023-07-14T19:28:29.775 DEBUG:teuthology.orchestra.run.smithi139:mds.i> sudo journalctl -f -n 0 -u ceph-9adea648-227b-11ee-9b33-001a4aab830c@mds.i.service 2023-07-14T19:28:29.802 INFO:journalctl@ceph.mds.f.smithi139.stdout:-- Logs begin at Fri 2023-07-14 18:56:10 UTC. -- 2023-07-14T19:28:29.804 DEBUG:teuthology.orchestra.run.smithi139:mds.l> sudo journalctl -f -n 0 -u ceph-9adea648-227b-11ee-9b33-001a4aab830c@mds.l.service 2023-07-14T19:28:29.831 INFO:journalctl@ceph.mds.i.smithi139.stdout:-- Logs begin at Fri 2023-07-14 18:56:10 UTC. -- 2023-07-14T19:28:29.832 INFO:tasks.cephadm:Setting up CephFS filesystem(s)... 2023-07-14T19:28:29.833 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-07-14T19:28:29.859 INFO:journalctl@ceph.mds.l.smithi139.stdout:-- Logs begin at Fri 2023-07-14 18:56:10 UTC. -- 2023-07-14T19:28:29.971 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:29.968+0000 7f86dd2f8700 1 -- 172.21.15.98:0/4216722136 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f86d81006a0 msgr2=0x7f86d8106f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:28:29.971 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:29.968+0000 7f86dd2f8700 1 --2- 172.21.15.98:0/4216722136 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f86d81006a0 0x7f86d8106f60 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f86c800b0a0 tx=0x7f86c800b3b0 comp rx=0 tx=0).stop 2023-07-14T19:28:29.971 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:29.968+0000 7f86dd2f8700 1 -- 172.21.15.98:0/4216722136 shutdown_connections 2023-07-14T19:28:29.971 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:29.968+0000 7f86dd2f8700 1 --2- 172.21.15.98:0/4216722136 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f86d81006a0 0x7f86d8106f60 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:28:29.971 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:29.968+0000 7f86dd2f8700 1 --2- 172.21.15.98:0/4216722136 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f86d80feb20 0x7f86d80fef30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:28:29.972 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:29.968+0000 7f86dd2f8700 1 --2- 172.21.15.98:0/4216722136 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f86d80ffd10 0x7f86d8100160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:28:29.972 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:29.968+0000 7f86dd2f8700 1 -- 172.21.15.98:0/4216722136 >> 172.21.15.98:0/4216722136 conn(0x7f86d80fa100 msgr2=0x7f86d80fc570 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:28:29.973 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:29.968+0000 7f86dd2f8700 1 -- 172.21.15.98:0/4216722136 shutdown_connections 2023-07-14T19:28:29.973 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:29.968+0000 7f86dd2f8700 1 -- 172.21.15.98:0/4216722136 wait complete. 2023-07-14T19:28:29.973 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:29.969+0000 7f86dd2f8700 1 Processor -- start 2023-07-14T19:28:29.973 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:29.969+0000 7f86dd2f8700 1 -- start start 2023-07-14T19:28:29.973 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:29.969+0000 7f86dd2f8700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f86d80feb20 0x7f86d8195ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:28:29.974 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:29.969+0000 7f86dd2f8700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f86d80ffd10 0x7f86d8196010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:28:29.974 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:29.969+0000 7f86dd2f8700 1 --2- >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f86d81006a0 0x7f86d819d080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:28:29.974 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:29.969+0000 7f86dd2f8700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f86d819d650 con 0x7f86d80ffd10 2023-07-14T19:28:29.974 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:29.969+0000 7f86dd2f8700 1 -- --> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f86d819d790 con 0x7f86d81006a0 2023-07-14T19:28:29.974 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:29.969+0000 7f86dd2f8700 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f86d819d920 con 0x7f86d80feb20 2023-07-14T19:28:29.974 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:29.969+0000 7f86d67fc700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f86d80ffd10 0x7f86d8196010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:28:29.974 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:29.969+0000 7f86d67fc700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f86d80ffd10 0x7f86d8196010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:45326/0 (socket says 172.21.15.98:45326) 2023-07-14T19:28:29.975 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:29.969+0000 7f86d67fc700 1 -- 172.21.15.98:0/240399753 learned_addr learned my addr 172.21.15.98:0/240399753 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-07-14T19:28:29.975 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:29.969+0000 7f86d77fe700 1 --2- 172.21.15.98:0/240399753 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f86d81006a0 0x7f86d819d080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:28:29.975 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:29.969+0000 7f86d6ffd700 1 --2- 172.21.15.98:0/240399753 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f86d80feb20 0x7f86d8195ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:28:29.975 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:29.970+0000 7f86d67fc700 1 -- 172.21.15.98:0/240399753 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f86d80feb20 msgr2=0x7f86d8195ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:28:29.975 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:29.970+0000 7f86d67fc700 1 --2- 172.21.15.98:0/240399753 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f86d80feb20 0x7f86d8195ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:28:29.975 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:29.970+0000 7f86d67fc700 1 -- 172.21.15.98:0/240399753 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f86d81006a0 msgr2=0x7f86d819d080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:28:29.975 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:29.970+0000 7f86d67fc700 1 --2- 172.21.15.98:0/240399753 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f86d81006a0 0x7f86d819d080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:28:29.976 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:29.970+0000 7f86d67fc700 1 -- 172.21.15.98:0/240399753 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f86c8009d00 con 0x7f86d80ffd10 2023-07-14T19:28:29.976 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:29.970+0000 7f86d6ffd700 1 --2- 172.21.15.98:0/240399753 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f86d80feb20 0x7f86d8195ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-07-14T19:28:29.976 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:29.970+0000 7f86d67fc700 1 --2- 172.21.15.98:0/240399753 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f86d80ffd10 0x7f86d8196010 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f86c000b530 tx=0x7f86c000b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:28:29.976 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:29.970+0000 7f86cffff700 1 -- 172.21.15.98:0/240399753 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f86c0011830 con 0x7f86d80ffd10 2023-07-14T19:28:29.976 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:29.970+0000 7f86dd2f8700 1 -- 172.21.15.98:0/240399753 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f86d819dc00 con 0x7f86d80ffd10 2023-07-14T19:28:29.976 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:29.970+0000 7f86cffff700 1 -- 172.21.15.98:0/240399753 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f86c0011e50 con 0x7f86d80ffd10 2023-07-14T19:28:29.976 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:29.970+0000 7f86cffff700 1 -- 172.21.15.98:0/240399753 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f86c000f2e0 con 0x7f86d80ffd10 2023-07-14T19:28:29.976 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:29.970+0000 7f86dd2f8700 1 -- 172.21.15.98:0/240399753 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f86d819e160 con 0x7f86d80ffd10 2023-07-14T19:28:29.977 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:29.971+0000 7f86dd2f8700 1 -- 172.21.15.98:0/240399753 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f86d810a3c0 con 0x7f86d80ffd10 2023-07-14T19:28:29.978 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:29.972+0000 7f86cffff700 1 -- 172.21.15.98:0/240399753 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 16) v1 ==== 135715+0+0 (secure 0 0 0) 0x7f86c0010380 con 0x7f86d80ffd10 2023-07-14T19:28:29.978 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:29.973+0000 7f86cffff700 1 --2- 172.21.15.98:0/240399753 >> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] conn(0x7f86c40a11a0 0x7f86c40a3650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:28:29.978 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:29.973+0000 7f86d6ffd700 1 --2- 172.21.15.98:0/240399753 >> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] conn(0x7f86c40a11a0 0x7f86c40a3650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:28:29.978 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:29.973+0000 7f86cffff700 1 -- 172.21.15.98:0/240399753 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(79..79 src has 1..79) v4 ==== 7127+0+0 (secure 0 0 0) 0x7f86c00cc510 con 0x7f86d80ffd10 2023-07-14T19:28:29.978 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:29.973+0000 7f86d6ffd700 1 --2- 172.21.15.98:0/240399753 >> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] conn(0x7f86c40a11a0 0x7f86c40a3650 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f86d0005950 tx=0x7f86d00058e0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:28:29.978 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:29.976+0000 7f86cffff700 1 -- 172.21.15.98:0/240399753 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f86c0096100 con 0x7f86d80ffd10 2023-07-14T19:28:30.108 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:28:30 smithi098 ceph-mon[127486]: from='client.24406 -' entity='client.admin' cmd=[{"prefix": "orch apply mds", "fs_name": "all", "placement": "12;smithi098=a;smithi098=d;smithi098=g;smithi098=j;smithi101=b;smithi101=e;smithi101=h;smithi101=k;smithi139=c;smithi139=f;smithi139=i;smithi139=l", "target": ["mon-mgr", ""]}]: dispatch 2023-07-14T19:28:30.170 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:30.169+0000 7f86dd2f8700 1 -- 172.21.15.98:0/240399753 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f86d819e770 con 0x7f86d80ffd10 2023-07-14T19:28:30.170 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:30.170+0000 7f86cffff700 1 -- 172.21.15.98:0/240399753 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v79) v1 ==== 74+0+18420 (secure 0 0 0) 0x7f86c0099720 con 0x7f86d80ffd10 2023-07-14T19:28:30.171 INFO:teuthology.orchestra.run.smithi098.stdout: 2023-07-14T19:28:30.171 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":79,"fsid":"9adea648-227b-11ee-9b33-001a4aab830c","created":"2023-07-14T19:22:07.178854+0000","modified":"2023-07-14T19:28:20.933203+0000","last_up_change":"2023-07-14T19:28:19.646827+0000","last_in_change":"2023-07-14T19:28:02.018325+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-07-14T19:25:09.845827+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"a3797e89-1754-41cf-9517-227e06ffdd72","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6802","nonce":120507228},{"type":"v1","addr":"172.21.15.98:6803","nonce":120507228}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6804","nonce":120507228},{"type":"v1","addr":"172.21.15.98:6805","nonce":120507228}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6808","nonce":120507228},{"type":"v1","addr":"172.21.15.98:6809","nonce":120507228}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6806","nonce":120507228},{"type":"v1","addr":"172.21.15.98:6807","nonce":120507228}]},"public_addr":"172.21.15.98:6803/120507228","cluster_addr":"172.21.15.98:6805/120507228","heartbeat_back_addr":"172.21.15.98:6809/120507228","heartbeat_front_addr":"172.21.15.98:6807/120507228","state":["exists","up"]},{"osd":1,"uuid":"2b001add-1aaa-4496-a403-7ef9725b80a0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.101:6800","nonce":2450584584},{"type":"v1","addr":"172.21.15.101:6801","nonce":2450584584}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.101:6802","nonce":2450584584},{"type":"v1","addr":"172.21.15.101:6803","nonce":2450584584}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.101:6806","nonce":2450584584},{"type":"v1","addr":"172.21.15.101:6807","nonce":2450584584}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.101:6804","nonce":2450584584},{"type":"v1","addr":"172.21.15.101:6805","nonce":2450584584}]},"public_addr":"172.21.15.101:6801/2450584584","cluster_addr":"172.21.15.101:6803/2450584584","heartbeat_back_addr":"172.21.15.101:6807/2450584584","heartbeat_front_addr":"172.21.15.101:6805/2450584584","state":["exists","up"]},{"osd":2,"uuid":"48e6d499-3ca9-424b-b0fb-3ba5f33b9fc9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":945038995},{"type":"v1","addr":"172.21.15.139:6801","nonce":945038995}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":945038995},{"type":"v1","addr":"172.21.15.139:6803","nonce":945038995}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":945038995},{"type":"v1","addr":"172.21.15.139:6807","nonce":945038995}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":945038995},{"type":"v1","addr":"172.21.15.139:6805","nonce":945038995}]},"public_addr":"172.21.15.139:6801/945038995","cluster_addr":"172.21.15.139:6803/945038995","heartbeat_back_addr":"172.21.15.139:6807/945038995","heartbeat_front_addr":"172.21.15.139:6805/945038995","state":["exists","up"]},{"osd":3,"uuid":"5530b265-31a5-447b-a1ea-17ebae9cfddf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6810","nonce":3095249665},{"type":"v1","addr":"172.21.15.98:6811","nonce":3095249665}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6812","nonce":3095249665},{"type":"v1","addr":"172.21.15.98:6813","nonce":3095249665}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6816","nonce":3095249665},{"type":"v1","addr":"172.21.15.98:6817","nonce":3095249665}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6814","nonce":3095249665},{"type":"v1","addr":"172.21.15.98:6815","nonce":3095249665}]},"public_addr":"172.21.15.98:6811/3095249665","cluster_addr":"172.21.15.98:6813/3095249665","heartbeat_back_addr":"172.21.15.98:6817/3095249665","heartbeat_front_addr":"172.21.15.98:6815/3095249665","state":["exists","up"]},{"osd":4,"uuid":"615d4642-a997-4f95-9e38-9fbdaa831400","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":32,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.101:6808","nonce":2844761406},{"type":"v1","addr":"172.21.15.101:6809","nonce":2844761406}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.101:6810","nonce":2844761406},{"type":"v1","addr":"172.21.15.101:6811","nonce":2844761406}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.101:6814","nonce":2844761406},{"type":"v1","addr":"172.21.15.101:6815","nonce":2844761406}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.101:6812","nonce":2844761406},{"type":"v1","addr":"172.21.15.101:6813","nonce":2844761406}]},"public_addr":"172.21.15.101:6809/2844761406","cluster_addr":"172.21.15.101:6811/2844761406","heartbeat_back_addr":"172.21.15.101:6815/2844761406","heartbeat_front_addr":"172.21.15.101:6813/2844761406","state":["exists","up"]},{"osd":5,"uuid":"415de5fa-607a-45a8-b732-28daf3be6dd3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":37,"up_thru":52,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":3532263042},{"type":"v1","addr":"172.21.15.139:6809","nonce":3532263042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6810","nonce":3532263042},{"type":"v1","addr":"172.21.15.139:6811","nonce":3532263042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6814","nonce":3532263042},{"type":"v1","addr":"172.21.15.139:6815","nonce":3532263042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":3532263042},{"type":"v1","addr":"172.21.15.139:6813","nonce":3532263042}]},"public_addr":"172.21.15.139:6809/3532263042","cluster_addr":"172.21.15.139:6811/3532263042","heartbeat_back_addr":"172.21.15.139:6815/3532263042","heartbeat_front_addr":"172.21.15.139:6813/3532263042","state":["exists","up"]},{"osd":6,"uuid":"d7f9d4ba-c309-4b75-a836-5a36384b528e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":45,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6818","nonce":658948743},{"type":"v1","addr":"172.21.15.98:6819","nonce":658948743}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6820","nonce":658948743},{"type":"v1","addr":"172.21.15.98:6821","nonce":658948743}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6824","nonce":658948743},{"type":"v1","addr":"172.21.15.98:6825","nonce":658948743}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6822","nonce":658948743},{"type":"v1","addr":"172.21.15.98:6823","nonce":658948743}]},"public_addr":"172.21.15.98:6819/658948743","cluster_addr":"172.21.15.98:6821/658948743","heartbeat_back_addr":"172.21.15.98:6825/658948743","heartbeat_front_addr":"172.21.15.98:6823/658948743","state":["exists","up"]},{"osd":7,"uuid":"eedb94d9-c5cd-4a93-8a8c-2648804b7195","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":50,"up_thru":73,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.101:6816","nonce":2881214066},{"type":"v1","addr":"172.21.15.101:6817","nonce":2881214066}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.101:6818","nonce":2881214066},{"type":"v1","addr":"172.21.15.101:6819","nonce":2881214066}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.101:6822","nonce":2881214066},{"type":"v1","addr":"172.21.15.101:6823","nonce":2881214066}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.101:6820","nonce":2881214066},{"type":"v1","addr":"172.21.15.101:6821","nonce":2881214066}]},"public_addr":"172.21.15.101:6817/2881214066","cluster_addr":"172.21.15.101:6819/2881214066","heartbeat_back_addr":"172.21.15.101:6823/2881214066","heartbeat_front_addr":"172.21.15.101:6821/2881214066","state":["exists","up"]},{"osd":8,"uuid":"6bfd77c2-5770-466e-aa6f-84b50c74ffb6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":57,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":3669514524},{"type":"v1","addr":"172.21.15.139:6817","nonce":3669514524}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":3669514524},{"type":"v1","addr":"172.21.15.139:6819","nonce":3669514524}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":3669514524},{"type":"v1","addr":"172.21.15.139:6823","nonce":3669514524}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6820","nonce":3669514524},{"type":"v1","addr":"172.21.15.139:6821","nonce":3669514524}]},"public_addr":"172.21.15.139:6817/3669514524","cluster_addr":"172.21.15.139:6819/3669514524","heartbeat_back_addr":"172.21.15.139:6823/3669514524","heartbeat_front_addr":"172.21.15.139:6821/3669514524","state":["exists","up"]},{"osd":9,"uuid":"19c5b54a-2242-4b23-b721-63be9559653d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":65,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6826","nonce":2893994080},{"type":"v1","addr":"172.21.15.98:6827","nonce":2893994080}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6828","nonce":2893994080},{"type":"v1","addr":"172.21.15.98:6829","nonce":2893994080}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6832","nonce":2893994080},{"type":"v1","addr":"172.21.15.98:6833","nonce":2893994080}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6830","nonce":2893994080},{"type":"v1","addr":"172.21.15.98:6831","nonce":2893994080}]},"public_addr":"172.21.15.98:6827/2893994080","cluster_addr":"172.21.15.98:6829/2893994080","heartbeat_back_addr":"172.21.15.98:6833/2893994080","heartbeat_front_addr":"172.21.15.98:6831/2893994080","state":["exists","up"]},{"osd":10,"uuid":"326f0209-ba9d-4b59-97cc-d61a758a8f29","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":70,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.101:6824","nonce":3545244225},{"type":"v1","addr":"172.21.15.101:6825","nonce":3545244225}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.101:6826","nonce":3545244225},{"type":"v1","addr":"172.21.15.101:6827","nonce":3545244225}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.101:6830","nonce":3545244225},{"type":"v1","addr":"172.21.15.101:6831","nonce":3545244225}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.101:6828","nonce":3545244225},{"type":"v1","addr":"172.21.15.101:6829","nonce":3545244225}]},"public_addr":"172.21.15.101:6825/3545244225","cluster_addr":"172.21.15.101:6827/3545244225","heartbeat_back_addr":"172.21.15.101:6831/3545244225","heartbeat_front_addr":"172.21.15.101:6829/3545244225","state":["exists","up"]},{"osd":11,"uuid":"c61c0320-1057-4f17-9bf1-7ad306ed7ed3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":78,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":2630023454},{"type":"v1","addr":"172.21.15.139:6825","nonce":2630023454}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6826","nonce":2630023454},{"type":"v1","addr":"172.21.15.139:6827","nonce":2630023454}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":2630023454},{"type":"v1","addr":"172.21.15.139:6831","nonce":2630023454}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":2630023454},{"type":"v1","addr":"172.21.15.139:6829","nonce":2630023454}]},"public_addr":"172.21.15.139:6825/2630023454","cluster_addr":"172.21.15.139:6827/2630023454","heartbeat_back_addr":"172.21.15.139:6831/2630023454","heartbeat_front_addr":"172.21.15.139:6829/2630023454","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T19:24:23.620028+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T19:24:45.779846+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T19:25:06.877949+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T19:25:27.994199+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T19:25:49.461139+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T19:26:10.409282+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T19:26:32.116390+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T19:26:53.536474+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T19:27:14.384839+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T19:27:35.635921+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T19:27:56.858744+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T19:28:18.342519+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.98:0/2164590253":"2023-07-15T19:23:01.520302+0000","172.21.15.98:0/2094734896":"2023-07-15T19:23:01.520302+0000","172.21.15.98:0/784745009":"2023-07-15T19:22:39.481502+0000","172.21.15.98:0/3657190818":"2023-07-15T19:22:39.481502+0000","172.21.15.98:6800/1732769087":"2023-07-15T19:23:01.520302+0000","172.21.15.98:0/3393542419":"2023-07-15T19:22:39.481502+0000","172.21.15.98:6800/749234665":"2023-07-15T19:22:39.481502+0000","172.21.15.98:6801/1732769087":"2023-07-15T19:23:01.520302+0000","172.21.15.98:6801/749234665":"2023-07-15T19:22:39.481502+0000","172.21.15.98:0/187822873":"2023-07-15T19:23:01.520302+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-07-14T19:28:30.173 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:30.172+0000 7f86dd2f8700 1 -- 172.21.15.98:0/240399753 >> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] conn(0x7f86c40a11a0 msgr2=0x7f86c40a3650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:28:30.173 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:30.172+0000 7f86dd2f8700 1 --2- 172.21.15.98:0/240399753 >> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] conn(0x7f86c40a11a0 0x7f86c40a3650 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f86d0005950 tx=0x7f86d00058e0 comp rx=0 tx=0).stop 2023-07-14T19:28:30.173 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:30.172+0000 7f86dd2f8700 1 -- 172.21.15.98:0/240399753 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f86d80ffd10 msgr2=0x7f86d8196010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:28:30.173 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:30.172+0000 7f86dd2f8700 1 --2- 172.21.15.98:0/240399753 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f86d80ffd10 0x7f86d8196010 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f86c000b530 tx=0x7f86c000b840 comp rx=0 tx=0).stop 2023-07-14T19:28:30.173 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:30.172+0000 7f86dd2f8700 1 -- 172.21.15.98:0/240399753 shutdown_connections 2023-07-14T19:28:30.174 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:30.172+0000 7f86dd2f8700 1 --2- 172.21.15.98:0/240399753 >> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] conn(0x7f86c40a11a0 0x7f86c40a3650 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:28:30.174 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:30.172+0000 7f86dd2f8700 1 --2- 172.21.15.98:0/240399753 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f86d80feb20 0x7f86d8195ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:28:30.174 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:30.172+0000 7f86dd2f8700 1 --2- 172.21.15.98:0/240399753 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f86d81006a0 0x7f86d819d080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:28:30.174 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:30.172+0000 7f86dd2f8700 1 --2- 172.21.15.98:0/240399753 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f86d80ffd10 0x7f86d8196010 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:28:30.174 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:30.172+0000 7f86dd2f8700 1 -- 172.21.15.98:0/240399753 >> 172.21.15.98:0/240399753 conn(0x7f86d80fa100 msgr2=0x7f86d8068870 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:28:30.174 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:30.173+0000 7f86dd2f8700 1 -- 172.21.15.98:0/240399753 shutdown_connections 2023-07-14T19:28:30.175 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:30.173+0000 7f86dd2f8700 1 -- 172.21.15.98:0/240399753 wait complete. 2023-07-14T19:28:30.186 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-07-14T19:25:09.845827+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-07-14T19:28:30.187 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-07-14T19:28:30.314 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:30.310+0000 7f35a8a49700 1 -- 172.21.15.98:0/4121524148 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f35a40ffd00 msgr2=0x7f35a4100150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:28:30.315 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:30.310+0000 7f35a8a49700 1 --2- 172.21.15.98:0/4121524148 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f35a40ffd00 0x7f35a4100150 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f3594009990 tx=0x7f3594009ca0 comp rx=0 tx=0).stop 2023-07-14T19:28:30.315 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:30.311+0000 7f35a8a49700 1 -- 172.21.15.98:0/4121524148 shutdown_connections 2023-07-14T19:28:30.315 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:30.311+0000 7f35a8a49700 1 --2- 172.21.15.98:0/4121524148 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f35a4100690 0x7f35a4106f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:28:30.315 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:30.311+0000 7f35a8a49700 1 --2- 172.21.15.98:0/4121524148 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f35a40feb10 0x7f35a40fef20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:28:30.315 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:30.311+0000 7f35a8a49700 1 --2- 172.21.15.98:0/4121524148 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f35a40ffd00 0x7f35a4100150 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:28:30.316 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:30.311+0000 7f35a8a49700 1 -- 172.21.15.98:0/4121524148 >> 172.21.15.98:0/4121524148 conn(0x7f35a40fa130 msgr2=0x7f35a40fc560 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:28:30.316 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:30.311+0000 7f35a8a49700 1 -- 172.21.15.98:0/4121524148 shutdown_connections 2023-07-14T19:28:30.316 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:30.311+0000 7f35a8a49700 1 -- 172.21.15.98:0/4121524148 wait complete. 2023-07-14T19:28:30.316 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:30.311+0000 7f35a8a49700 1 Processor -- start 2023-07-14T19:28:30.316 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:30.311+0000 7f35a8a49700 1 -- start start 2023-07-14T19:28:30.316 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:30.312+0000 7f35a8a49700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f35a40feb10 0x7f35a4068310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:28:30.316 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:30.312+0000 7f35a8a49700 1 --2- >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f35a40ffd00 0x7f35a419a650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:28:30.317 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:30.312+0000 7f35a8a49700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f35a4100690 0x7f35a419cc90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:28:30.317 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:30.312+0000 7f35a8a49700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f35a419d430 con 0x7f35a40feb10 2023-07-14T19:28:30.317 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:30.312+0000 7f35a8a49700 1 -- --> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f35a419d5a0 con 0x7f35a40ffd00 2023-07-14T19:28:30.317 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:30.312+0000 7f35a8a49700 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f35a419d760 con 0x7f35a4100690 2023-07-14T19:28:30.317 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:30.312+0000 7f35a259c700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f35a40feb10 0x7f35a4068310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:28:30.317 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:30.312+0000 7f35a259c700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f35a40feb10 0x7f35a4068310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:45338/0 (socket says 172.21.15.98:45338) 2023-07-14T19:28:30.317 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:30.312+0000 7f35a259c700 1 -- 172.21.15.98:0/3210300283 learned_addr learned my addr 172.21.15.98:0/3210300283 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-07-14T19:28:30.317 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:30.312+0000 7f35a2d9d700 1 --2- 172.21.15.98:0/3210300283 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f35a4100690 0x7f35a419cc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:28:30.318 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:30.312+0000 7f35a1d9b700 1 --2- 172.21.15.98:0/3210300283 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f35a40ffd00 0x7f35a419a650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:28:30.318 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:30.312+0000 7f35a259c700 1 -- 172.21.15.98:0/3210300283 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f35a4100690 msgr2=0x7f35a419cc90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:28:30.318 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:30.312+0000 7f35a259c700 1 --2- 172.21.15.98:0/3210300283 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f35a4100690 0x7f35a419cc90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:28:30.318 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:30.312+0000 7f35a259c700 1 -- 172.21.15.98:0/3210300283 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f35a40ffd00 msgr2=0x7f35a419a650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:28:30.318 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:30.312+0000 7f35a259c700 1 --2- 172.21.15.98:0/3210300283 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f35a40ffd00 0x7f35a419a650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:28:30.318 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:30.312+0000 7f35a259c700 1 -- 172.21.15.98:0/3210300283 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f35940095f0 con 0x7f35a40feb10 2023-07-14T19:28:30.318 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:30.312+0000 7f35a259c700 1 --2- 172.21.15.98:0/3210300283 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f35a40feb10 0x7f35a4068310 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f359800c990 tx=0x7f359800cca0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:28:30.319 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:30.312+0000 7f35937fe700 1 -- 172.21.15.98:0/3210300283 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3598007b00 con 0x7f35a40feb10 2023-07-14T19:28:30.319 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:30.312+0000 7f35937fe700 1 -- 172.21.15.98:0/3210300283 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f3598007c60 con 0x7f35a40feb10 2023-07-14T19:28:30.319 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:30.312+0000 7f35a8a49700 1 -- 172.21.15.98:0/3210300283 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f35a419da40 con 0x7f35a40feb10 2023-07-14T19:28:30.319 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:30.312+0000 7f35937fe700 1 -- 172.21.15.98:0/3210300283 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3598018660 con 0x7f35a40feb10 2023-07-14T19:28:30.319 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:30.312+0000 7f35a8a49700 1 -- 172.21.15.98:0/3210300283 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f35a419df50 con 0x7f35a40feb10 2023-07-14T19:28:30.319 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:30.313+0000 7f35a8a49700 1 -- 172.21.15.98:0/3210300283 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f35a41060a0 con 0x7f35a40feb10 2023-07-14T19:28:30.320 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:30.315+0000 7f35937fe700 1 -- 172.21.15.98:0/3210300283 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 16) v1 ==== 135715+0+0 (secure 0 0 0) 0x7f3598004020 con 0x7f35a40feb10 2023-07-14T19:28:30.320 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:30.315+0000 7f35937fe700 1 --2- 172.21.15.98:0/3210300283 >> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] conn(0x7f358c0a1150 0x7f358c0a3600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:28:30.321 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:30.315+0000 7f35a1d9b700 1 --2- 172.21.15.98:0/3210300283 >> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] conn(0x7f358c0a1150 0x7f358c0a3600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:28:30.321 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:30.316+0000 7f35937fe700 1 -- 172.21.15.98:0/3210300283 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(79..79 src has 1..79) v4 ==== 7127+0+0 (secure 0 0 0) 0x7f35980cc5d0 con 0x7f35a40feb10 2023-07-14T19:28:30.321 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:30.316+0000 7f35a1d9b700 1 --2- 172.21.15.98:0/3210300283 >> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] conn(0x7f358c0a1150 0x7f358c0a3600 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f3594006010 tx=0x7f359400b540 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:28:30.322 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:30.319+0000 7f35937fe700 1 -- 172.21.15.98:0/3210300283 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f35980961b0 con 0x7f35a40feb10 2023-07-14T19:28:30.442 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:28:30 smithi101 ceph-mon[128807]: from='client.24406 -' entity='client.admin' cmd=[{"prefix": "orch apply mds", "fs_name": "all", "placement": "12;smithi098=a;smithi098=d;smithi098=g;smithi098=j;smithi101=b;smithi101=e;smithi101=h;smithi101=k;smithi139=c;smithi139=f;smithi139=i;smithi139=l", "target": ["mon-mgr", ""]}]: dispatch 2023-07-14T19:28:30.442 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:28:30 smithi101 ceph-mon[128807]: Saving service mds.all spec with placement smithi098=a;smithi098=d;smithi098=g;smithi098=j;smithi101=b;smithi101=e;smithi101=h;smithi101=k;smithi139=c;smithi139=f;smithi139=i;smithi139=l;count:12 2023-07-14T19:28:30.442 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:28:30 smithi101 ceph-mon[128807]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' 2023-07-14T19:28:30.442 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:28:30 smithi101 ceph-mon[128807]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-14T19:28:30.442 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:28:30 smithi101 ceph-mon[128807]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T19:28:30.443 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:28:30 smithi101 ceph-mon[128807]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-14T19:28:30.443 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:28:30 smithi101 ceph-mon[128807]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' 2023-07-14T19:28:30.443 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:28:30 smithi101 ceph-mon[128807]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-07-14T19:28:30.443 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:28:30 smithi101 ceph-mon[128807]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-07-14T19:28:30.443 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:28:30 smithi101 ceph-mon[128807]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T19:28:30.443 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:28:30 smithi101 ceph-mon[128807]: Deploying daemon mds.l on smithi139 2023-07-14T19:28:30.443 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:28:30 smithi101 ceph-mon[128807]: pgmap v233: 1 pgs: 1 active+clean; 577 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-14T19:28:30.459 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:28:30 smithi139 ceph-mon[129293]: from='client.24406 -' entity='client.admin' cmd=[{"prefix": "orch apply mds", "fs_name": "all", "placement": "12;smithi098=a;smithi098=d;smithi098=g;smithi098=j;smithi101=b;smithi101=e;smithi101=h;smithi101=k;smithi139=c;smithi139=f;smithi139=i;smithi139=l", "target": ["mon-mgr", ""]}]: dispatch 2023-07-14T19:28:30.459 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:28:30 smithi139 ceph-mon[129293]: Saving service mds.all spec with placement smithi098=a;smithi098=d;smithi098=g;smithi098=j;smithi101=b;smithi101=e;smithi101=h;smithi101=k;smithi139=c;smithi139=f;smithi139=i;smithi139=l;count:12 2023-07-14T19:28:30.459 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:28:30 smithi139 ceph-mon[129293]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' 2023-07-14T19:28:30.460 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:28:30 smithi139 ceph-mon[129293]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-14T19:28:30.460 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:28:30 smithi139 ceph-mon[129293]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T19:28:30.460 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:28:30 smithi139 ceph-mon[129293]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-14T19:28:30.460 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:28:30 smithi139 ceph-mon[129293]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' 2023-07-14T19:28:30.460 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:28:30 smithi139 ceph-mon[129293]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-07-14T19:28:30.460 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:28:30 smithi139 ceph-mon[129293]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-07-14T19:28:30.460 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:28:30 smithi139 ceph-mon[129293]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T19:28:30.461 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:28:30 smithi139 ceph-mon[129293]: Deploying daemon mds.l on smithi139 2023-07-14T19:28:30.461 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:28:30 smithi139 ceph-mon[129293]: pgmap v233: 1 pgs: 1 active+clean; 577 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-14T19:28:30.509 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:30.508+0000 7f35a8a49700 1 -- 172.21.15.98:0/3210300283 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f35a402ca70 con 0x7f35a40feb10 2023-07-14T19:28:30.509 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:30.508+0000 7f35937fe700 1 -- 172.21.15.98:0/3210300283 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v79) v1 ==== 93+0+10 (secure 0 0 0) 0x7f35980997d0 con 0x7f35a40feb10 2023-07-14T19:28:30.509 INFO:teuthology.orchestra.run.smithi098.stdout:pg_num: 1 2023-07-14T19:28:30.511 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:30.510+0000 7f35a8a49700 1 -- 172.21.15.98:0/3210300283 >> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] conn(0x7f358c0a1150 msgr2=0x7f358c0a3600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:28:30.511 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:30.510+0000 7f35a8a49700 1 --2- 172.21.15.98:0/3210300283 >> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] conn(0x7f358c0a1150 0x7f358c0a3600 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f3594006010 tx=0x7f359400b540 comp rx=0 tx=0).stop 2023-07-14T19:28:30.511 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:30.510+0000 7f35a8a49700 1 -- 172.21.15.98:0/3210300283 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f35a40feb10 msgr2=0x7f35a4068310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:28:30.511 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:30.510+0000 7f35a8a49700 1 --2- 172.21.15.98:0/3210300283 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f35a40feb10 0x7f35a4068310 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f359800c990 tx=0x7f359800cca0 comp rx=0 tx=0).stop 2023-07-14T19:28:30.511 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:30.510+0000 7f35a8a49700 1 -- 172.21.15.98:0/3210300283 shutdown_connections 2023-07-14T19:28:30.512 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:30.510+0000 7f35a8a49700 1 --2- 172.21.15.98:0/3210300283 >> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] conn(0x7f358c0a1150 0x7f358c0a3600 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:28:30.512 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:30.510+0000 7f35a8a49700 1 --2- 172.21.15.98:0/3210300283 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f35a4100690 0x7f35a419cc90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:28:30.512 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:30.510+0000 7f35a8a49700 1 --2- 172.21.15.98:0/3210300283 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f35a40feb10 0x7f35a4068310 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:28:30.512 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:30.510+0000 7f35a8a49700 1 --2- 172.21.15.98:0/3210300283 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f35a40ffd00 0x7f35a419a650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:28:30.512 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:30.510+0000 7f35a8a49700 1 -- 172.21.15.98:0/3210300283 >> 172.21.15.98:0/3210300283 conn(0x7f35a40fa130 msgr2=0x7f35a41031e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:28:30.512 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:30.510+0000 7f35a8a49700 1 -- 172.21.15.98:0/3210300283 shutdown_connections 2023-07-14T19:28:30.512 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:30.510+0000 7f35a8a49700 1 -- 172.21.15.98:0/3210300283 wait complete. 2023-07-14T19:28:30.522 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:28:30 smithi098 ceph-mon[127486]: Saving service mds.all spec with placement smithi098=a;smithi098=d;smithi098=g;smithi098=j;smithi101=b;smithi101=e;smithi101=h;smithi101=k;smithi139=c;smithi139=f;smithi139=i;smithi139=l;count:12 2023-07-14T19:28:30.523 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:28:30 smithi098 ceph-mon[127486]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' 2023-07-14T19:28:30.523 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:28:30 smithi098 ceph-mon[127486]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-14T19:28:30.523 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:28:30 smithi098 ceph-mon[127486]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T19:28:30.523 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:28:30 smithi098 ceph-mon[127486]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-14T19:28:30.523 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:28:30 smithi098 ceph-mon[127486]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' 2023-07-14T19:28:30.523 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:28:30 smithi098 ceph-mon[127486]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-07-14T19:28:30.523 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:28:30 smithi098 ceph-mon[127486]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-07-14T19:28:30.524 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:28:30 smithi098 ceph-mon[127486]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T19:28:30.524 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:28:30 smithi098 ceph-mon[127486]: Deploying daemon mds.l on smithi139 2023-07-14T19:28:30.524 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:28:30 smithi098 ceph-mon[127486]: pgmap v233: 1 pgs: 1 active+clean; 577 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-14T19:28:30.525 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-07-14T19:28:30.648 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:30.645+0000 7fe8325da700 1 -- 172.21.15.98:0/1962467621 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe82c0feb00 msgr2=0x7fe82c0fef10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:28:30.648 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:30.645+0000 7fe8325da700 1 --2- 172.21.15.98:0/1962467621 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe82c0feb00 0x7fe82c0fef10 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7fe81c009990 tx=0x7fe81c009ca0 comp rx=0 tx=0).stop 2023-07-14T19:28:30.648 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:30.645+0000 7fe8325da700 1 -- 172.21.15.98:0/1962467621 shutdown_connections 2023-07-14T19:28:30.649 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:30.645+0000 7fe8325da700 1 --2- 172.21.15.98:0/1962467621 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fe82c100680 0x7fe82c106f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:28:30.649 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:30.645+0000 7fe8325da700 1 --2- 172.21.15.98:0/1962467621 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe82c0feb00 0x7fe82c0fef10 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:28:30.649 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:30.645+0000 7fe8325da700 1 --2- 172.21.15.98:0/1962467621 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7fe82c0ffcf0 0x7fe82c100140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:28:30.649 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:30.645+0000 7fe8325da700 1 -- 172.21.15.98:0/1962467621 >> 172.21.15.98:0/1962467621 conn(0x7fe82c0fa100 msgr2=0x7fe82c0fc550 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:28:30.649 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:30.645+0000 7fe8325da700 1 -- 172.21.15.98:0/1962467621 shutdown_connections 2023-07-14T19:28:30.649 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:30.645+0000 7fe8325da700 1 -- 172.21.15.98:0/1962467621 wait complete. 2023-07-14T19:28:30.649 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:30.646+0000 7fe8325da700 1 Processor -- start 2023-07-14T19:28:30.650 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:30.646+0000 7fe8325da700 1 -- start start 2023-07-14T19:28:30.650 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:30.646+0000 7fe8325da700 1 --2- >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7fe82c0feb00 0x7fe82c195970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:28:30.650 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:30.646+0000 7fe8325da700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe82c0ffcf0 0x7fe82c195eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:28:30.650 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:30.646+0000 7fe8325da700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fe82c100680 0x7fe82c19cf20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:28:30.650 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:30.646+0000 7fe8325da700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe82c19d4f0 con 0x7fe82c0ffcf0 2023-07-14T19:28:30.650 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:30.646+0000 7fe8325da700 1 -- --> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe82c19d630 con 0x7fe82c0feb00 2023-07-14T19:28:30.650 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:30.646+0000 7fe8325da700 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe82c19d7c0 con 0x7fe82c100680 2023-07-14T19:28:30.650 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:30.646+0000 7fe82b7fe700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe82c0ffcf0 0x7fe82c195eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:28:30.651 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:30.646+0000 7fe82b7fe700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe82c0ffcf0 0x7fe82c195eb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:45354/0 (socket says 172.21.15.98:45354) 2023-07-14T19:28:30.651 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:30.646+0000 7fe82b7fe700 1 -- 172.21.15.98:0/1565398591 learned_addr learned my addr 172.21.15.98:0/1565398591 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-07-14T19:28:30.651 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:30.646+0000 7fe830b77700 1 --2- 172.21.15.98:0/1565398591 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fe82c100680 0x7fe82c19cf20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:28:30.651 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:30.646+0000 7fe82bfff700 1 --2- 172.21.15.98:0/1565398591 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7fe82c0feb00 0x7fe82c195970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:28:30.651 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:30.646+0000 7fe82b7fe700 1 -- 172.21.15.98:0/1565398591 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fe82c100680 msgr2=0x7fe82c19cf20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:28:30.651 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:30.646+0000 7fe82b7fe700 1 --2- 172.21.15.98:0/1565398591 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fe82c100680 0x7fe82c19cf20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:28:30.651 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:30.646+0000 7fe82b7fe700 1 -- 172.21.15.98:0/1565398591 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7fe82c0feb00 msgr2=0x7fe82c195970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:28:30.652 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:30.646+0000 7fe82b7fe700 1 --2- 172.21.15.98:0/1565398591 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7fe82c0feb00 0x7fe82c195970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:28:30.652 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:30.646+0000 7fe82b7fe700 1 -- 172.21.15.98:0/1565398591 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe81c0095f0 con 0x7fe82c0ffcf0 2023-07-14T19:28:30.652 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:30.647+0000 7fe82bfff700 1 --2- 172.21.15.98:0/1565398591 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7fe82c0feb00 0x7fe82c195970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-07-14T19:28:30.652 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:30.647+0000 7fe830b77700 1 --2- 172.21.15.98:0/1565398591 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fe82c100680 0x7fe82c19cf20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-07-14T19:28:30.652 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:30.647+0000 7fe82b7fe700 1 --2- 172.21.15.98:0/1565398591 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe82c0ffcf0 0x7fe82c195eb0 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7fe82000d950 tx=0x7fe82000dd10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:28:30.653 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:30.647+0000 7fe8297fa700 1 -- 172.21.15.98:0/1565398591 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe820009b40 con 0x7fe82c0ffcf0 2023-07-14T19:28:30.653 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:30.647+0000 7fe8297fa700 1 -- 172.21.15.98:0/1565398591 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fe820009ca0 con 0x7fe82c0ffcf0 2023-07-14T19:28:30.653 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:30.647+0000 7fe8325da700 1 -- 172.21.15.98:0/1565398591 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe82c19daa0 con 0x7fe82c0ffcf0 2023-07-14T19:28:30.653 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:30.647+0000 7fe8297fa700 1 -- 172.21.15.98:0/1565398591 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe82000f620 con 0x7fe82c0ffcf0 2023-07-14T19:28:30.653 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:30.647+0000 7fe8325da700 1 -- 172.21.15.98:0/1565398591 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe82c19e000 con 0x7fe82c0ffcf0 2023-07-14T19:28:30.655 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:30.648+0000 7fe8325da700 1 -- 172.21.15.98:0/1565398591 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe82c10b4a0 con 0x7fe82c0ffcf0 2023-07-14T19:28:30.655 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:30.649+0000 7fe8297fa700 1 -- 172.21.15.98:0/1565398591 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 16) v1 ==== 135715+0+0 (secure 0 0 0) 0x7fe820010460 con 0x7fe82c0ffcf0 2023-07-14T19:28:30.656 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:30.649+0000 7fe8297fa700 1 --2- 172.21.15.98:0/1565398591 >> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] conn(0x7fe8140a1220 0x7fe8140a36d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:28:30.656 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:30.650+0000 7fe82bfff700 1 --2- 172.21.15.98:0/1565398591 >> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] conn(0x7fe8140a1220 0x7fe8140a36d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:28:30.656 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:30.650+0000 7fe8297fa700 1 -- 172.21.15.98:0/1565398591 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(79..79 src has 1..79) v4 ==== 7127+0+0 (secure 0 0 0) 0x7fe8200cc750 con 0x7fe82c0ffcf0 2023-07-14T19:28:30.657 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:30.650+0000 7fe82bfff700 1 --2- 172.21.15.98:0/1565398591 >> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] conn(0x7fe8140a1220 0x7fe8140a36d0 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7fe81c006010 tx=0x7fe81c00b540 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:28:30.657 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:30.654+0000 7fe8297fa700 1 -- 172.21.15.98:0/1565398591 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7fe820096340 con 0x7fe82c0ffcf0 2023-07-14T19:28:30.720 INFO:journalctl@ceph.mds.l.smithi139.stdout:Jul 14 19:28:30 smithi139 systemd[1]: Starting Ceph mds.l for 9adea648-227b-11ee-9b33-001a4aab830c... 2023-07-14T19:28:30.885 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:30.885+0000 7fe8325da700 1 -- 172.21.15.98:0/1565398591 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fe82c105990 con 0x7fe82c0ffcf0 2023-07-14T19:28:30.886 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:30.885+0000 7fe8297fa700 1 -- 172.21.15.98:0/1565398591 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 1 v1) v1 ==== 93+0+501 (secure 0 0 0) 0x7fe820099960 con 0x7fe82c0ffcf0 2023-07-14T19:28:30.886 INFO:teuthology.orchestra.run.smithi098.stdout: 2023-07-14T19:28:30.886 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":1,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[]} 2023-07-14T19:28:30.888 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:30.887+0000 7fe8325da700 1 -- 172.21.15.98:0/1565398591 >> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] conn(0x7fe8140a1220 msgr2=0x7fe8140a36d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:28:30.888 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:30.887+0000 7fe8325da700 1 --2- 172.21.15.98:0/1565398591 >> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] conn(0x7fe8140a1220 0x7fe8140a36d0 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7fe81c006010 tx=0x7fe81c00b540 comp rx=0 tx=0).stop 2023-07-14T19:28:30.888 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:30.887+0000 7fe8325da700 1 -- 172.21.15.98:0/1565398591 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe82c0ffcf0 msgr2=0x7fe82c195eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:28:30.888 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:30.887+0000 7fe8325da700 1 --2- 172.21.15.98:0/1565398591 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe82c0ffcf0 0x7fe82c195eb0 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7fe82000d950 tx=0x7fe82000dd10 comp rx=0 tx=0).stop 2023-07-14T19:28:30.888 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:30.887+0000 7fe8325da700 1 -- 172.21.15.98:0/1565398591 shutdown_connections 2023-07-14T19:28:30.889 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:30.887+0000 7fe8325da700 1 --2- 172.21.15.98:0/1565398591 >> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] conn(0x7fe8140a1220 0x7fe8140a36d0 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:28:30.889 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:30.887+0000 7fe8325da700 1 --2- 172.21.15.98:0/1565398591 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fe82c100680 0x7fe82c19cf20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:28:30.889 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:30.887+0000 7fe8325da700 1 --2- 172.21.15.98:0/1565398591 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7fe82c0feb00 0x7fe82c195970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:28:30.889 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:30.887+0000 7fe8325da700 1 --2- 172.21.15.98:0/1565398591 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe82c0ffcf0 0x7fe82c195eb0 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:28:30.889 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:30.887+0000 7fe8325da700 1 -- 172.21.15.98:0/1565398591 >> 172.21.15.98:0/1565398591 conn(0x7fe82c0fa100 msgr2=0x7fe82c0fc330 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:28:30.889 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:30.887+0000 7fe8325da700 1 -- 172.21.15.98:0/1565398591 shutdown_connections 2023-07-14T19:28:30.889 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:30.887+0000 7fe8325da700 1 -- 172.21.15.98:0/1565398591 wait complete. 2023-07-14T19:28:30.890 INFO:teuthology.orchestra.run.smithi098.stderr:dumped fsmap epoch 1 2023-07-14T19:28:31.287 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:28:31 smithi139 ceph-mon[129293]: from='client.? 172.21.15.98:0/240399753' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-07-14T19:28:31.287 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:28:31 smithi139 ceph-mon[129293]: from='client.? 172.21.15.98:0/3210300283' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2023-07-14T19:28:31.288 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:28:31 smithi139 ceph-mon[129293]: from='client.? 172.21.15.98:0/1565398591' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-07-14T19:28:31.288 INFO:journalctl@ceph.mds.l.smithi139.stdout:Jul 14 19:28:30 smithi139 podman[154503]: 2023-07-14T19:28:31.288 INFO:journalctl@ceph.mds.l.smithi139.stdout:Jul 14 19:28:31 smithi139 ceph-mds[154543]: -- 172.21.15.139:0/3208810175 <== mon.1 v2:172.21.15.139:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55805a3211e0 con 0x55805a2f6c00 2023-07-14T19:28:31.289 INFO:journalctl@ceph.mds.l.smithi139.stdout:Jul 14 19:28:31 smithi139 bash[154503]: 44462b89c5cd36156d2a6d8ca1d494b117f3303804c789a0c7c06be021ee3534 2023-07-14T19:28:31.289 INFO:journalctl@ceph.mds.l.smithi139.stdout:Jul 14 19:28:31 smithi139 ceph-9adea648-227b-11ee-9b33-001a4aab830c-mds-l[154519]: starting mds.l at 2023-07-14T19:28:31.441 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:28:31 smithi101 ceph-mon[128807]: from='client.? 172.21.15.98:0/240399753' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-07-14T19:28:31.441 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:28:31 smithi101 ceph-mon[128807]: from='client.? 172.21.15.98:0/3210300283' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2023-07-14T19:28:31.442 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:28:31 smithi101 ceph-mon[128807]: from='client.? 172.21.15.98:0/1565398591' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-07-14T19:28:31.531 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:28:31 smithi098 ceph-mon[127486]: from='client.? 172.21.15.98:0/240399753' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-07-14T19:28:31.531 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:28:31 smithi098 ceph-mon[127486]: from='client.? 172.21.15.98:0/3210300283' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2023-07-14T19:28:31.531 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:28:31 smithi098 ceph-mon[127486]: from='client.? 172.21.15.98:0/1565398591' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-07-14T19:28:31.720 INFO:journalctl@ceph.mds.l.smithi139.stdout:Jul 14 19:28:31 smithi139 systemd[1]: Started Ceph mds.l for 9adea648-227b-11ee-9b33-001a4aab830c. 2023-07-14T19:28:32.363 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:28:32 smithi101 ceph-mon[128807]: mds.? [v2:172.21.15.139:6832/3505477820,v1:172.21.15.139:6833/3505477820] up:boot 2023-07-14T19:28:32.363 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:28:32 smithi101 ceph-mon[128807]: fsmap 1 up:standby 2023-07-14T19:28:32.363 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:28:32 smithi101 ceph-mon[128807]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2023-07-14T19:28:32.364 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:28:32 smithi101 ceph-mon[128807]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' 2023-07-14T19:28:32.364 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:28:32 smithi101 ceph-mon[128807]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' 2023-07-14T19:28:32.364 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:28:32 smithi101 ceph-mon[128807]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' 2023-07-14T19:28:32.364 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:28:32 smithi101 ceph-mon[128807]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-07-14T19:28:32.364 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:28:32 smithi101 ceph-mon[128807]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-07-14T19:28:32.364 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:28:32 smithi101 ceph-mon[128807]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T19:28:32.364 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:28:32 smithi101 ceph-mon[128807]: Deploying daemon mds.k on smithi101 2023-07-14T19:28:32.365 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:28:32 smithi101 ceph-mon[128807]: pgmap v234: 1 pgs: 1 active+clean; 577 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-14T19:28:32.531 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:28:32 smithi098 ceph-mon[127486]: mds.? [v2:172.21.15.139:6832/3505477820,v1:172.21.15.139:6833/3505477820] up:boot 2023-07-14T19:28:32.531 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:28:32 smithi098 ceph-mon[127486]: fsmap 1 up:standby 2023-07-14T19:28:32.532 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:28:32 smithi098 ceph-mon[127486]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2023-07-14T19:28:32.532 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:28:32 smithi098 ceph-mon[127486]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' 2023-07-14T19:28:32.532 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:28:32 smithi098 ceph-mon[127486]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' 2023-07-14T19:28:32.532 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:28:32 smithi098 ceph-mon[127486]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' 2023-07-14T19:28:32.532 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:28:32 smithi098 ceph-mon[127486]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-07-14T19:28:32.532 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:28:32 smithi098 ceph-mon[127486]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-07-14T19:28:32.533 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:28:32 smithi098 ceph-mon[127486]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T19:28:32.533 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:28:32 smithi098 ceph-mon[127486]: Deploying daemon mds.k on smithi101 2023-07-14T19:28:32.533 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:28:32 smithi098 ceph-mon[127486]: pgmap v234: 1 pgs: 1 active+clean; 577 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-14T19:28:32.721 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:28:32 smithi139 ceph-mon[129293]: mds.? [v2:172.21.15.139:6832/3505477820,v1:172.21.15.139:6833/3505477820] up:boot 2023-07-14T19:28:32.721 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:28:32 smithi139 ceph-mon[129293]: fsmap 1 up:standby 2023-07-14T19:28:32.722 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:28:32 smithi139 ceph-mon[129293]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2023-07-14T19:28:32.722 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:28:32 smithi139 ceph-mon[129293]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' 2023-07-14T19:28:32.722 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:28:32 smithi139 ceph-mon[129293]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' 2023-07-14T19:28:32.722 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:28:32 smithi139 ceph-mon[129293]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' 2023-07-14T19:28:32.723 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:28:32 smithi139 ceph-mon[129293]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-07-14T19:28:32.723 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:28:32 smithi139 ceph-mon[129293]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-07-14T19:28:32.723 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:28:32 smithi139 ceph-mon[129293]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T19:28:32.724 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:28:32 smithi139 ceph-mon[129293]: Deploying daemon mds.k on smithi101 2023-07-14T19:28:32.724 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:28:32 smithi139 ceph-mon[129293]: pgmap v234: 1 pgs: 1 active+clean; 577 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-14T19:28:32.901 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-07-14T19:28:33.040 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:33.036+0000 7fc852f86700 1 -- 172.21.15.98:0/3660471452 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc84c0feb30 msgr2=0x7fc84c0fef40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:28:33.040 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:33.036+0000 7fc852f86700 1 --2- 172.21.15.98:0/3660471452 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc84c0feb30 0x7fc84c0fef40 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7fc840009990 tx=0x7fc840009ca0 comp rx=0 tx=0).stop 2023-07-14T19:28:33.040 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:33.036+0000 7fc852f86700 1 -- 172.21.15.98:0/3660471452 shutdown_connections 2023-07-14T19:28:33.041 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:33.036+0000 7fc852f86700 1 --2- 172.21.15.98:0/3660471452 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7fc84c1006b0 0x7fc84c106f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:28:33.041 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:33.036+0000 7fc852f86700 1 --2- 172.21.15.98:0/3660471452 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc84c0feb30 0x7fc84c0fef40 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:28:33.041 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:33.036+0000 7fc852f86700 1 --2- 172.21.15.98:0/3660471452 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fc84c0ffd20 0x7fc84c100170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:28:33.041 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:33.036+0000 7fc852f86700 1 -- 172.21.15.98:0/3660471452 >> 172.21.15.98:0/3660471452 conn(0x7fc84c0fa150 msgr2=0x7fc84c0fc580 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:28:33.042 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:33.036+0000 7fc852f86700 1 -- 172.21.15.98:0/3660471452 shutdown_connections 2023-07-14T19:28:33.042 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:33.036+0000 7fc852f86700 1 -- 172.21.15.98:0/3660471452 wait complete. 2023-07-14T19:28:33.042 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:33.037+0000 7fc852f86700 1 Processor -- start 2023-07-14T19:28:33.042 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:33.037+0000 7fc852f86700 1 -- start start 2023-07-14T19:28:33.043 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:33.037+0000 7fc852f86700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc84c0feb30 0x7fc84c195940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:28:33.043 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:33.037+0000 7fc852f86700 1 --2- >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7fc84c0ffd20 0x7fc84c195e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:28:33.043 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:33.037+0000 7fc852f86700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fc84c1006b0 0x7fc84c19cef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:28:33.044 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:33.037+0000 7fc852f86700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc84c19d4c0 con 0x7fc84c0feb30 2023-07-14T19:28:33.044 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:33.037+0000 7fc852f86700 1 -- --> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc84c19d600 con 0x7fc84c0ffd20 2023-07-14T19:28:33.044 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:33.037+0000 7fc852f86700 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc84c19d790 con 0x7fc84c1006b0 2023-07-14T19:28:33.044 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:33.037+0000 7fc850d22700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc84c0feb30 0x7fc84c195940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:28:33.045 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:33.037+0000 7fc850d22700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc84c0feb30 0x7fc84c195940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:45366/0 (socket says 172.21.15.98:45366) 2023-07-14T19:28:33.045 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:33.037+0000 7fc850d22700 1 -- 172.21.15.98:0/1780990938 learned_addr learned my addr 172.21.15.98:0/1780990938 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-07-14T19:28:33.045 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:33.037+0000 7fc84bfff700 1 --2- 172.21.15.98:0/1780990938 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7fc84c0ffd20 0x7fc84c195e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:28:33.046 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:33.037+0000 7fc851523700 1 --2- 172.21.15.98:0/1780990938 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fc84c1006b0 0x7fc84c19cef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:28:33.046 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:33.037+0000 7fc850d22700 1 -- 172.21.15.98:0/1780990938 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fc84c1006b0 msgr2=0x7fc84c19cef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:28:33.046 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:33.037+0000 7fc850d22700 1 --2- 172.21.15.98:0/1780990938 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fc84c1006b0 0x7fc84c19cef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:28:33.046 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:33.037+0000 7fc850d22700 1 -- 172.21.15.98:0/1780990938 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7fc84c0ffd20 msgr2=0x7fc84c195e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:28:33.047 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:33.037+0000 7fc850d22700 1 --2- 172.21.15.98:0/1780990938 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7fc84c0ffd20 0x7fc84c195e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:28:33.047 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:33.037+0000 7fc850d22700 1 -- 172.21.15.98:0/1780990938 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc8400095f0 con 0x7fc84c0feb30 2023-07-14T19:28:33.047 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:33.037+0000 7fc851523700 1 --2- 172.21.15.98:0/1780990938 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fc84c1006b0 0x7fc84c19cef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-07-14T19:28:33.047 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:33.038+0000 7fc850d22700 1 --2- 172.21.15.98:0/1780990938 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc84c0feb30 0x7fc84c195940 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7fc840000c00 tx=0x7fc8400047b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:28:33.048 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:33.038+0000 7fc849ffb700 1 -- 172.21.15.98:0/1780990938 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc84001d070 con 0x7fc84c0feb30 2023-07-14T19:28:33.048 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:33.038+0000 7fc852f86700 1 -- 172.21.15.98:0/1780990938 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc84c19da10 con 0x7fc84c0feb30 2023-07-14T19:28:33.049 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:33.038+0000 7fc849ffb700 1 -- 172.21.15.98:0/1780990938 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fc840004060 con 0x7fc84c0feb30 2023-07-14T19:28:33.049 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:33.038+0000 7fc849ffb700 1 -- 172.21.15.98:0/1780990938 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc840022d30 con 0x7fc84c0feb30 2023-07-14T19:28:33.049 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:33.038+0000 7fc852f86700 1 -- 172.21.15.98:0/1780990938 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc84c19dec0 con 0x7fc84c0feb30 2023-07-14T19:28:33.049 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:33.039+0000 7fc852f86700 1 -- 172.21.15.98:0/1780990938 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc84c10a300 con 0x7fc84c0feb30 2023-07-14T19:28:33.051 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:33.041+0000 7fc849ffb700 1 -- 172.21.15.98:0/1780990938 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 16) v1 ==== 135715+0+0 (secure 0 0 0) 0x7fc840003ba0 con 0x7fc84c0feb30 2023-07-14T19:28:33.051 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:33.041+0000 7fc849ffb700 1 --2- 172.21.15.98:0/1780990938 >> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] conn(0x7fc83c0a11a0 0x7fc83c0a3650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:28:33.051 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:33.041+0000 7fc84bfff700 1 --2- 172.21.15.98:0/1780990938 >> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] conn(0x7fc83c0a11a0 0x7fc83c0a3650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:28:33.051 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:33.041+0000 7fc849ffb700 1 -- 172.21.15.98:0/1780990938 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(79..79 src has 1..79) v4 ==== 7127+0+0 (secure 0 0 0) 0x7fc8400cdaf0 con 0x7fc84c0feb30 2023-07-14T19:28:33.052 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:33.042+0000 7fc84bfff700 1 --2- 172.21.15.98:0/1780990938 >> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] conn(0x7fc83c0a11a0 0x7fc83c0a3650 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7fc838005950 tx=0x7fc83800b410 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:28:33.052 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:33.044+0000 7fc849ffb700 1 -- 172.21.15.98:0/1780990938 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7fc840097770 con 0x7fc84c0feb30 2023-07-14T19:28:33.191 INFO:journalctl@ceph.mds.k.smithi101.stdout:Jul 14 19:28:32 smithi101 systemd[1]: Starting Ceph mds.k for 9adea648-227b-11ee-9b33-001a4aab830c... 2023-07-14T19:28:33.275 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:33.274+0000 7fc852f86700 1 -- 172.21.15.98:0/1780990938 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc84c10a4f0 con 0x7fc84c0feb30 2023-07-14T19:28:33.275 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:33.274+0000 7fc849ffb700 1 -- 172.21.15.98:0/1780990938 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 2 v2) v1 ==== 93+0+1260 (secure 0 0 0) 0x7fc84009ad90 con 0x7fc84c0feb30 2023-07-14T19:28:33.276 INFO:teuthology.orchestra.run.smithi098.stdout: 2023-07-14T19:28:33.276 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":2,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24421,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.139:6833/3505477820","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6832","nonce":3505477820},{"type":"v1","addr":"172.21.15.139:6833","nonce":3505477820}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2}],"filesystems":[]} 2023-07-14T19:28:33.277 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:33.276+0000 7fc852f86700 1 -- 172.21.15.98:0/1780990938 >> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] conn(0x7fc83c0a11a0 msgr2=0x7fc83c0a3650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:28:33.277 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:33.276+0000 7fc852f86700 1 --2- 172.21.15.98:0/1780990938 >> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] conn(0x7fc83c0a11a0 0x7fc83c0a3650 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7fc838005950 tx=0x7fc83800b410 comp rx=0 tx=0).stop 2023-07-14T19:28:33.277 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:33.276+0000 7fc852f86700 1 -- 172.21.15.98:0/1780990938 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc84c0feb30 msgr2=0x7fc84c195940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:28:33.278 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:33.276+0000 7fc852f86700 1 --2- 172.21.15.98:0/1780990938 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc84c0feb30 0x7fc84c195940 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7fc840000c00 tx=0x7fc8400047b0 comp rx=0 tx=0).stop 2023-07-14T19:28:33.278 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:33.276+0000 7fc852f86700 1 -- 172.21.15.98:0/1780990938 shutdown_connections 2023-07-14T19:28:33.278 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:33.276+0000 7fc852f86700 1 --2- 172.21.15.98:0/1780990938 >> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] conn(0x7fc83c0a11a0 0x7fc83c0a3650 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:28:33.278 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:33.276+0000 7fc852f86700 1 --2- 172.21.15.98:0/1780990938 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fc84c1006b0 0x7fc84c19cef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:28:33.278 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:33.276+0000 7fc852f86700 1 --2- 172.21.15.98:0/1780990938 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc84c0feb30 0x7fc84c195940 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:28:33.278 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:33.276+0000 7fc852f86700 1 --2- 172.21.15.98:0/1780990938 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7fc84c0ffd20 0x7fc84c195e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:28:33.278 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:33.276+0000 7fc852f86700 1 -- 172.21.15.98:0/1780990938 >> 172.21.15.98:0/1780990938 conn(0x7fc84c0fa150 msgr2=0x7fc84c0fa9c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:28:33.279 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:33.276+0000 7fc852f86700 1 -- 172.21.15.98:0/1780990938 shutdown_connections 2023-07-14T19:28:33.279 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:33.276+0000 7fc852f86700 1 -- 172.21.15.98:0/1780990938 wait complete. 2023-07-14T19:28:33.279 INFO:teuthology.orchestra.run.smithi098.stderr:dumped fsmap epoch 2 2023-07-14T19:28:33.514 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:28:33 smithi101 ceph-mon[128807]: from='client.? 172.21.15.98:0/1780990938' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-07-14T19:28:33.514 INFO:journalctl@ceph.mds.k.smithi101.stdout:Jul 14 19:28:33 smithi101 podman[153023]: 2023-07-14T19:28:33.720 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:28:33 smithi139 ceph-mon[129293]: from='client.? 172.21.15.98:0/1780990938' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-07-14T19:28:33.781 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:28:33 smithi098 ceph-mon[127486]: from='client.? 172.21.15.98:0/1780990938' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-07-14T19:28:33.941 INFO:journalctl@ceph.mds.k.smithi101.stdout:Jul 14 19:28:33 smithi101 bash[153023]: d9f2148b19933ae947c8e3b17f1eacbe0914d773ac595772a07a4ce0bbdafae1 2023-07-14T19:28:33.941 INFO:journalctl@ceph.mds.k.smithi101.stdout:Jul 14 19:28:33 smithi101 ceph-mds[153063]: -- 172.21.15.101:0/1801466861 <== mon.2 v2:172.21.15.101:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x56238515f1e0 con 0x562385135000 2023-07-14T19:28:33.942 INFO:journalctl@ceph.mds.k.smithi101.stdout:Jul 14 19:28:33 smithi101 ceph-9adea648-227b-11ee-9b33-001a4aab830c-mds-k[153039]: starting mds.k at 2023-07-14T19:28:33.942 INFO:journalctl@ceph.mds.k.smithi101.stdout:Jul 14 19:28:33 smithi101 systemd[1]: Started Ceph mds.k for 9adea648-227b-11ee-9b33-001a4aab830c. 2023-07-14T19:28:34.934 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:28:34 smithi098 ceph-mon[127486]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' 2023-07-14T19:28:34.934 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:28:34 smithi098 ceph-mon[127486]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' 2023-07-14T19:28:34.935 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:28:34 smithi098 ceph-mon[127486]: pgmap v235: 1 pgs: 1 active+clean; 577 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-14T19:28:34.935 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:28:34 smithi098 ceph-mon[127486]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' 2023-07-14T19:28:34.935 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:28:34 smithi098 ceph-mon[127486]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-07-14T19:28:34.935 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:28:34 smithi098 ceph-mon[127486]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-07-14T19:28:34.936 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:28:34 smithi098 ceph-mon[127486]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T19:28:34.936 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:28:34 smithi098 ceph-mon[127486]: Deploying daemon mds.j on smithi098 2023-07-14T19:28:34.936 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:28:34 smithi098 ceph-mon[127486]: mds.? [v2:172.21.15.101:6832/3474460171,v1:172.21.15.101:6833/3474460171] up:boot 2023-07-14T19:28:34.937 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:28:34 smithi098 ceph-mon[127486]: fsmap 2 up:standby 2023-07-14T19:28:34.937 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:28:34 smithi098 ceph-mon[127486]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2023-07-14T19:28:34.941 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:28:34 smithi101 ceph-mon[128807]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' 2023-07-14T19:28:34.942 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:28:34 smithi101 ceph-mon[128807]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' 2023-07-14T19:28:34.942 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:28:34 smithi101 ceph-mon[128807]: pgmap v235: 1 pgs: 1 active+clean; 577 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-14T19:28:34.942 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:28:34 smithi101 ceph-mon[128807]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' 2023-07-14T19:28:34.942 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:28:34 smithi101 ceph-mon[128807]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-07-14T19:28:34.942 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:28:34 smithi101 ceph-mon[128807]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-07-14T19:28:34.943 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:28:34 smithi101 ceph-mon[128807]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T19:28:34.943 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:28:34 smithi101 ceph-mon[128807]: Deploying daemon mds.j on smithi098 2023-07-14T19:28:34.943 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:28:34 smithi101 ceph-mon[128807]: mds.? [v2:172.21.15.101:6832/3474460171,v1:172.21.15.101:6833/3474460171] up:boot 2023-07-14T19:28:34.944 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:28:34 smithi101 ceph-mon[128807]: fsmap 2 up:standby 2023-07-14T19:28:34.944 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:28:34 smithi101 ceph-mon[128807]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2023-07-14T19:28:34.970 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:28:34 smithi139 ceph-mon[129293]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' 2023-07-14T19:28:34.970 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:28:34 smithi139 ceph-mon[129293]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' 2023-07-14T19:28:34.971 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:28:34 smithi139 ceph-mon[129293]: pgmap v235: 1 pgs: 1 active+clean; 577 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-14T19:28:34.971 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:28:34 smithi139 ceph-mon[129293]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' 2023-07-14T19:28:34.971 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:28:34 smithi139 ceph-mon[129293]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-07-14T19:28:34.971 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:28:34 smithi139 ceph-mon[129293]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-07-14T19:28:34.971 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:28:34 smithi139 ceph-mon[129293]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T19:28:34.971 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:28:34 smithi139 ceph-mon[129293]: Deploying daemon mds.j on smithi098 2023-07-14T19:28:34.971 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:28:34 smithi139 ceph-mon[129293]: mds.? [v2:172.21.15.101:6832/3474460171,v1:172.21.15.101:6833/3474460171] up:boot 2023-07-14T19:28:34.972 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:28:34 smithi139 ceph-mon[129293]: fsmap 2 up:standby 2023-07-14T19:28:34.972 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:28:34 smithi139 ceph-mon[129293]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2023-07-14T19:28:35.281 INFO:journalctl@ceph.mds.j.smithi098.stdout:Jul 14 19:28:35 smithi098 systemd[1]: Starting Ceph mds.j for 9adea648-227b-11ee-9b33-001a4aab830c... 2023-07-14T19:28:35.290 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-07-14T19:28:35.428 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:35.424+0000 7ff9ddde2700 1 -- 172.21.15.98:0/104370360 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff9d80feb00 msgr2=0x7ff9d80fef10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:28:35.429 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:35.424+0000 7ff9ddde2700 1 --2- 172.21.15.98:0/104370360 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff9d80feb00 0x7ff9d80fef10 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7ff9c8009990 tx=0x7ff9c8009ca0 comp rx=0 tx=0).stop 2023-07-14T19:28:35.429 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:35.424+0000 7ff9ddde2700 1 -- 172.21.15.98:0/104370360 shutdown_connections 2023-07-14T19:28:35.430 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:35.424+0000 7ff9ddde2700 1 --2- 172.21.15.98:0/104370360 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7ff9d8100680 0x7ff9d8106f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:28:35.430 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:35.424+0000 7ff9ddde2700 1 --2- 172.21.15.98:0/104370360 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff9d80feb00 0x7ff9d80fef10 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:28:35.430 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:35.424+0000 7ff9ddde2700 1 --2- 172.21.15.98:0/104370360 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7ff9d80ffcf0 0x7ff9d8100140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:28:35.430 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:35.424+0000 7ff9ddde2700 1 -- 172.21.15.98:0/104370360 >> 172.21.15.98:0/104370360 conn(0x7ff9d80fa100 msgr2=0x7ff9d80fc550 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:28:35.430 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:35.424+0000 7ff9ddde2700 1 -- 172.21.15.98:0/104370360 shutdown_connections 2023-07-14T19:28:35.430 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:35.424+0000 7ff9ddde2700 1 -- 172.21.15.98:0/104370360 wait complete. 2023-07-14T19:28:35.430 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:35.424+0000 7ff9ddde2700 1 Processor -- start 2023-07-14T19:28:35.431 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:35.425+0000 7ff9ddde2700 1 -- start start 2023-07-14T19:28:35.431 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:35.425+0000 7ff9ddde2700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff9d80feb00 0x7ff9d8068370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:28:35.431 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:35.425+0000 7ff9ddde2700 1 --2- >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7ff9d80ffcf0 0x7ff9d819c990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:28:35.431 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:35.425+0000 7ff9ddde2700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7ff9d8100680 0x7ff9d819eee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:28:35.431 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:35.425+0000 7ff9ddde2700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9d819f5c0 con 0x7ff9d80feb00 2023-07-14T19:28:35.431 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:35.425+0000 7ff9ddde2700 1 -- --> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9d819f730 con 0x7ff9d80ffcf0 2023-07-14T19:28:35.431 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:35.425+0000 7ff9ddde2700 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9d819f8f0 con 0x7ff9d8100680 2023-07-14T19:28:35.432 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:35.425+0000 7ff9d77fe700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff9d80feb00 0x7ff9d8068370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:28:35.432 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:35.425+0000 7ff9d77fe700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff9d80feb00 0x7ff9d8068370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:45380/0 (socket says 172.21.15.98:45380) 2023-07-14T19:28:35.433 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:35.425+0000 7ff9d77fe700 1 -- 172.21.15.98:0/3639906861 learned_addr learned my addr 172.21.15.98:0/3639906861 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-07-14T19:28:35.434 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:35.425+0000 7ff9d7fff700 1 --2- 172.21.15.98:0/3639906861 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7ff9d8100680 0x7ff9d819eee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:28:35.434 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:35.425+0000 7ff9d6ffd700 1 --2- 172.21.15.98:0/3639906861 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7ff9d80ffcf0 0x7ff9d819c990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:28:35.434 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:35.425+0000 7ff9d77fe700 1 -- 172.21.15.98:0/3639906861 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7ff9d8100680 msgr2=0x7ff9d819eee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:28:35.434 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:35.425+0000 7ff9d77fe700 1 --2- 172.21.15.98:0/3639906861 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7ff9d8100680 0x7ff9d819eee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:28:35.434 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:35.425+0000 7ff9d77fe700 1 -- 172.21.15.98:0/3639906861 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7ff9d80ffcf0 msgr2=0x7ff9d819c990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:28:35.434 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:35.425+0000 7ff9d77fe700 1 --2- 172.21.15.98:0/3639906861 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7ff9d80ffcf0 0x7ff9d819c990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:28:35.434 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:35.425+0000 7ff9d77fe700 1 -- 172.21.15.98:0/3639906861 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff9c80095f0 con 0x7ff9d80feb00 2023-07-14T19:28:35.435 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:35.425+0000 7ff9d6ffd700 1 --2- 172.21.15.98:0/3639906861 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7ff9d80ffcf0 0x7ff9d819c990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-07-14T19:28:35.435 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:35.425+0000 7ff9d77fe700 1 --2- 172.21.15.98:0/3639906861 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff9d80feb00 0x7ff9d8068370 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7ff9c8004760 tx=0x7ff9c8004790 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:28:35.435 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:35.426+0000 7ff9d4ff9700 1 -- 172.21.15.98:0/3639906861 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff9c801d070 con 0x7ff9d80feb00 2023-07-14T19:28:35.435 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:35.426+0000 7ff9d4ff9700 1 -- 172.21.15.98:0/3639906861 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7ff9c80041d0 con 0x7ff9d80feb00 2023-07-14T19:28:35.435 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:35.426+0000 7ff9ddde2700 1 -- 172.21.15.98:0/3639906861 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff9d819fb70 con 0x7ff9d80feb00 2023-07-14T19:28:35.435 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:35.426+0000 7ff9d4ff9700 1 -- 172.21.15.98:0/3639906861 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff9c80053e0 con 0x7ff9d80feb00 2023-07-14T19:28:35.435 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:35.426+0000 7ff9ddde2700 1 -- 172.21.15.98:0/3639906861 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff9d819ff30 con 0x7ff9d80feb00 2023-07-14T19:28:35.436 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:35.427+0000 7ff9ddde2700 1 -- 172.21.15.98:0/3639906861 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff9d8105910 con 0x7ff9d80feb00 2023-07-14T19:28:35.436 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:35.428+0000 7ff9d4ff9700 1 -- 172.21.15.98:0/3639906861 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 16) v1 ==== 135715+0+0 (secure 0 0 0) 0x7ff9c8005540 con 0x7ff9d80feb00 2023-07-14T19:28:35.437 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:35.428+0000 7ff9d4ff9700 1 --2- 172.21.15.98:0/3639906861 >> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] conn(0x7ff9c00a11a0 0x7ff9c00a3650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:28:35.437 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:35.429+0000 7ff9d6ffd700 1 --2- 172.21.15.98:0/3639906861 >> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] conn(0x7ff9c00a11a0 0x7ff9c00a3650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:28:35.437 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:35.429+0000 7ff9d4ff9700 1 -- 172.21.15.98:0/3639906861 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(79..79 src has 1..79) v4 ==== 7127+0+0 (secure 0 0 0) 0x7ff9c80cdd60 con 0x7ff9d80feb00 2023-07-14T19:28:35.437 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:35.429+0000 7ff9d6ffd700 1 --2- 172.21.15.98:0/3639906861 >> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] conn(0x7ff9c00a11a0 0x7ff9c00a3650 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7ff9cc005fd0 tx=0x7ff9cc005e20 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:28:35.437 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:35.432+0000 7ff9d4ff9700 1 -- 172.21.15.98:0/3639906861 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7ff9c80979e0 con 0x7ff9d80feb00 2023-07-14T19:28:35.609 INFO:journalctl@ceph.mds.j.smithi098.stdout:Jul 14 19:28:35 smithi098 podman[160768]: 2023-07-14T19:28:35.675 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:35.675+0000 7ff9ddde2700 1 -- 172.21.15.98:0/3639906861 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff9d802c8e0 con 0x7ff9d80feb00 2023-07-14T19:28:35.676 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:35.675+0000 7ff9d4ff9700 1 -- 172.21.15.98:0/3639906861 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 3 v3) v1 ==== 93+0+2020 (secure 0 0 0) 0x7ff9c809b000 con 0x7ff9d80feb00 2023-07-14T19:28:35.676 INFO:teuthology.orchestra.run.smithi098.stdout: 2023-07-14T19:28:35.676 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":3,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24421,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.139:6833/3505477820","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6832","nonce":3505477820},{"type":"v1","addr":"172.21.15.139:6833","nonce":3505477820}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":24437,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.101:6833/3474460171","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.101:6832","nonce":3474460171},{"type":"v1","addr":"172.21.15.101:6833","nonce":3474460171}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3}],"filesystems":[]} 2023-07-14T19:28:35.678 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:35.677+0000 7ff9ddde2700 1 -- 172.21.15.98:0/3639906861 >> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] conn(0x7ff9c00a11a0 msgr2=0x7ff9c00a3650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:28:35.678 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:35.677+0000 7ff9ddde2700 1 --2- 172.21.15.98:0/3639906861 >> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] conn(0x7ff9c00a11a0 0x7ff9c00a3650 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7ff9cc005fd0 tx=0x7ff9cc005e20 comp rx=0 tx=0).stop 2023-07-14T19:28:35.678 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:35.677+0000 7ff9ddde2700 1 -- 172.21.15.98:0/3639906861 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff9d80feb00 msgr2=0x7ff9d8068370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:28:35.678 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:35.677+0000 7ff9ddde2700 1 --2- 172.21.15.98:0/3639906861 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff9d80feb00 0x7ff9d8068370 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7ff9c8004760 tx=0x7ff9c8004790 comp rx=0 tx=0).stop 2023-07-14T19:28:35.678 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:35.677+0000 7ff9ddde2700 1 -- 172.21.15.98:0/3639906861 shutdown_connections 2023-07-14T19:28:35.679 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:35.677+0000 7ff9ddde2700 1 --2- 172.21.15.98:0/3639906861 >> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] conn(0x7ff9c00a11a0 0x7ff9c00a3650 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:28:35.679 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:35.677+0000 7ff9ddde2700 1 --2- 172.21.15.98:0/3639906861 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7ff9d8100680 0x7ff9d819eee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:28:35.679 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:35.677+0000 7ff9ddde2700 1 --2- 172.21.15.98:0/3639906861 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff9d80feb00 0x7ff9d8068370 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:28:35.679 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:35.677+0000 7ff9ddde2700 1 --2- 172.21.15.98:0/3639906861 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7ff9d80ffcf0 0x7ff9d819c990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:28:35.679 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:35.677+0000 7ff9ddde2700 1 -- 172.21.15.98:0/3639906861 >> 172.21.15.98:0/3639906861 conn(0x7ff9d80fa100 msgr2=0x7ff9d80fc280 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:28:35.679 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:35.677+0000 7ff9ddde2700 1 -- 172.21.15.98:0/3639906861 shutdown_connections 2023-07-14T19:28:35.679 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:35.677+0000 7ff9ddde2700 1 -- 172.21.15.98:0/3639906861 wait complete. 2023-07-14T19:28:35.680 INFO:teuthology.orchestra.run.smithi098.stderr:dumped fsmap epoch 3 2023-07-14T19:28:36.032 INFO:journalctl@ceph.mds.j.smithi098.stdout:Jul 14 19:28:35 smithi098 ceph-mds[160834]: -- 172.21.15.98:0/2854661563 <== mon.0 v2:172.21.15.98:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x561f6e7751e0 con 0x561f6e74ac00 2023-07-14T19:28:36.032 INFO:journalctl@ceph.mds.j.smithi098.stdout:Jul 14 19:28:35 smithi098 ceph-9adea648-227b-11ee-9b33-001a4aab830c-mds-j[160808]: starting mds.j at 2023-07-14T19:28:36.032 INFO:journalctl@ceph.mds.j.smithi098.stdout:Jul 14 19:28:35 smithi098 bash[160768]: 76133ab42514ba52e47d4721a47939dfafc078af03d3a9aea70d74f0d086de18 2023-07-14T19:28:36.032 INFO:journalctl@ceph.mds.j.smithi098.stdout:Jul 14 19:28:35 smithi098 systemd[1]: Started Ceph mds.j for 9adea648-227b-11ee-9b33-001a4aab830c. 2023-07-14T19:28:36.741 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:28:36 smithi098 ceph-mon[127486]: pgmap v236: 1 pgs: 1 active+clean; 577 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-14T19:28:36.741 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:28:36 smithi098 ceph-mon[127486]: from='client.? 172.21.15.98:0/3639906861' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-07-14T19:28:36.742 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:28:36 smithi098 ceph-mon[127486]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' 2023-07-14T19:28:36.742 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:28:36 smithi098 ceph-mon[127486]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' 2023-07-14T19:28:36.742 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:28:36 smithi098 ceph-mon[127486]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' 2023-07-14T19:28:36.742 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:28:36 smithi098 ceph-mon[127486]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-07-14T19:28:36.742 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:28:36 smithi098 ceph-mon[127486]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-07-14T19:28:36.742 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:28:36 smithi098 ceph-mon[127486]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T19:28:36.742 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:28:36 smithi098 ceph-mon[127486]: Deploying daemon mds.d on smithi098 2023-07-14T19:28:36.941 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:28:36 smithi101 ceph-mon[128807]: pgmap v236: 1 pgs: 1 active+clean; 577 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-14T19:28:36.942 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:28:36 smithi101 ceph-mon[128807]: from='client.? 172.21.15.98:0/3639906861' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-07-14T19:28:36.942 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:28:36 smithi101 ceph-mon[128807]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' 2023-07-14T19:28:36.942 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:28:36 smithi101 ceph-mon[128807]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' 2023-07-14T19:28:36.942 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:28:36 smithi101 ceph-mon[128807]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' 2023-07-14T19:28:36.943 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:28:36 smithi101 ceph-mon[128807]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-07-14T19:28:36.943 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:28:36 smithi101 ceph-mon[128807]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-07-14T19:28:36.943 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:28:36 smithi101 ceph-mon[128807]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T19:28:36.944 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:28:36 smithi101 ceph-mon[128807]: Deploying daemon mds.d on smithi098 2023-07-14T19:28:36.970 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:28:36 smithi139 ceph-mon[129293]: pgmap v236: 1 pgs: 1 active+clean; 577 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-14T19:28:36.970 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:28:36 smithi139 ceph-mon[129293]: from='client.? 172.21.15.98:0/3639906861' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-07-14T19:28:36.971 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:28:36 smithi139 ceph-mon[129293]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' 2023-07-14T19:28:36.971 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:28:36 smithi139 ceph-mon[129293]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' 2023-07-14T19:28:36.971 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:28:36 smithi139 ceph-mon[129293]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' 2023-07-14T19:28:36.971 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:28:36 smithi139 ceph-mon[129293]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-07-14T19:28:36.972 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:28:36 smithi139 ceph-mon[129293]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-07-14T19:28:36.972 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:28:36 smithi139 ceph-mon[129293]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T19:28:36.972 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:28:36 smithi139 ceph-mon[129293]: Deploying daemon mds.d on smithi098 2023-07-14T19:28:37.531 INFO:journalctl@ceph.mds.d.smithi098.stdout:Jul 14 19:28:37 smithi098 systemd[1]: Starting Ceph mds.d for 9adea648-227b-11ee-9b33-001a4aab830c... 2023-07-14T19:28:37.691 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-07-14T19:28:37.823 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:37.823+0000 7f7c0bc19700 1 -- 172.21.15.98:0/3635692585 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7c040feb30 msgr2=0x7f7c040fef40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:28:37.824 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:37.823+0000 7f7c0bc19700 1 --2- 172.21.15.98:0/3635692585 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7c040feb30 0x7f7c040fef40 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f7c00009990 tx=0x7f7c00009ca0 comp rx=0 tx=0).stop 2023-07-14T19:28:37.824 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:37.823+0000 7f7c0bc19700 1 -- 172.21.15.98:0/3635692585 shutdown_connections 2023-07-14T19:28:37.824 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:37.823+0000 7f7c0bc19700 1 --2- 172.21.15.98:0/3635692585 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f7c041006b0 0x7f7c04106f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:28:37.824 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:37.823+0000 7f7c0bc19700 1 --2- 172.21.15.98:0/3635692585 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7c040feb30 0x7f7c040fef40 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:28:37.824 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:37.823+0000 7f7c0bc19700 1 --2- 172.21.15.98:0/3635692585 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f7c040ffd20 0x7f7c04100170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:28:37.824 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:37.823+0000 7f7c0bc19700 1 -- 172.21.15.98:0/3635692585 >> 172.21.15.98:0/3635692585 conn(0x7f7c040fa110 msgr2=0x7f7c040fc580 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:28:37.825 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:37.823+0000 7f7c0bc19700 1 -- 172.21.15.98:0/3635692585 shutdown_connections 2023-07-14T19:28:37.825 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:37.823+0000 7f7c0bc19700 1 -- 172.21.15.98:0/3635692585 wait complete. 2023-07-14T19:28:37.827 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:37.823+0000 7f7c0bc19700 1 Processor -- start 2023-07-14T19:28:37.827 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:37.823+0000 7f7c0bc19700 1 -- start start 2023-07-14T19:28:37.827 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:37.823+0000 7f7c0bc19700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7c040feb30 0x7f7c04069320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:28:37.827 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:37.824+0000 7f7c0bc19700 1 --2- >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f7c040ffd20 0x7f7c0419ca50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:28:37.828 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:37.824+0000 7f7c0bc19700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f7c041006b0 0x7f7c0419efa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:28:37.828 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:37.824+0000 7f7c0bc19700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7c0419f650 con 0x7f7c040feb30 2023-07-14T19:28:37.828 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:37.824+0000 7f7c0bc19700 1 -- --> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7c0419f7c0 con 0x7f7c040ffd20 2023-07-14T19:28:37.828 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:37.824+0000 7f7c0bc19700 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7c0419f980 con 0x7f7c041006b0 2023-07-14T19:28:37.828 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:37.824+0000 7f7c099b5700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7c040feb30 0x7f7c04069320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:28:37.828 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:37.824+0000 7f7c099b5700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7c040feb30 0x7f7c04069320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:45408/0 (socket says 172.21.15.98:45408) 2023-07-14T19:28:37.828 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:37.824+0000 7f7c099b5700 1 -- 172.21.15.98:0/2294794458 learned_addr learned my addr 172.21.15.98:0/2294794458 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-07-14T19:28:37.829 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:37.824+0000 7f7c091b4700 1 --2- 172.21.15.98:0/2294794458 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f7c040ffd20 0x7f7c0419ca50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:28:37.829 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:37.824+0000 7f7c0a1b6700 1 --2- 172.21.15.98:0/2294794458 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f7c041006b0 0x7f7c0419efa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:28:37.829 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:37.824+0000 7f7c099b5700 1 -- 172.21.15.98:0/2294794458 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f7c041006b0 msgr2=0x7f7c0419efa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:28:37.829 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:37.824+0000 7f7c099b5700 1 --2- 172.21.15.98:0/2294794458 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f7c041006b0 0x7f7c0419efa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:28:37.829 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:37.824+0000 7f7c099b5700 1 -- 172.21.15.98:0/2294794458 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f7c040ffd20 msgr2=0x7f7c0419ca50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:28:37.829 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:37.824+0000 7f7c099b5700 1 --2- 172.21.15.98:0/2294794458 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f7c040ffd20 0x7f7c0419ca50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:28:37.829 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:37.824+0000 7f7c099b5700 1 -- 172.21.15.98:0/2294794458 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7c000095f0 con 0x7f7c040feb30 2023-07-14T19:28:37.830 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:37.824+0000 7f7c091b4700 1 --2- 172.21.15.98:0/2294794458 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f7c040ffd20 0x7f7c0419ca50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-07-14T19:28:37.830 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:37.824+0000 7f7c0a1b6700 1 --2- 172.21.15.98:0/2294794458 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f7c041006b0 0x7f7c0419efa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-07-14T19:28:37.830 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:37.824+0000 7f7c099b5700 1 --2- 172.21.15.98:0/2294794458 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7c040feb30 0x7f7c04069320 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f7c00000c00 tx=0x7f7c0000f510 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:28:37.830 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:37.824+0000 7f7bfaffd700 1 -- 172.21.15.98:0/2294794458 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7c0001d070 con 0x7f7c040feb30 2023-07-14T19:28:37.830 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:37.824+0000 7f7c0bc19700 1 -- 172.21.15.98:0/2294794458 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7c0419fc00 con 0x7f7c040feb30 2023-07-14T19:28:37.830 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:37.824+0000 7f7bfaffd700 1 -- 172.21.15.98:0/2294794458 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f7c0000fc60 con 0x7f7c040feb30 2023-07-14T19:28:37.830 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:37.824+0000 7f7bfaffd700 1 -- 172.21.15.98:0/2294794458 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7c00017650 con 0x7f7c040feb30 2023-07-14T19:28:37.831 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:37.824+0000 7f7c0bc19700 1 -- 172.21.15.98:0/2294794458 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7c041a0080 con 0x7f7c040feb30 2023-07-14T19:28:37.831 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:37.825+0000 7f7c0bc19700 1 -- 172.21.15.98:0/2294794458 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7c04106090 con 0x7f7c040feb30 2023-07-14T19:28:37.831 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:37.826+0000 7f7bfaffd700 1 -- 172.21.15.98:0/2294794458 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 16) v1 ==== 135715+0+0 (secure 0 0 0) 0x7f7c00004830 con 0x7f7c040feb30 2023-07-14T19:28:37.832 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:37.827+0000 7f7bfaffd700 1 --2- 172.21.15.98:0/2294794458 >> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] conn(0x7f7bf00a11a0 0x7f7bf00a3650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:28:37.832 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:37.827+0000 7f7c091b4700 1 --2- 172.21.15.98:0/2294794458 >> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] conn(0x7f7bf00a11a0 0x7f7bf00a3650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:28:37.832 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:37.827+0000 7f7bfaffd700 1 -- 172.21.15.98:0/2294794458 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(79..79 src has 1..79) v4 ==== 7127+0+0 (secure 0 0 0) 0x7f7c000cda80 con 0x7f7c040feb30 2023-07-14T19:28:37.832 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:37.827+0000 7f7c091b4700 1 --2- 172.21.15.98:0/2294794458 >> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] conn(0x7f7bf00a11a0 0x7f7bf00a3650 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f7bf4009cb0 tx=0x7f7bf4009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:28:37.833 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:28:37 smithi098 ceph-mon[127486]: mds.? [v2:172.21.15.98:6834/586280675,v1:172.21.15.98:6835/586280675] up:boot 2023-07-14T19:28:37.833 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:28:37 smithi098 ceph-mon[127486]: fsmap 3 up:standby 2023-07-14T19:28:37.833 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:28:37 smithi098 ceph-mon[127486]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2023-07-14T19:28:37.833 INFO:journalctl@ceph.mds.d.smithi098.stdout:Jul 14 19:28:37 smithi098 podman[161135]: 2023-07-14T19:28:37.833 INFO:journalctl@ceph.mds.d.smithi098.stdout:Jul 14 19:28:37 smithi098 bash[161135]: 37123ac74367f93bfcafc86dd69ad1881f96434afb702396ef2c792ad5204641 2023-07-14T19:28:37.834 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:37.834+0000 7f7bfaffd700 1 -- 172.21.15.98:0/2294794458 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f7c00097700 con 0x7f7c040feb30 2023-07-14T19:28:37.942 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:28:37 smithi101 ceph-mon[128807]: mds.? [v2:172.21.15.98:6834/586280675,v1:172.21.15.98:6835/586280675] up:boot 2023-07-14T19:28:37.942 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:28:37 smithi101 ceph-mon[128807]: fsmap 3 up:standby 2023-07-14T19:28:37.942 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:28:37 smithi101 ceph-mon[128807]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2023-07-14T19:28:37.970 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:28:37 smithi139 ceph-mon[129293]: mds.? [v2:172.21.15.98:6834/586280675,v1:172.21.15.98:6835/586280675] up:boot 2023-07-14T19:28:37.971 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:28:37 smithi139 ceph-mon[129293]: fsmap 3 up:standby 2023-07-14T19:28:37.971 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:28:37 smithi139 ceph-mon[129293]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2023-07-14T19:28:38.075 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:38.075+0000 7f7c0bc19700 1 -- 172.21.15.98:0/2294794458 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f7c0402c8e0 con 0x7f7c040feb30 2023-07-14T19:28:38.076 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:38.075+0000 7f7bfaffd700 1 -- 172.21.15.98:0/2294794458 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 4 v4) v1 ==== 93+0+2774 (secure 0 0 0) 0x7f7c0009ad20 con 0x7f7c040feb30 2023-07-14T19:28:38.077 INFO:teuthology.orchestra.run.smithi098.stdout: 2023-07-14T19:28:38.077 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":4,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14565,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6835/586280675","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6834","nonce":586280675},{"type":"v1","addr":"172.21.15.98:6835","nonce":586280675}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":4},{"gid":24421,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.139:6833/3505477820","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6832","nonce":3505477820},{"type":"v1","addr":"172.21.15.139:6833","nonce":3505477820}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":24437,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.101:6833/3474460171","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.101:6832","nonce":3474460171},{"type":"v1","addr":"172.21.15.101:6833","nonce":3474460171}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3}],"filesystems":[]} 2023-07-14T19:28:38.078 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:38.077+0000 7f7c0bc19700 1 -- 172.21.15.98:0/2294794458 >> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] conn(0x7f7bf00a11a0 msgr2=0x7f7bf00a3650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:28:38.078 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:38.077+0000 7f7c0bc19700 1 --2- 172.21.15.98:0/2294794458 >> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] conn(0x7f7bf00a11a0 0x7f7bf00a3650 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f7bf4009cb0 tx=0x7f7bf4009260 comp rx=0 tx=0).stop 2023-07-14T19:28:38.079 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:38.077+0000 7f7c0bc19700 1 -- 172.21.15.98:0/2294794458 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7c040feb30 msgr2=0x7f7c04069320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:28:38.079 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:38.077+0000 7f7c0bc19700 1 --2- 172.21.15.98:0/2294794458 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7c040feb30 0x7f7c04069320 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f7c00000c00 tx=0x7f7c0000f510 comp rx=0 tx=0).stop 2023-07-14T19:28:38.079 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:38.078+0000 7f7c0bc19700 1 -- 172.21.15.98:0/2294794458 shutdown_connections 2023-07-14T19:28:38.079 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:38.078+0000 7f7c0bc19700 1 --2- 172.21.15.98:0/2294794458 >> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] conn(0x7f7bf00a11a0 0x7f7bf00a3650 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:28:38.079 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:38.078+0000 7f7c0bc19700 1 --2- 172.21.15.98:0/2294794458 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f7c041006b0 0x7f7c0419efa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:28:38.079 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:38.078+0000 7f7c0bc19700 1 --2- 172.21.15.98:0/2294794458 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7c040feb30 0x7f7c04069320 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:28:38.079 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:38.078+0000 7f7c0bc19700 1 --2- 172.21.15.98:0/2294794458 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f7c040ffd20 0x7f7c0419ca50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:28:38.080 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:38.078+0000 7f7c0bc19700 1 -- 172.21.15.98:0/2294794458 >> 172.21.15.98:0/2294794458 conn(0x7f7c040fa110 msgr2=0x7f7c041031d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:28:38.080 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:38.078+0000 7f7c0bc19700 1 -- 172.21.15.98:0/2294794458 shutdown_connections 2023-07-14T19:28:38.080 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:38.078+0000 7f7c0bc19700 1 -- 172.21.15.98:0/2294794458 wait complete. 2023-07-14T19:28:38.080 INFO:teuthology.orchestra.run.smithi098.stderr:dumped fsmap epoch 4 2023-07-14T19:28:38.089 INFO:journalctl@ceph.mds.d.smithi098.stdout:Jul 14 19:28:37 smithi098 ceph-mds[161199]: -- 172.21.15.98:0/1573047175 <== mon.0 v2:172.21.15.98:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55df273951e0 con 0x55df2736ac00 2023-07-14T19:28:38.089 INFO:journalctl@ceph.mds.d.smithi098.stdout:Jul 14 19:28:37 smithi098 ceph-9adea648-227b-11ee-9b33-001a4aab830c-mds-d[161151]: starting mds.d at 2023-07-14T19:28:38.090 INFO:journalctl@ceph.mds.d.smithi098.stdout:Jul 14 19:28:37 smithi098 systemd[1]: Started Ceph mds.d for 9adea648-227b-11ee-9b33-001a4aab830c. 2023-07-14T19:28:39.165 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:28:38 smithi101 ceph-mon[128807]: pgmap v237: 1 pgs: 1 active+clean; 577 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-14T19:28:39.165 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:28:38 smithi101 ceph-mon[128807]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' 2023-07-14T19:28:39.165 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:28:38 smithi101 ceph-mon[128807]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' 2023-07-14T19:28:39.165 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:28:38 smithi101 ceph-mon[128807]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' 2023-07-14T19:28:39.166 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:28:38 smithi101 ceph-mon[128807]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-07-14T19:28:39.166 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:28:38 smithi101 ceph-mon[128807]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-07-14T19:28:39.166 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:28:38 smithi101 ceph-mon[128807]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T19:28:39.166 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:28:38 smithi101 ceph-mon[128807]: Deploying daemon mds.e on smithi101 2023-07-14T19:28:39.166 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:28:38 smithi101 ceph-mon[128807]: from='client.? 172.21.15.98:0/2294794458' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-07-14T19:28:39.166 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:28:38 smithi101 ceph-mon[128807]: mds.? [v2:172.21.15.98:6836/487577257,v1:172.21.15.98:6837/487577257] up:boot 2023-07-14T19:28:39.167 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:28:38 smithi101 ceph-mon[128807]: fsmap 4 up:standby 2023-07-14T19:28:39.167 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:28:38 smithi101 ceph-mon[128807]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2023-07-14T19:28:39.220 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:28:38 smithi139 ceph-mon[129293]: pgmap v237: 1 pgs: 1 active+clean; 577 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-14T19:28:39.221 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:28:38 smithi139 ceph-mon[129293]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' 2023-07-14T19:28:39.221 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:28:38 smithi139 ceph-mon[129293]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' 2023-07-14T19:28:39.221 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:28:38 smithi139 ceph-mon[129293]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' 2023-07-14T19:28:39.221 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:28:38 smithi139 ceph-mon[129293]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-07-14T19:28:39.221 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:28:38 smithi139 ceph-mon[129293]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-07-14T19:28:39.221 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:28:38 smithi139 ceph-mon[129293]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T19:28:39.222 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:28:38 smithi139 ceph-mon[129293]: Deploying daemon mds.e on smithi101 2023-07-14T19:28:39.222 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:28:38 smithi139 ceph-mon[129293]: from='client.? 172.21.15.98:0/2294794458' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-07-14T19:28:39.222 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:28:38 smithi139 ceph-mon[129293]: mds.? [v2:172.21.15.98:6836/487577257,v1:172.21.15.98:6837/487577257] up:boot 2023-07-14T19:28:39.222 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:28:38 smithi139 ceph-mon[129293]: fsmap 4 up:standby 2023-07-14T19:28:39.223 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:28:38 smithi139 ceph-mon[129293]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2023-07-14T19:28:39.281 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:28:38 smithi098 ceph-mon[127486]: pgmap v237: 1 pgs: 1 active+clean; 577 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-14T19:28:39.281 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:28:38 smithi098 ceph-mon[127486]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' 2023-07-14T19:28:39.282 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:28:38 smithi098 ceph-mon[127486]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' 2023-07-14T19:28:39.282 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:28:38 smithi098 ceph-mon[127486]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' 2023-07-14T19:28:39.282 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:28:38 smithi098 ceph-mon[127486]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-07-14T19:28:39.282 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:28:38 smithi098 ceph-mon[127486]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-07-14T19:28:39.282 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:28:38 smithi098 ceph-mon[127486]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T19:28:39.282 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:28:38 smithi098 ceph-mon[127486]: Deploying daemon mds.e on smithi101 2023-07-14T19:28:39.283 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:28:38 smithi098 ceph-mon[127486]: from='client.? 172.21.15.98:0/2294794458' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-07-14T19:28:39.283 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:28:38 smithi098 ceph-mon[127486]: mds.? [v2:172.21.15.98:6836/487577257,v1:172.21.15.98:6837/487577257] up:boot 2023-07-14T19:28:39.283 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:28:38 smithi098 ceph-mon[127486]: fsmap 4 up:standby 2023-07-14T19:28:39.283 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:28:38 smithi098 ceph-mon[127486]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2023-07-14T19:28:39.821 INFO:journalctl@ceph.mds.e.smithi101.stdout:Jul 14 19:28:39 smithi101 systemd[1]: Starting Ceph mds.e for 9adea648-227b-11ee-9b33-001a4aab830c... 2023-07-14T19:28:40.087 INFO:journalctl@ceph.mds.e.smithi101.stdout:Jul 14 19:28:39 smithi101 podman[153366]: 2023-07-14T19:28:40.091 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-07-14T19:28:40.231 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:40.228+0000 7fbbfa4f7700 1 -- 172.21.15.98:0/1134176362 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbbf41017b0 msgr2=0x7fbbf4108070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:28:40.232 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:40.228+0000 7fbbfa4f7700 1 --2- 172.21.15.98:0/1134176362 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbbf41017b0 0x7fbbf4108070 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7fbbec00b0a0 tx=0x7fbbec00b3b0 comp rx=0 tx=0).stop 2023-07-14T19:28:40.232 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:40.228+0000 7fbbfa4f7700 1 -- 172.21.15.98:0/1134176362 shutdown_connections 2023-07-14T19:28:40.232 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:40.228+0000 7fbbfa4f7700 1 --2- 172.21.15.98:0/1134176362 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fbbf40ffc30 0x7fbbf4100040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:28:40.233 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:40.228+0000 7fbbfa4f7700 1 --2- 172.21.15.98:0/1134176362 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbbf41017b0 0x7fbbf4108070 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:28:40.233 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:40.228+0000 7fbbfa4f7700 1 --2- 172.21.15.98:0/1134176362 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7fbbf4100e20 0x7fbbf4101270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:28:40.233 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:40.228+0000 7fbbfa4f7700 1 -- 172.21.15.98:0/1134176362 >> 172.21.15.98:0/1134176362 conn(0x7fbbf40fb210 msgr2=0x7fbbf40fd680 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:28:40.233 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:40.228+0000 7fbbfa4f7700 1 -- 172.21.15.98:0/1134176362 shutdown_connections 2023-07-14T19:28:40.234 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:40.228+0000 7fbbfa4f7700 1 -- 172.21.15.98:0/1134176362 wait complete. 2023-07-14T19:28:40.234 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:40.229+0000 7fbbfa4f7700 1 Processor -- start 2023-07-14T19:28:40.234 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:40.229+0000 7fbbfa4f7700 1 -- start start 2023-07-14T19:28:40.234 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:40.229+0000 7fbbfa4f7700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbbf40ffc30 0x7fbbf40743f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:28:40.235 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:40.229+0000 7fbbfa4f7700 1 --2- >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7fbbf4100e20 0x7fbbf4072a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:28:40.235 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:40.229+0000 7fbbfa4f7700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fbbf41017b0 0x7fbbf4072f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:28:40.235 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:40.229+0000 7fbbfa4f7700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbbf4073710 con 0x7fbbf40ffc30 2023-07-14T19:28:40.235 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:40.229+0000 7fbbfa4f7700 1 -- --> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbbf4073880 con 0x7fbbf4100e20 2023-07-14T19:28:40.236 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:40.229+0000 7fbbfa4f7700 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbbf419fa90 con 0x7fbbf41017b0 2023-07-14T19:28:40.236 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:40.229+0000 7fbbf3fff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbbf40ffc30 0x7fbbf40743f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:28:40.237 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:40.229+0000 7fbbf3fff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbbf40ffc30 0x7fbbf40743f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:45436/0 (socket says 172.21.15.98:45436) 2023-07-14T19:28:40.237 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:40.229+0000 7fbbf3fff700 1 -- 172.21.15.98:0/4131307568 learned_addr learned my addr 172.21.15.98:0/4131307568 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-07-14T19:28:40.237 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:40.229+0000 7fbbf8a94700 1 --2- 172.21.15.98:0/4131307568 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fbbf41017b0 0x7fbbf4072f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:28:40.237 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:40.229+0000 7fbbf37fe700 1 --2- 172.21.15.98:0/4131307568 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7fbbf4100e20 0x7fbbf4072a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:28:40.238 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:40.229+0000 7fbbf3fff700 1 -- 172.21.15.98:0/4131307568 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fbbf41017b0 msgr2=0x7fbbf4072f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:28:40.238 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:40.229+0000 7fbbf3fff700 1 --2- 172.21.15.98:0/4131307568 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fbbf41017b0 0x7fbbf4072f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:28:40.238 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:40.229+0000 7fbbf3fff700 1 -- 172.21.15.98:0/4131307568 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7fbbf4100e20 msgr2=0x7fbbf4072a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:28:40.238 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:40.229+0000 7fbbf3fff700 1 --2- 172.21.15.98:0/4131307568 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7fbbf4100e20 0x7fbbf4072a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:28:40.239 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:40.229+0000 7fbbf3fff700 1 -- 172.21.15.98:0/4131307568 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbbe8009520 con 0x7fbbf40ffc30 2023-07-14T19:28:40.239 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:40.230+0000 7fbbf3fff700 1 --2- 172.21.15.98:0/4131307568 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbbf40ffc30 0x7fbbf40743f0 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7fbbe400d980 tx=0x7fbbe400dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:28:40.239 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:40.230+0000 7fbbf17fa700 1 -- 172.21.15.98:0/4131307568 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbbe4004020 con 0x7fbbf40ffc30 2023-07-14T19:28:40.240 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:40.230+0000 7fbbfa4f7700 1 -- 172.21.15.98:0/4131307568 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbbec009d00 con 0x7fbbf40ffc30 2023-07-14T19:28:40.240 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:40.230+0000 7fbbf17fa700 1 -- 172.21.15.98:0/4131307568 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fbbe400f690 con 0x7fbbf40ffc30 2023-07-14T19:28:40.240 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:40.230+0000 7fbbf17fa700 1 -- 172.21.15.98:0/4131307568 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbbe4003a60 con 0x7fbbf40ffc30 2023-07-14T19:28:40.240 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:40.230+0000 7fbbfa4f7700 1 -- 172.21.15.98:0/4131307568 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbbf41a00c0 con 0x7fbbf40ffc30 2023-07-14T19:28:40.240 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:40.231+0000 7fbbfa4f7700 1 -- 172.21.15.98:0/4131307568 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbbf41070d0 con 0x7fbbf40ffc30 2023-07-14T19:28:40.241 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:40.232+0000 7fbbf17fa700 1 -- 172.21.15.98:0/4131307568 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 16) v1 ==== 135715+0+0 (secure 0 0 0) 0x7fbbe400fb00 con 0x7fbbf40ffc30 2023-07-14T19:28:40.241 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:40.232+0000 7fbbf17fa700 1 --2- 172.21.15.98:0/4131307568 >> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] conn(0x7fbbdc0a1220 0x7fbbdc0a36d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:28:40.241 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:40.232+0000 7fbbf37fe700 1 --2- 172.21.15.98:0/4131307568 >> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] conn(0x7fbbdc0a1220 0x7fbbdc0a36d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:28:40.242 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:40.233+0000 7fbbf17fa700 1 -- 172.21.15.98:0/4131307568 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(79..79 src has 1..79) v4 ==== 7127+0+0 (secure 0 0 0) 0x7fbbe40cc440 con 0x7fbbf40ffc30 2023-07-14T19:28:40.242 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:40.233+0000 7fbbf37fe700 1 --2- 172.21.15.98:0/4131307568 >> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] conn(0x7fbbdc0a1220 0x7fbbdc0a36d0 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7fbbe8009df0 tx=0x7fbbe8009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:28:40.242 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:40.236+0000 7fbbf17fa700 1 -- 172.21.15.98:0/4131307568 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7fbbe4095f90 con 0x7fbbf40ffc30 2023-07-14T19:28:40.441 INFO:journalctl@ceph.mds.e.smithi101.stdout:Jul 14 19:28:40 smithi101 ceph-mds[153406]: -- 172.21.15.101:0/1871417503 <== mon.2 v2:172.21.15.101:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55c1ca3b31e0 con 0x55c1ca389000 2023-07-14T19:28:40.442 INFO:journalctl@ceph.mds.e.smithi101.stdout:Jul 14 19:28:40 smithi101 bash[153366]: 852dc3d2dd8969a47433393977eb3c0cf1e0a49873afb5bd8aa528c572edb767 2023-07-14T19:28:40.442 INFO:journalctl@ceph.mds.e.smithi101.stdout:Jul 14 19:28:40 smithi101 ceph-9adea648-227b-11ee-9b33-001a4aab830c-mds-e[153382]: starting mds.e at 2023-07-14T19:28:40.442 INFO:journalctl@ceph.mds.e.smithi101.stdout:Jul 14 19:28:40 smithi101 systemd[1]: Started Ceph mds.e for 9adea648-227b-11ee-9b33-001a4aab830c. 2023-07-14T19:28:40.467 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:40.466+0000 7fbbfa4f7700 1 -- 172.21.15.98:0/4131307568 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fbbf402cb90 con 0x7fbbf40ffc30 2023-07-14T19:28:40.467 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:40.467+0000 7fbbf17fa700 1 -- 172.21.15.98:0/4131307568 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 5 v5) v1 ==== 93+0+3528 (secure 0 0 0) 0x7fbbe4017de0 con 0x7fbbf40ffc30 2023-07-14T19:28:40.468 INFO:teuthology.orchestra.run.smithi098.stdout: 2023-07-14T19:28:40.468 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":5,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14565,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6835/586280675","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6834","nonce":586280675},{"type":"v1","addr":"172.21.15.98:6835","nonce":586280675}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":4},{"gid":14577,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6837/487577257","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6836","nonce":487577257},{"type":"v1","addr":"172.21.15.98:6837","nonce":487577257}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":5},{"gid":24421,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.139:6833/3505477820","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6832","nonce":3505477820},{"type":"v1","addr":"172.21.15.139:6833","nonce":3505477820}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":24437,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.101:6833/3474460171","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.101:6832","nonce":3474460171},{"type":"v1","addr":"172.21.15.101:6833","nonce":3474460171}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3}],"filesystems":[]} 2023-07-14T19:28:40.470 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:40.468+0000 7fbbfa4f7700 1 -- 172.21.15.98:0/4131307568 >> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] conn(0x7fbbdc0a1220 msgr2=0x7fbbdc0a36d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:28:40.470 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:40.469+0000 7fbbfa4f7700 1 --2- 172.21.15.98:0/4131307568 >> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] conn(0x7fbbdc0a1220 0x7fbbdc0a36d0 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7fbbe8009df0 tx=0x7fbbe8009260 comp rx=0 tx=0).stop 2023-07-14T19:28:40.470 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:40.469+0000 7fbbfa4f7700 1 -- 172.21.15.98:0/4131307568 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbbf40ffc30 msgr2=0x7fbbf40743f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:28:40.471 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:40.469+0000 7fbbfa4f7700 1 --2- 172.21.15.98:0/4131307568 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbbf40ffc30 0x7fbbf40743f0 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7fbbe400d980 tx=0x7fbbe400dc90 comp rx=0 tx=0).stop 2023-07-14T19:28:40.471 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:40.469+0000 7fbbfa4f7700 1 -- 172.21.15.98:0/4131307568 shutdown_connections 2023-07-14T19:28:40.471 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:40.469+0000 7fbbfa4f7700 1 --2- 172.21.15.98:0/4131307568 >> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] conn(0x7fbbdc0a1220 0x7fbbdc0a36d0 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:28:40.471 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:40.469+0000 7fbbfa4f7700 1 --2- 172.21.15.98:0/4131307568 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fbbf41017b0 0x7fbbf4072f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:28:40.471 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:40.469+0000 7fbbfa4f7700 1 --2- 172.21.15.98:0/4131307568 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbbf40ffc30 0x7fbbf40743f0 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:28:40.471 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:40.469+0000 7fbbfa4f7700 1 --2- 172.21.15.98:0/4131307568 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7fbbf4100e20 0x7fbbf4072a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:28:40.471 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:40.469+0000 7fbbfa4f7700 1 -- 172.21.15.98:0/4131307568 >> 172.21.15.98:0/4131307568 conn(0x7fbbf40fb210 msgr2=0x7fbbf4104210 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:28:40.472 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:40.469+0000 7fbbfa4f7700 1 -- 172.21.15.98:0/4131307568 shutdown_connections 2023-07-14T19:28:40.472 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:40.469+0000 7fbbfa4f7700 1 -- 172.21.15.98:0/4131307568 wait complete. 2023-07-14T19:28:40.472 INFO:teuthology.orchestra.run.smithi098.stderr:dumped fsmap epoch 5 2023-07-14T19:28:41.090 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:28:40 smithi139 ceph-mon[129293]: pgmap v238: 1 pgs: 1 active+clean; 577 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-14T19:28:41.090 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:28:40 smithi139 ceph-mon[129293]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' 2023-07-14T19:28:41.090 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:28:40 smithi139 ceph-mon[129293]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' 2023-07-14T19:28:41.090 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:28:40 smithi139 ceph-mon[129293]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' 2023-07-14T19:28:41.091 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:28:40 smithi139 ceph-mon[129293]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-07-14T19:28:41.091 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:28:40 smithi139 ceph-mon[129293]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-07-14T19:28:41.091 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:28:40 smithi139 ceph-mon[129293]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T19:28:41.091 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:28:40 smithi139 ceph-mon[129293]: from='client.? 172.21.15.98:0/4131307568' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-07-14T19:28:41.191 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:28:40 smithi101 ceph-mon[128807]: pgmap v238: 1 pgs: 1 active+clean; 577 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-14T19:28:41.192 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:28:40 smithi101 ceph-mon[128807]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' 2023-07-14T19:28:41.192 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:28:40 smithi101 ceph-mon[128807]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' 2023-07-14T19:28:41.192 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:28:40 smithi101 ceph-mon[128807]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' 2023-07-14T19:28:41.192 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:28:40 smithi101 ceph-mon[128807]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-07-14T19:28:41.192 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:28:40 smithi101 ceph-mon[128807]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-07-14T19:28:41.192 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:28:40 smithi101 ceph-mon[128807]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T19:28:41.193 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:28:40 smithi101 ceph-mon[128807]: from='client.? 172.21.15.98:0/4131307568' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-07-14T19:28:41.281 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:28:40 smithi098 ceph-mon[127486]: pgmap v238: 1 pgs: 1 active+clean; 577 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-14T19:28:41.281 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:28:40 smithi098 ceph-mon[127486]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' 2023-07-14T19:28:41.282 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:28:40 smithi098 ceph-mon[127486]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' 2023-07-14T19:28:41.282 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:28:40 smithi098 ceph-mon[127486]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' 2023-07-14T19:28:41.282 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:28:40 smithi098 ceph-mon[127486]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-07-14T19:28:41.282 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:28:40 smithi098 ceph-mon[127486]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-07-14T19:28:41.282 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:28:40 smithi098 ceph-mon[127486]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T19:28:41.282 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:28:40 smithi098 ceph-mon[127486]: from='client.? 172.21.15.98:0/4131307568' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-07-14T19:28:41.886 INFO:journalctl@ceph.mds.f.smithi139.stdout:Jul 14 19:28:41 smithi139 systemd[1]: Starting Ceph mds.f for 9adea648-227b-11ee-9b33-001a4aab830c... 2023-07-14T19:28:42.170 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:28:41 smithi139 ceph-mon[129293]: Deploying daemon mds.f on smithi139 2023-07-14T19:28:42.171 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:28:41 smithi139 ceph-mon[129293]: mds.? [v2:172.21.15.101:6834/1583264648,v1:172.21.15.101:6835/1583264648] up:boot 2023-07-14T19:28:42.171 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:28:41 smithi139 ceph-mon[129293]: fsmap 5 up:standby 2023-07-14T19:28:42.172 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:28:41 smithi139 ceph-mon[129293]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2023-07-14T19:28:42.172 INFO:journalctl@ceph.mds.f.smithi139.stdout:Jul 14 19:28:41 smithi139 podman[154845]: 2023-07-14T19:28:42.191 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:28:41 smithi101 ceph-mon[128807]: Deploying daemon mds.f on smithi139 2023-07-14T19:28:42.192 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:28:41 smithi101 ceph-mon[128807]: mds.? [v2:172.21.15.101:6834/1583264648,v1:172.21.15.101:6835/1583264648] up:boot 2023-07-14T19:28:42.192 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:28:41 smithi101 ceph-mon[128807]: fsmap 5 up:standby 2023-07-14T19:28:42.192 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:28:41 smithi101 ceph-mon[128807]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2023-07-14T19:28:42.281 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:28:41 smithi098 ceph-mon[127486]: Deploying daemon mds.f on smithi139 2023-07-14T19:28:42.281 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:28:41 smithi098 ceph-mon[127486]: mds.? [v2:172.21.15.101:6834/1583264648,v1:172.21.15.101:6835/1583264648] up:boot 2023-07-14T19:28:42.282 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:28:41 smithi098 ceph-mon[127486]: fsmap 5 up:standby 2023-07-14T19:28:42.282 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:28:41 smithi098 ceph-mon[127486]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2023-07-14T19:28:42.470 INFO:journalctl@ceph.mds.f.smithi139.stdout:Jul 14 19:28:42 smithi139 ceph-mds[154885]: -- 172.21.15.139:0/84342261 <== mon.1 v2:172.21.15.139:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x5646d9a2b1e0 con 0x5646d9a01000 2023-07-14T19:28:42.471 INFO:journalctl@ceph.mds.f.smithi139.stdout:Jul 14 19:28:42 smithi139 ceph-9adea648-227b-11ee-9b33-001a4aab830c-mds-f[154861]: starting mds.f at 2023-07-14T19:28:42.471 INFO:journalctl@ceph.mds.f.smithi139.stdout:Jul 14 19:28:42 smithi139 bash[154845]: 98c4e4395a70fd70e013dd43af333ffb3b967ea6fb1be8cd5f1f5644be99b4d2 2023-07-14T19:28:42.471 INFO:journalctl@ceph.mds.f.smithi139.stdout:Jul 14 19:28:42 smithi139 systemd[1]: Started Ceph mds.f for 9adea648-227b-11ee-9b33-001a4aab830c. 2023-07-14T19:28:42.482 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-07-14T19:28:42.616 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:42.615+0000 7f8be6337700 1 -- 172.21.15.98:0/2659572894 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8be01017a0 msgr2=0x7f8be0108060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:28:42.617 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:42.615+0000 7f8be6337700 1 --2- 172.21.15.98:0/2659572894 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8be01017a0 0x7f8be0108060 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f8bd800b0a0 tx=0x7f8bd800b3b0 comp rx=0 tx=0).stop 2023-07-14T19:28:42.617 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:42.616+0000 7f8be6337700 1 -- 172.21.15.98:0/2659572894 shutdown_connections 2023-07-14T19:28:42.617 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:42.616+0000 7f8be6337700 1 --2- 172.21.15.98:0/2659572894 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f8be00ffc20 0x7f8be0100030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:28:42.617 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:42.616+0000 7f8be6337700 1 --2- 172.21.15.98:0/2659572894 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8be01017a0 0x7f8be0108060 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:28:42.617 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:42.616+0000 7f8be6337700 1 --2- 172.21.15.98:0/2659572894 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f8be0100e10 0x7f8be0101260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:28:42.617 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:42.616+0000 7f8be6337700 1 -- 172.21.15.98:0/2659572894 >> 172.21.15.98:0/2659572894 conn(0x7f8be00fb260 msgr2=0x7f8be00fd670 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:28:42.618 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:42.616+0000 7f8be6337700 1 -- 172.21.15.98:0/2659572894 shutdown_connections 2023-07-14T19:28:42.618 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:42.616+0000 7f8be6337700 1 -- 172.21.15.98:0/2659572894 wait complete. 2023-07-14T19:28:42.618 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:42.616+0000 7f8be6337700 1 Processor -- start 2023-07-14T19:28:42.618 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:42.616+0000 7f8be6337700 1 -- start start 2023-07-14T19:28:42.620 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:42.617+0000 7f8be6337700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f8be00ffc20 0x7f8be01959b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:28:42.620 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:42.617+0000 7f8be6337700 1 --2- >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f8be0100e10 0x7f8be0195ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:28:42.620 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:42.617+0000 7f8be6337700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8be01017a0 0x7f8be019cf60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:28:42.620 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:42.617+0000 7f8be6337700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8be019d530 con 0x7f8be01017a0 2023-07-14T19:28:42.620 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:42.617+0000 7f8be6337700 1 -- --> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8be019d670 con 0x7f8be0100e10 2023-07-14T19:28:42.620 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:42.617+0000 7f8be6337700 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8be019d800 con 0x7f8be00ffc20 2023-07-14T19:28:42.621 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:42.617+0000 7f8be48d4700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8be01017a0 0x7f8be019cf60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:28:42.621 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:42.617+0000 7f8be48d4700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8be01017a0 0x7f8be019cf60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:45454/0 (socket says 172.21.15.98:45454) 2023-07-14T19:28:42.621 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:42.617+0000 7f8be48d4700 1 -- 172.21.15.98:0/4162714571 learned_addr learned my addr 172.21.15.98:0/4162714571 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-07-14T19:28:42.621 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:42.617+0000 7f8bdffff700 1 --2- 172.21.15.98:0/4162714571 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f8be00ffc20 0x7f8be01959b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:28:42.621 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:42.617+0000 7f8bdf7fe700 1 --2- 172.21.15.98:0/4162714571 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f8be0100e10 0x7f8be0195ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:28:42.621 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:42.617+0000 7f8be48d4700 1 -- 172.21.15.98:0/4162714571 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f8be00ffc20 msgr2=0x7f8be01959b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:28:42.621 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:42.617+0000 7f8be48d4700 1 --2- 172.21.15.98:0/4162714571 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f8be00ffc20 0x7f8be01959b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:28:42.622 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:42.617+0000 7f8be48d4700 1 -- 172.21.15.98:0/4162714571 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f8be0100e10 msgr2=0x7f8be0195ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:28:42.622 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:42.617+0000 7f8be48d4700 1 --2- 172.21.15.98:0/4162714571 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f8be0100e10 0x7f8be0195ef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:28:42.622 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:42.617+0000 7f8be48d4700 1 -- 172.21.15.98:0/4162714571 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8bd8009d00 con 0x7f8be01017a0 2023-07-14T19:28:42.622 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:42.617+0000 7f8bdffff700 1 --2- 172.21.15.98:0/4162714571 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f8be00ffc20 0x7f8be01959b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-07-14T19:28:42.622 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:42.617+0000 7f8be48d4700 1 --2- 172.21.15.98:0/4162714571 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8be01017a0 0x7f8be019cf60 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f8bd800b830 tx=0x7f8bd8004d30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:28:42.622 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:42.617+0000 7f8bdd7fa700 1 -- 172.21.15.98:0/4162714571 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8bd800e040 con 0x7f8be01017a0 2023-07-14T19:28:42.622 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:42.617+0000 7f8be6337700 1 -- 172.21.15.98:0/4162714571 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8be019da80 con 0x7f8be01017a0 2023-07-14T19:28:42.623 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:42.617+0000 7f8bdd7fa700 1 -- 172.21.15.98:0/4162714571 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f8bd80094c0 con 0x7f8be01017a0 2023-07-14T19:28:42.623 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:42.617+0000 7f8bdd7fa700 1 -- 172.21.15.98:0/4162714571 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8bd801bc70 con 0x7f8be01017a0 2023-07-14T19:28:42.623 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:42.617+0000 7f8be6337700 1 -- 172.21.15.98:0/4162714571 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8be019df50 con 0x7f8be01017a0 2023-07-14T19:28:42.623 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:42.618+0000 7f8be6337700 1 -- 172.21.15.98:0/4162714571 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8be010b400 con 0x7f8be01017a0 2023-07-14T19:28:42.623 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:42.619+0000 7f8bdd7fa700 1 -- 172.21.15.98:0/4162714571 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 16) v1 ==== 135715+0+0 (secure 0 0 0) 0x7f8bd801d890 con 0x7f8be01017a0 2023-07-14T19:28:42.624 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:42.620+0000 7f8bdd7fa700 1 --2- 172.21.15.98:0/4162714571 >> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] conn(0x7f8bc80a11a0 0x7f8bc80a3650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:28:42.624 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:42.620+0000 7f8bdffff700 1 --2- 172.21.15.98:0/4162714571 >> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] conn(0x7f8bc80a11a0 0x7f8bc80a3650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:28:42.624 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:42.620+0000 7f8bdd7fa700 1 -- 172.21.15.98:0/4162714571 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(79..79 src has 1..79) v4 ==== 7127+0+0 (secure 0 0 0) 0x7f8bd80ce780 con 0x7f8be01017a0 2023-07-14T19:28:42.624 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:42.620+0000 7f8bdffff700 1 --2- 172.21.15.98:0/4162714571 >> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] conn(0x7f8bc80a11a0 0x7f8bc80a3650 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f8be0100c70 tx=0x7f8bd0005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:28:42.624 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:42.623+0000 7f8bdd7fa700 1 -- 172.21.15.98:0/4162714571 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f8bd8098400 con 0x7f8be01017a0 2023-07-14T19:28:42.858 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:42.857+0000 7f8be6337700 1 -- 172.21.15.98:0/4162714571 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f8be01069e0 con 0x7f8be01017a0 2023-07-14T19:28:42.859 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:42.858+0000 7f8bdd7fa700 1 -- 172.21.15.98:0/4162714571 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 6 v6) v1 ==== 93+0+4288 (secure 0 0 0) 0x7f8bd809ba20 con 0x7f8be01017a0 2023-07-14T19:28:42.859 INFO:teuthology.orchestra.run.smithi098.stdout: 2023-07-14T19:28:42.860 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":6,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14565,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6835/586280675","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6834","nonce":586280675},{"type":"v1","addr":"172.21.15.98:6835","nonce":586280675}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":4},{"gid":14577,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6837/487577257","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6836","nonce":487577257},{"type":"v1","addr":"172.21.15.98:6837","nonce":487577257}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":5},{"gid":24421,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.139:6833/3505477820","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6832","nonce":3505477820},{"type":"v1","addr":"172.21.15.139:6833","nonce":3505477820}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":24437,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.101:6833/3474460171","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.101:6832","nonce":3474460171},{"type":"v1","addr":"172.21.15.101:6833","nonce":3474460171}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":24455,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.101:6835/1583264648","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.101:6834","nonce":1583264648},{"type":"v1","addr":"172.21.15.101:6835","nonce":1583264648}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":6}],"filesystems":[]} 2023-07-14T19:28:42.861 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:42.860+0000 7f8be6337700 1 -- 172.21.15.98:0/4162714571 >> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] conn(0x7f8bc80a11a0 msgr2=0x7f8bc80a3650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:28:42.862 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:42.860+0000 7f8be6337700 1 --2- 172.21.15.98:0/4162714571 >> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] conn(0x7f8bc80a11a0 0x7f8bc80a3650 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f8be0100c70 tx=0x7f8bd0005f90 comp rx=0 tx=0).stop 2023-07-14T19:28:42.862 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:42.860+0000 7f8be6337700 1 -- 172.21.15.98:0/4162714571 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8be01017a0 msgr2=0x7f8be019cf60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:28:42.862 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:42.860+0000 7f8be6337700 1 --2- 172.21.15.98:0/4162714571 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8be01017a0 0x7f8be019cf60 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f8bd800b830 tx=0x7f8bd8004d30 comp rx=0 tx=0).stop 2023-07-14T19:28:42.862 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:42.860+0000 7f8be6337700 1 -- 172.21.15.98:0/4162714571 shutdown_connections 2023-07-14T19:28:42.863 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:42.860+0000 7f8be6337700 1 --2- 172.21.15.98:0/4162714571 >> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] conn(0x7f8bc80a11a0 0x7f8bc80a3650 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:28:42.863 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:42.860+0000 7f8be6337700 1 --2- 172.21.15.98:0/4162714571 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f8be00ffc20 0x7f8be01959b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:28:42.863 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:42.860+0000 7f8be6337700 1 --2- 172.21.15.98:0/4162714571 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8be01017a0 0x7f8be019cf60 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:28:42.863 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:42.860+0000 7f8be6337700 1 --2- 172.21.15.98:0/4162714571 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f8be0100e10 0x7f8be0195ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:28:42.864 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:42.860+0000 7f8be6337700 1 -- 172.21.15.98:0/4162714571 >> 172.21.15.98:0/4162714571 conn(0x7f8be00fb260 msgr2=0x7f8be00fd350 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:28:42.864 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:42.860+0000 7f8be6337700 1 -- 172.21.15.98:0/4162714571 shutdown_connections 2023-07-14T19:28:42.864 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:42.860+0000 7f8be6337700 1 -- 172.21.15.98:0/4162714571 wait complete. 2023-07-14T19:28:42.864 INFO:teuthology.orchestra.run.smithi098.stderr:dumped fsmap epoch 6 2023-07-14T19:28:43.191 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:28:42 smithi101 ceph-mon[128807]: pgmap v239: 1 pgs: 1 active+clean; 577 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-14T19:28:43.192 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:28:42 smithi101 ceph-mon[128807]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' 2023-07-14T19:28:43.192 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:28:42 smithi101 ceph-mon[128807]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' 2023-07-14T19:28:43.192 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:28:42 smithi101 ceph-mon[128807]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' 2023-07-14T19:28:43.192 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:28:42 smithi101 ceph-mon[128807]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-07-14T19:28:43.192 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:28:42 smithi101 ceph-mon[128807]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-07-14T19:28:43.193 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:28:42 smithi101 ceph-mon[128807]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T19:28:43.193 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:28:42 smithi101 ceph-mon[128807]: from='client.? 172.21.15.98:0/4162714571' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-07-14T19:28:43.220 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:28:42 smithi139 ceph-mon[129293]: pgmap v239: 1 pgs: 1 active+clean; 577 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-14T19:28:43.221 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:28:42 smithi139 ceph-mon[129293]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' 2023-07-14T19:28:43.221 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:28:42 smithi139 ceph-mon[129293]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' 2023-07-14T19:28:43.221 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:28:42 smithi139 ceph-mon[129293]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' 2023-07-14T19:28:43.221 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:28:42 smithi139 ceph-mon[129293]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-07-14T19:28:43.221 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:28:42 smithi139 ceph-mon[129293]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-07-14T19:28:43.221 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:28:42 smithi139 ceph-mon[129293]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T19:28:43.222 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:28:42 smithi139 ceph-mon[129293]: from='client.? 172.21.15.98:0/4162714571' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-07-14T19:28:43.281 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:28:42 smithi098 ceph-mon[127486]: pgmap v239: 1 pgs: 1 active+clean; 577 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-14T19:28:43.282 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:28:42 smithi098 ceph-mon[127486]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' 2023-07-14T19:28:43.282 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:28:42 smithi098 ceph-mon[127486]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' 2023-07-14T19:28:43.282 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:28:42 smithi098 ceph-mon[127486]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' 2023-07-14T19:28:43.282 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:28:42 smithi098 ceph-mon[127486]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-07-14T19:28:43.282 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:28:42 smithi098 ceph-mon[127486]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-07-14T19:28:43.282 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:28:42 smithi098 ceph-mon[127486]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T19:28:43.282 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:28:42 smithi098 ceph-mon[127486]: from='client.? 172.21.15.98:0/4162714571' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-07-14T19:28:43.936 INFO:journalctl@ceph.mds.c.smithi139.stdout:Jul 14 19:28:43 smithi139 systemd[1]: Starting Ceph mds.c for 9adea648-227b-11ee-9b33-001a4aab830c... 2023-07-14T19:28:44.191 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:28:43 smithi101 ceph-mon[128807]: Deploying daemon mds.c on smithi139 2023-07-14T19:28:44.192 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:28:43 smithi101 ceph-mon[128807]: mds.? [v2:172.21.15.139:6834/2315281724,v1:172.21.15.139:6835/2315281724] up:boot 2023-07-14T19:28:44.192 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:28:43 smithi101 ceph-mon[128807]: fsmap 6 up:standby 2023-07-14T19:28:44.192 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:28:43 smithi101 ceph-mon[128807]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2023-07-14T19:28:44.220 INFO:journalctl@ceph.mds.c.smithi139.stdout:Jul 14 19:28:44 smithi139 podman[155185]: 2023-07-14T19:28:44.221 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:28:43 smithi139 ceph-mon[129293]: Deploying daemon mds.c on smithi139 2023-07-14T19:28:44.221 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:28:43 smithi139 ceph-mon[129293]: mds.? [v2:172.21.15.139:6834/2315281724,v1:172.21.15.139:6835/2315281724] up:boot 2023-07-14T19:28:44.221 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:28:43 smithi139 ceph-mon[129293]: fsmap 6 up:standby 2023-07-14T19:28:44.221 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:28:43 smithi139 ceph-mon[129293]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2023-07-14T19:28:44.281 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:28:43 smithi098 ceph-mon[127486]: Deploying daemon mds.c on smithi139 2023-07-14T19:28:44.281 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:28:43 smithi098 ceph-mon[127486]: mds.? [v2:172.21.15.139:6834/2315281724,v1:172.21.15.139:6835/2315281724] up:boot 2023-07-14T19:28:44.281 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:28:43 smithi098 ceph-mon[127486]: fsmap 6 up:standby 2023-07-14T19:28:44.282 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:28:43 smithi098 ceph-mon[127486]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2023-07-14T19:28:44.720 INFO:journalctl@ceph.mds.c.smithi139.stdout:Jul 14 19:28:44 smithi139 ceph-mds[155225]: -- 172.21.15.139:0/168653151 <== mon.1 v2:172.21.15.139:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55cd07a271e0 con 0x55cd079fcc00 2023-07-14T19:28:44.721 INFO:journalctl@ceph.mds.c.smithi139.stdout:Jul 14 19:28:44 smithi139 ceph-9adea648-227b-11ee-9b33-001a4aab830c-mds-c[155201]: starting mds.c at 2023-07-14T19:28:44.721 INFO:journalctl@ceph.mds.c.smithi139.stdout:Jul 14 19:28:44 smithi139 bash[155185]: 5ee99378ab7a95b254e92843046e60ffa1382bd43e927e03806a428d8577e967 2023-07-14T19:28:44.721 INFO:journalctl@ceph.mds.c.smithi139.stdout:Jul 14 19:28:44 smithi139 systemd[1]: Started Ceph mds.c for 9adea648-227b-11ee-9b33-001a4aab830c. 2023-07-14T19:28:44.873 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-07-14T19:28:45.010 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:45.008+0000 7f7a92ebe700 1 -- 172.21.15.98:0/3419700473 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7a8c0fdee0 msgr2=0x7f7a8c0fe2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:28:45.011 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:45.008+0000 7f7a92ebe700 1 --2- 172.21.15.98:0/3419700473 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7a8c0fdee0 0x7f7a8c0fe2f0 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f7a80009990 tx=0x7f7a80009ca0 comp rx=0 tx=0).stop 2023-07-14T19:28:45.011 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:45.009+0000 7f7a92ebe700 1 -- 172.21.15.98:0/3419700473 shutdown_connections 2023-07-14T19:28:45.011 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:45.009+0000 7f7a92ebe700 1 --2- 172.21.15.98:0/3419700473 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f7a8c1018e0 0x7f7a8c1081a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:28:45.011 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:45.009+0000 7f7a92ebe700 1 --2- 172.21.15.98:0/3419700473 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7a8c0fdee0 0x7f7a8c0fe2f0 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:28:45.011 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:45.009+0000 7f7a92ebe700 1 --2- 172.21.15.98:0/3419700473 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f7a8c100f50 0x7f7a8c1013a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:28:45.012 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:45.009+0000 7f7a92ebe700 1 -- 172.21.15.98:0/3419700473 >> 172.21.15.98:0/3419700473 conn(0x7f7a8c0f98a0 msgr2=0x7f7a8c0fbcd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:28:45.012 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:45.009+0000 7f7a92ebe700 1 -- 172.21.15.98:0/3419700473 shutdown_connections 2023-07-14T19:28:45.012 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:45.009+0000 7f7a92ebe700 1 -- 172.21.15.98:0/3419700473 wait complete. 2023-07-14T19:28:45.012 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:45.009+0000 7f7a92ebe700 1 Processor -- start 2023-07-14T19:28:45.012 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:45.009+0000 7f7a92ebe700 1 -- start start 2023-07-14T19:28:45.012 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:45.009+0000 7f7a92ebe700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7a8c0fdee0 0x7f7a8c195920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:28:45.012 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:45.009+0000 7f7a92ebe700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f7a8c100f50 0x7f7a8c195e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:28:45.013 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:45.009+0000 7f7a92ebe700 1 --2- >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f7a8c1018e0 0x7f7a8c19ced0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:28:45.013 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:45.009+0000 7f7a92ebe700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7a8c19d4a0 con 0x7f7a8c0fdee0 2023-07-14T19:28:45.013 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:45.009+0000 7f7a92ebe700 1 -- --> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7a8c19d5e0 con 0x7f7a8c1018e0 2023-07-14T19:28:45.013 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:45.009+0000 7f7a92ebe700 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7a8c19d770 con 0x7f7a8c100f50 2023-07-14T19:28:45.013 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:45.009+0000 7f7a90c5a700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7a8c0fdee0 0x7f7a8c195920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:28:45.013 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:45.009+0000 7f7a90c5a700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7a8c0fdee0 0x7f7a8c195920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:45464/0 (socket says 172.21.15.98:45464) 2023-07-14T19:28:45.013 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:45.009+0000 7f7a90c5a700 1 -- 172.21.15.98:0/2958357794 learned_addr learned my addr 172.21.15.98:0/2958357794 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-07-14T19:28:45.013 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:45.010+0000 7f7a8bfff700 1 --2- 172.21.15.98:0/2958357794 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f7a8c100f50 0x7f7a8c195e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:28:45.014 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:45.010+0000 7f7a9145b700 1 --2- 172.21.15.98:0/2958357794 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f7a8c1018e0 0x7f7a8c19ced0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:28:45.014 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:45.010+0000 7f7a90c5a700 1 -- 172.21.15.98:0/2958357794 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f7a8c100f50 msgr2=0x7f7a8c195e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:28:45.014 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:45.010+0000 7f7a90c5a700 1 --2- 172.21.15.98:0/2958357794 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f7a8c100f50 0x7f7a8c195e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:28:45.014 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:45.010+0000 7f7a90c5a700 1 -- 172.21.15.98:0/2958357794 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f7a8c1018e0 msgr2=0x7f7a8c19ced0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:28:45.014 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:45.010+0000 7f7a90c5a700 1 --2- 172.21.15.98:0/2958357794 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f7a8c1018e0 0x7f7a8c19ced0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:28:45.014 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:45.010+0000 7f7a90c5a700 1 -- 172.21.15.98:0/2958357794 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7a800095f0 con 0x7f7a8c0fdee0 2023-07-14T19:28:45.014 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:45.010+0000 7f7a90c5a700 1 --2- 172.21.15.98:0/2958357794 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7a8c0fdee0 0x7f7a8c195920 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f7a800095c0 tx=0x7f7a8000f690 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:28:45.015 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:45.010+0000 7f7a89ffb700 1 -- 172.21.15.98:0/2958357794 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a8001d070 con 0x7f7a8c0fdee0 2023-07-14T19:28:45.015 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:45.010+0000 7f7a89ffb700 1 -- 172.21.15.98:0/2958357794 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f7a80004830 con 0x7f7a8c0fdee0 2023-07-14T19:28:45.015 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:45.010+0000 7f7a92ebe700 1 -- 172.21.15.98:0/2958357794 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7a8c19d9f0 con 0x7f7a8c0fdee0 2023-07-14T19:28:45.015 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:45.010+0000 7f7a89ffb700 1 -- 172.21.15.98:0/2958357794 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a80005420 con 0x7f7a8c0fdee0 2023-07-14T19:28:45.015 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:45.010+0000 7f7a92ebe700 1 -- 172.21.15.98:0/2958357794 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7a8c19ddb0 con 0x7f7a8c0fdee0 2023-07-14T19:28:45.017 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:45.011+0000 7f7a92ebe700 1 -- 172.21.15.98:0/2958357794 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7a8c1072d0 con 0x7f7a8c0fdee0 2023-07-14T19:28:45.017 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:45.012+0000 7f7a89ffb700 1 -- 172.21.15.98:0/2958357794 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 16) v1 ==== 135715+0+0 (secure 0 0 0) 0x7f7a800055c0 con 0x7f7a8c0fdee0 2023-07-14T19:28:45.017 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:45.013+0000 7f7a89ffb700 1 --2- 172.21.15.98:0/2958357794 >> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] conn(0x7f7a7c0a11a0 0x7f7a7c0a3650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:28:45.017 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:45.013+0000 7f7a8bfff700 1 --2- 172.21.15.98:0/2958357794 >> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] conn(0x7f7a7c0a11a0 0x7f7a7c0a3650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:28:45.017 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:45.013+0000 7f7a89ffb700 1 -- 172.21.15.98:0/2958357794 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(79..79 src has 1..79) v4 ==== 7127+0+0 (secure 0 0 0) 0x7f7a800cd660 con 0x7f7a8c0fdee0 2023-07-14T19:28:45.017 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:45.013+0000 7f7a8bfff700 1 --2- 172.21.15.98:0/2958357794 >> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] conn(0x7f7a7c0a11a0 0x7f7a7c0a3650 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f7a78005950 tx=0x7f7a7800b410 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:28:45.018 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:45.017+0000 7f7a89ffb700 1 -- 172.21.15.98:0/2958357794 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f7a8005b780 con 0x7f7a8c0fdee0 2023-07-14T19:28:45.191 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:28:44 smithi101 ceph-mon[128807]: pgmap v240: 1 pgs: 1 active+clean; 577 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-14T19:28:45.192 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:28:44 smithi101 ceph-mon[128807]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' 2023-07-14T19:28:45.192 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:28:44 smithi101 ceph-mon[128807]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' 2023-07-14T19:28:45.192 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:28:44 smithi101 ceph-mon[128807]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' 2023-07-14T19:28:45.192 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:28:44 smithi101 ceph-mon[128807]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-07-14T19:28:45.193 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:28:44 smithi101 ceph-mon[128807]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-07-14T19:28:45.193 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:28:44 smithi101 ceph-mon[128807]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T19:28:45.220 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:28:44 smithi139 ceph-mon[129293]: pgmap v240: 1 pgs: 1 active+clean; 577 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-14T19:28:45.220 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:28:44 smithi139 ceph-mon[129293]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' 2023-07-14T19:28:45.221 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:28:44 smithi139 ceph-mon[129293]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' 2023-07-14T19:28:45.221 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:28:44 smithi139 ceph-mon[129293]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' 2023-07-14T19:28:45.221 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:28:44 smithi139 ceph-mon[129293]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-07-14T19:28:45.221 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:28:44 smithi139 ceph-mon[129293]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-07-14T19:28:45.221 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:28:44 smithi139 ceph-mon[129293]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T19:28:45.250 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:45.249+0000 7f7a92ebe700 1 -- 172.21.15.98:0/2958357794 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f7a8c02c8e0 con 0x7f7a8c0fdee0 2023-07-14T19:28:45.251 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:45.250+0000 7f7a89ffb700 1 -- 172.21.15.98:0/2958357794 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 8 v8) v1 ==== 93+0+5808 (secure 0 0 0) 0x7f7a8009ad70 con 0x7f7a8c0fdee0 2023-07-14T19:28:45.251 INFO:teuthology.orchestra.run.smithi098.stdout: 2023-07-14T19:28:45.251 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":8,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14565,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6835/586280675","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6834","nonce":586280675},{"type":"v1","addr":"172.21.15.98:6835","nonce":586280675}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":4},{"gid":14577,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6837/487577257","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6836","nonce":487577257},{"type":"v1","addr":"172.21.15.98:6837","nonce":487577257}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":5},{"gid":24421,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.139:6833/3505477820","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6832","nonce":3505477820},{"type":"v1","addr":"172.21.15.139:6833","nonce":3505477820}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":24437,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.101:6833/3474460171","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.101:6832","nonce":3474460171},{"type":"v1","addr":"172.21.15.101:6833","nonce":3474460171}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":24455,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.101:6835/1583264648","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.101:6834","nonce":1583264648},{"type":"v1","addr":"172.21.15.101:6835","nonce":1583264648}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":6},{"gid":24460,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.139:6835/2315281724","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6834","nonce":2315281724},{"type":"v1","addr":"172.21.15.139:6835","nonce":2315281724}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":7},{"gid":24469,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.139:6837/3597660552","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6836","nonce":3597660552},{"type":"v1","addr":"172.21.15.139:6837","nonce":3597660552}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":8}],"filesystems":[]} 2023-07-14T19:28:45.252 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:45.252+0000 7f7a92ebe700 1 -- 172.21.15.98:0/2958357794 >> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] conn(0x7f7a7c0a11a0 msgr2=0x7f7a7c0a3650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:28:45.253 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:45.252+0000 7f7a92ebe700 1 --2- 172.21.15.98:0/2958357794 >> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] conn(0x7f7a7c0a11a0 0x7f7a7c0a3650 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f7a78005950 tx=0x7f7a7800b410 comp rx=0 tx=0).stop 2023-07-14T19:28:45.253 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:45.252+0000 7f7a92ebe700 1 -- 172.21.15.98:0/2958357794 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7a8c0fdee0 msgr2=0x7f7a8c195920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:28:45.253 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:45.252+0000 7f7a92ebe700 1 --2- 172.21.15.98:0/2958357794 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7a8c0fdee0 0x7f7a8c195920 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f7a800095c0 tx=0x7f7a8000f690 comp rx=0 tx=0).stop 2023-07-14T19:28:45.253 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:45.252+0000 7f7a92ebe700 1 -- 172.21.15.98:0/2958357794 shutdown_connections 2023-07-14T19:28:45.253 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:45.252+0000 7f7a92ebe700 1 --2- 172.21.15.98:0/2958357794 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f7a8c1018e0 0x7f7a8c19ced0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:28:45.253 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:45.252+0000 7f7a92ebe700 1 --2- 172.21.15.98:0/2958357794 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7a8c0fdee0 0x7f7a8c195920 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:28:45.254 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:45.252+0000 7f7a92ebe700 1 --2- 172.21.15.98:0/2958357794 >> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] conn(0x7f7a7c0a11a0 0x7f7a7c0a3650 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:28:45.254 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:45.252+0000 7f7a92ebe700 1 --2- 172.21.15.98:0/2958357794 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f7a8c100f50 0x7f7a8c195e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:28:45.254 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:45.252+0000 7f7a92ebe700 1 -- 172.21.15.98:0/2958357794 >> 172.21.15.98:0/2958357794 conn(0x7f7a8c0f98a0 msgr2=0x7f7a8c104410 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:28:45.254 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:45.252+0000 7f7a92ebe700 1 -- 172.21.15.98:0/2958357794 shutdown_connections 2023-07-14T19:28:45.254 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:45.252+0000 7f7a92ebe700 1 -- 172.21.15.98:0/2958357794 wait complete. 2023-07-14T19:28:45.254 INFO:teuthology.orchestra.run.smithi098.stderr:dumped fsmap epoch 8 2023-07-14T19:28:45.264 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:28:44 smithi098 ceph-mon[127486]: pgmap v240: 1 pgs: 1 active+clean; 577 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-14T19:28:45.264 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:28:44 smithi098 ceph-mon[127486]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' 2023-07-14T19:28:45.264 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:28:44 smithi098 ceph-mon[127486]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' 2023-07-14T19:28:45.265 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:28:44 smithi098 ceph-mon[127486]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' 2023-07-14T19:28:45.265 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:28:44 smithi098 ceph-mon[127486]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-07-14T19:28:45.265 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:28:44 smithi098 ceph-mon[127486]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-07-14T19:28:45.265 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:28:44 smithi098 ceph-mon[127486]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T19:28:46.114 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:28:45 smithi101 ceph-mon[128807]: Deploying daemon mds.h on smithi101 2023-07-14T19:28:46.114 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:28:45 smithi101 ceph-mon[128807]: mds.? [v2:172.21.15.139:6836/3597660552,v1:172.21.15.139:6837/3597660552] up:boot 2023-07-14T19:28:46.114 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:28:45 smithi101 ceph-mon[128807]: fsmap 7 up:standby 2023-07-14T19:28:46.114 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:28:45 smithi101 ceph-mon[128807]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2023-07-14T19:28:46.114 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:28:45 smithi101 ceph-mon[128807]: from='client.? 172.21.15.98:0/2958357794' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-07-14T19:28:46.221 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:28:45 smithi139 ceph-mon[129293]: Deploying daemon mds.h on smithi101 2023-07-14T19:28:46.221 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:28:45 smithi139 ceph-mon[129293]: mds.? [v2:172.21.15.139:6836/3597660552,v1:172.21.15.139:6837/3597660552] up:boot 2023-07-14T19:28:46.221 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:28:45 smithi139 ceph-mon[129293]: fsmap 7 up:standby 2023-07-14T19:28:46.221 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:28:45 smithi139 ceph-mon[129293]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2023-07-14T19:28:46.221 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:28:45 smithi139 ceph-mon[129293]: from='client.? 172.21.15.98:0/2958357794' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-07-14T19:28:46.281 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:28:45 smithi098 ceph-mon[127486]: Deploying daemon mds.h on smithi101 2023-07-14T19:28:46.281 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:28:45 smithi098 ceph-mon[127486]: mds.? [v2:172.21.15.139:6836/3597660552,v1:172.21.15.139:6837/3597660552] up:boot 2023-07-14T19:28:46.282 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:28:45 smithi098 ceph-mon[127486]: fsmap 7 up:standby 2023-07-14T19:28:46.282 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:28:45 smithi098 ceph-mon[127486]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2023-07-14T19:28:46.282 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:28:45 smithi098 ceph-mon[127486]: from='client.? 172.21.15.98:0/2958357794' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-07-14T19:28:46.427 INFO:journalctl@ceph.mds.h.smithi101.stdout:Jul 14 19:28:46 smithi101 systemd[1]: Starting Ceph mds.h for 9adea648-227b-11ee-9b33-001a4aab830c... 2023-07-14T19:28:46.686 INFO:journalctl@ceph.mds.h.smithi101.stdout:Jul 14 19:28:46 smithi101 podman[153709]: 2023-07-14T19:28:46.941 INFO:journalctl@ceph.mds.h.smithi101.stdout:Jul 14 19:28:46 smithi101 ceph-mds[153750]: -- 172.21.15.101:0/4218054635 <== mon.2 v2:172.21.15.101:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55f15de111e0 con 0x55f15dde6c00 2023-07-14T19:28:46.942 INFO:journalctl@ceph.mds.h.smithi101.stdout:Jul 14 19:28:46 smithi101 bash[153709]: 72be8a5e632bfb70ccaa9f12053470bbbf9ea68fe867809993e40ce8aec98f84 2023-07-14T19:28:46.942 INFO:journalctl@ceph.mds.h.smithi101.stdout:Jul 14 19:28:46 smithi101 ceph-9adea648-227b-11ee-9b33-001a4aab830c-mds-h[153725]: starting mds.h at 2023-07-14T19:28:46.942 INFO:journalctl@ceph.mds.h.smithi101.stdout:Jul 14 19:28:46 smithi101 systemd[1]: Started Ceph mds.h for 9adea648-227b-11ee-9b33-001a4aab830c. 2023-07-14T19:28:47.220 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:28:46 smithi139 ceph-mon[129293]: pgmap v241: 1 pgs: 1 active+clean; 577 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-14T19:28:47.220 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:28:46 smithi139 ceph-mon[129293]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' 2023-07-14T19:28:47.221 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:28:46 smithi139 ceph-mon[129293]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' 2023-07-14T19:28:47.221 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:28:46 smithi139 ceph-mon[129293]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' 2023-07-14T19:28:47.221 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:28:46 smithi139 ceph-mon[129293]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-07-14T19:28:47.221 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:28:46 smithi139 ceph-mon[129293]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-07-14T19:28:47.221 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:28:46 smithi139 ceph-mon[129293]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T19:28:47.267 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-07-14T19:28:47.281 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:28:46 smithi098 ceph-mon[127486]: pgmap v241: 1 pgs: 1 active+clean; 577 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-14T19:28:47.282 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:28:46 smithi098 ceph-mon[127486]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' 2023-07-14T19:28:47.282 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:28:46 smithi098 ceph-mon[127486]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' 2023-07-14T19:28:47.282 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:28:46 smithi098 ceph-mon[127486]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' 2023-07-14T19:28:47.283 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:28:46 smithi098 ceph-mon[127486]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-07-14T19:28:47.283 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:28:46 smithi098 ceph-mon[127486]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-07-14T19:28:47.283 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:28:46 smithi098 ceph-mon[127486]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T19:28:47.406 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:47.401+0000 7fa941eb3700 1 -- 172.21.15.98:0/3293444177 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa93c1018c0 msgr2=0x7fa93c108180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:28:47.407 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:47.401+0000 7fa941eb3700 1 --2- 172.21.15.98:0/3293444177 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa93c1018c0 0x7fa93c108180 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7fa92c009990 tx=0x7fa92c009ca0 comp rx=0 tx=0).stop 2023-07-14T19:28:47.407 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:47.401+0000 7fa941eb3700 1 -- 172.21.15.98:0/3293444177 shutdown_connections 2023-07-14T19:28:47.407 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:47.401+0000 7fa941eb3700 1 --2- 172.21.15.98:0/3293444177 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa93c1018c0 0x7fa93c108180 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:28:47.407 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:47.401+0000 7fa941eb3700 1 --2- 172.21.15.98:0/3293444177 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7fa93c0fdec0 0x7fa93c0fe2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:28:47.407 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:47.401+0000 7fa941eb3700 1 --2- 172.21.15.98:0/3293444177 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fa93c100f30 0x7fa93c101380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:28:47.408 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:47.401+0000 7fa941eb3700 1 -- 172.21.15.98:0/3293444177 >> 172.21.15.98:0/3293444177 conn(0x7fa93c0f98a0 msgr2=0x7fa93c0fbcd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:28:47.408 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:47.401+0000 7fa941eb3700 1 -- 172.21.15.98:0/3293444177 shutdown_connections 2023-07-14T19:28:47.408 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:47.401+0000 7fa941eb3700 1 -- 172.21.15.98:0/3293444177 wait complete. 2023-07-14T19:28:47.408 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:47.402+0000 7fa941eb3700 1 Processor -- start 2023-07-14T19:28:47.408 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:47.402+0000 7fa941eb3700 1 -- start start 2023-07-14T19:28:47.408 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:47.402+0000 7fa941eb3700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa93c0fdec0 0x7fa93c195850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:28:47.409 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:47.402+0000 7fa941eb3700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fa93c100f30 0x7fa93c195d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:28:47.409 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:47.402+0000 7fa941eb3700 1 --2- >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7fa93c1018c0 0x7fa93c19ce00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:28:47.409 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:47.402+0000 7fa941eb3700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa93c19d3d0 con 0x7fa93c0fdec0 2023-07-14T19:28:47.409 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:47.402+0000 7fa941eb3700 1 -- --> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa93c19d510 con 0x7fa93c1018c0 2023-07-14T19:28:47.409 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:47.402+0000 7fa941eb3700 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa93c19d6a0 con 0x7fa93c100f30 2023-07-14T19:28:47.409 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:47.402+0000 7fa93b7fe700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa93c0fdec0 0x7fa93c195850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:28:47.409 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:47.402+0000 7fa93affd700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fa93c100f30 0x7fa93c195d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:28:47.410 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:47.402+0000 7fa93b7fe700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa93c0fdec0 0x7fa93c195850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:45478/0 (socket says 172.21.15.98:45478) 2023-07-14T19:28:47.410 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:47.402+0000 7fa93b7fe700 1 -- 172.21.15.98:0/963357868 learned_addr learned my addr 172.21.15.98:0/963357868 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-07-14T19:28:47.410 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:47.402+0000 7fa93bfff700 1 --2- 172.21.15.98:0/963357868 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7fa93c1018c0 0x7fa93c19ce00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:28:47.410 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:47.403+0000 7fa93b7fe700 1 -- 172.21.15.98:0/963357868 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fa93c100f30 msgr2=0x7fa93c195d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:28:47.410 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:47.403+0000 7fa93b7fe700 1 --2- 172.21.15.98:0/963357868 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fa93c100f30 0x7fa93c195d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:28:47.411 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:47.403+0000 7fa93b7fe700 1 -- 172.21.15.98:0/963357868 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7fa93c1018c0 msgr2=0x7fa93c19ce00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:28:47.411 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:47.403+0000 7fa93b7fe700 1 --2- 172.21.15.98:0/963357868 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7fa93c1018c0 0x7fa93c19ce00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:28:47.411 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:47.403+0000 7fa93b7fe700 1 -- 172.21.15.98:0/963357868 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa92c0095f0 con 0x7fa93c0fdec0 2023-07-14T19:28:47.411 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:47.403+0000 7fa93b7fe700 1 --2- 172.21.15.98:0/963357868 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa93c0fdec0 0x7fa93c195850 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7fa934007f10 tx=0x7fa93400d3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:28:47.411 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:47.403+0000 7fa938ff9700 1 -- 172.21.15.98:0/963357868 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa934017070 con 0x7fa93c0fdec0 2023-07-14T19:28:47.412 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:47.403+0000 7fa938ff9700 1 -- 172.21.15.98:0/963357868 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fa93400f040 con 0x7fa93c0fdec0 2023-07-14T19:28:47.412 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:47.403+0000 7fa941eb3700 1 -- 172.21.15.98:0/963357868 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa93c19d980 con 0x7fa93c0fdec0 2023-07-14T19:28:47.412 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:47.403+0000 7fa938ff9700 1 -- 172.21.15.98:0/963357868 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa934004580 con 0x7fa93c0fdec0 2023-07-14T19:28:47.412 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:47.403+0000 7fa941eb3700 1 -- 172.21.15.98:0/963357868 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa93c19dee0 con 0x7fa93c0fdec0 2023-07-14T19:28:47.412 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:47.404+0000 7fa941eb3700 1 -- 172.21.15.98:0/963357868 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa93c107210 con 0x7fa93c0fdec0 2023-07-14T19:28:47.412 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:47.405+0000 7fa938ff9700 1 -- 172.21.15.98:0/963357868 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 16) v1 ==== 135715+0+0 (secure 0 0 0) 0x7fa9340046e0 con 0x7fa93c0fdec0 2023-07-14T19:28:47.413 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:47.406+0000 7fa938ff9700 1 --2- 172.21.15.98:0/963357868 >> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] conn(0x7fa9280a1220 0x7fa9280a36d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:28:47.413 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:47.406+0000 7fa93affd700 1 --2- 172.21.15.98:0/963357868 >> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] conn(0x7fa9280a1220 0x7fa9280a36d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:28:47.413 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:47.406+0000 7fa938ff9700 1 -- 172.21.15.98:0/963357868 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(79..79 src has 1..79) v4 ==== 7127+0+0 (secure 0 0 0) 0x7fa9340cba80 con 0x7fa93c0fdec0 2023-07-14T19:28:47.413 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:47.406+0000 7fa93affd700 1 --2- 172.21.15.98:0/963357868 >> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] conn(0x7fa9280a1220 0x7fa9280a36d0 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7fa924009c60 tx=0x7fa924009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:28:47.413 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:47.410+0000 7fa938ff9700 1 -- 172.21.15.98:0/963357868 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7fa934095670 con 0x7fa93c0fdec0 2023-07-14T19:28:47.441 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:28:46 smithi101 ceph-mon[128807]: pgmap v241: 1 pgs: 1 active+clean; 577 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-14T19:28:47.441 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:28:46 smithi101 ceph-mon[128807]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' 2023-07-14T19:28:47.442 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:28:46 smithi101 ceph-mon[128807]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' 2023-07-14T19:28:47.442 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:28:46 smithi101 ceph-mon[128807]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' 2023-07-14T19:28:47.442 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:28:46 smithi101 ceph-mon[128807]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-07-14T19:28:47.442 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:28:46 smithi101 ceph-mon[128807]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-07-14T19:28:47.442 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:28:46 smithi101 ceph-mon[128807]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T19:28:47.641 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:47.641+0000 7fa941eb3700 1 -- 172.21.15.98:0/963357868 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa93c02caa0 con 0x7fa93c0fdec0 2023-07-14T19:28:47.642 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:47.641+0000 7fa938ff9700 1 -- 172.21.15.98:0/963357868 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 9 v9) v1 ==== 93+0+6565 (secure 0 0 0) 0x7fa934098c90 con 0x7fa93c0fdec0 2023-07-14T19:28:47.643 INFO:teuthology.orchestra.run.smithi098.stdout: 2023-07-14T19:28:47.643 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":9,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14565,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6835/586280675","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6834","nonce":586280675},{"type":"v1","addr":"172.21.15.98:6835","nonce":586280675}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":4},{"gid":14577,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6837/487577257","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6836","nonce":487577257},{"type":"v1","addr":"172.21.15.98:6837","nonce":487577257}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":5},{"gid":24421,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.139:6833/3505477820","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6832","nonce":3505477820},{"type":"v1","addr":"172.21.15.139:6833","nonce":3505477820}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":24437,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.101:6833/3474460171","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.101:6832","nonce":3474460171},{"type":"v1","addr":"172.21.15.101:6833","nonce":3474460171}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":24455,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.101:6835/1583264648","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.101:6834","nonce":1583264648},{"type":"v1","addr":"172.21.15.101:6835","nonce":1583264648}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":6},{"gid":24460,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.139:6835/2315281724","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6834","nonce":2315281724},{"type":"v1","addr":"172.21.15.139:6835","nonce":2315281724}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":7},{"gid":24469,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.139:6837/3597660552","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6836","nonce":3597660552},{"type":"v1","addr":"172.21.15.139:6837","nonce":3597660552}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":8},{"gid":24479,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.101:6837/485741528","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.101:6836","nonce":485741528},{"type":"v1","addr":"172.21.15.101:6837","nonce":485741528}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":9}],"filesystems":[]} 2023-07-14T19:28:47.645 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:47.643+0000 7fa941eb3700 1 -- 172.21.15.98:0/963357868 >> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] conn(0x7fa9280a1220 msgr2=0x7fa9280a36d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:28:47.645 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:47.643+0000 7fa941eb3700 1 --2- 172.21.15.98:0/963357868 >> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] conn(0x7fa9280a1220 0x7fa9280a36d0 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7fa924009c60 tx=0x7fa924009260 comp rx=0 tx=0).stop 2023-07-14T19:28:47.645 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:47.643+0000 7fa941eb3700 1 -- 172.21.15.98:0/963357868 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa93c0fdec0 msgr2=0x7fa93c195850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:28:47.646 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:47.643+0000 7fa941eb3700 1 --2- 172.21.15.98:0/963357868 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa93c0fdec0 0x7fa93c195850 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7fa934007f10 tx=0x7fa93400d3b0 comp rx=0 tx=0).stop 2023-07-14T19:28:47.646 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:47.643+0000 7fa941eb3700 1 -- 172.21.15.98:0/963357868 shutdown_connections 2023-07-14T19:28:47.646 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:47.643+0000 7fa941eb3700 1 --2- 172.21.15.98:0/963357868 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7fa93c1018c0 0x7fa93c19ce00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:28:47.646 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:47.643+0000 7fa941eb3700 1 --2- 172.21.15.98:0/963357868 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa93c0fdec0 0x7fa93c195850 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:28:47.647 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:47.643+0000 7fa941eb3700 1 --2- 172.21.15.98:0/963357868 >> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] conn(0x7fa9280a1220 0x7fa9280a36d0 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:28:47.648 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:47.643+0000 7fa941eb3700 1 --2- 172.21.15.98:0/963357868 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fa93c100f30 0x7fa93c195d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:28:47.648 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:47.643+0000 7fa941eb3700 1 -- 172.21.15.98:0/963357868 >> 172.21.15.98:0/963357868 conn(0x7fa93c0f98a0 msgr2=0x7fa93c104350 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:28:47.648 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:47.643+0000 7fa941eb3700 1 -- 172.21.15.98:0/963357868 shutdown_connections 2023-07-14T19:28:47.648 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:47.643+0000 7fa941eb3700 1 -- 172.21.15.98:0/963357868 wait complete. 2023-07-14T19:28:47.649 INFO:teuthology.orchestra.run.smithi098.stderr:dumped fsmap epoch 9 2023-07-14T19:28:48.220 INFO:journalctl@ceph.mds.i.smithi139.stdout:Jul 14 19:28:48 smithi139 systemd[1]: Starting Ceph mds.i for 9adea648-227b-11ee-9b33-001a4aab830c... 2023-07-14T19:28:48.221 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:28:47 smithi139 ceph-mon[129293]: Deploying daemon mds.i on smithi139 2023-07-14T19:28:48.221 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:28:47 smithi139 ceph-mon[129293]: mds.? [v2:172.21.15.101:6836/485741528,v1:172.21.15.101:6837/485741528] up:boot 2023-07-14T19:28:48.221 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:28:47 smithi139 ceph-mon[129293]: fsmap 8 up:standby 2023-07-14T19:28:48.221 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:28:47 smithi139 ceph-mon[129293]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2023-07-14T19:28:48.221 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:28:47 smithi139 ceph-mon[129293]: from='client.? 172.21.15.98:0/963357868' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-07-14T19:28:48.281 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:28:47 smithi098 ceph-mon[127486]: Deploying daemon mds.i on smithi139 2023-07-14T19:28:48.281 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:28:47 smithi098 ceph-mon[127486]: mds.? [v2:172.21.15.101:6836/485741528,v1:172.21.15.101:6837/485741528] up:boot 2023-07-14T19:28:48.282 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:28:47 smithi098 ceph-mon[127486]: fsmap 8 up:standby 2023-07-14T19:28:48.282 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:28:47 smithi098 ceph-mon[127486]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2023-07-14T19:28:48.282 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:28:47 smithi098 ceph-mon[127486]: from='client.? 172.21.15.98:0/963357868' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-07-14T19:28:48.441 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:28:47 smithi101 ceph-mon[128807]: Deploying daemon mds.i on smithi139 2023-07-14T19:28:48.441 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:28:47 smithi101 ceph-mon[128807]: mds.? [v2:172.21.15.101:6836/485741528,v1:172.21.15.101:6837/485741528] up:boot 2023-07-14T19:28:48.442 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:28:47 smithi101 ceph-mon[128807]: fsmap 8 up:standby 2023-07-14T19:28:48.442 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:28:47 smithi101 ceph-mon[128807]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2023-07-14T19:28:48.442 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:28:47 smithi101 ceph-mon[128807]: from='client.? 172.21.15.98:0/963357868' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-07-14T19:28:48.732 INFO:journalctl@ceph.mds.i.smithi139.stdout:Jul 14 19:28:48 smithi139 podman[155527]: 2023-07-14T19:28:49.220 INFO:journalctl@ceph.mds.i.smithi139.stdout:Jul 14 19:28:48 smithi139 ceph-mds[155567]: -- 172.21.15.139:0/145235849 <== mon.1 v2:172.21.15.139:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55a2545cf1e0 con 0x55a2545a5000 2023-07-14T19:28:49.220 INFO:journalctl@ceph.mds.i.smithi139.stdout:Jul 14 19:28:48 smithi139 ceph-9adea648-227b-11ee-9b33-001a4aab830c-mds-i[155543]: starting mds.i at 2023-07-14T19:28:49.221 INFO:journalctl@ceph.mds.i.smithi139.stdout:Jul 14 19:28:48 smithi139 bash[155527]: d07608af6635cf293f73171a977a41fc29c266b312ca4ff912746904e10fbe06 2023-07-14T19:28:49.221 INFO:journalctl@ceph.mds.i.smithi139.stdout:Jul 14 19:28:48 smithi139 systemd[1]: Started Ceph mds.i for 9adea648-227b-11ee-9b33-001a4aab830c. 2023-07-14T19:28:49.221 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:28:48 smithi139 ceph-mon[129293]: pgmap v242: 1 pgs: 1 active+clean; 577 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-14T19:28:49.221 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:28:48 smithi139 ceph-mon[129293]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' 2023-07-14T19:28:49.221 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:28:48 smithi139 ceph-mon[129293]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' 2023-07-14T19:28:49.221 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:28:48 smithi139 ceph-mon[129293]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' 2023-07-14T19:28:49.222 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:28:48 smithi139 ceph-mon[129293]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-07-14T19:28:49.222 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:28:48 smithi139 ceph-mon[129293]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-07-14T19:28:49.222 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:28:48 smithi139 ceph-mon[129293]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T19:28:49.281 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:28:48 smithi098 ceph-mon[127486]: pgmap v242: 1 pgs: 1 active+clean; 577 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-14T19:28:49.281 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:28:48 smithi098 ceph-mon[127486]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' 2023-07-14T19:28:49.281 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:28:48 smithi098 ceph-mon[127486]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' 2023-07-14T19:28:49.282 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:28:48 smithi098 ceph-mon[127486]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' 2023-07-14T19:28:49.282 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:28:48 smithi098 ceph-mon[127486]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-07-14T19:28:49.282 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:28:48 smithi098 ceph-mon[127486]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-07-14T19:28:49.282 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:28:48 smithi098 ceph-mon[127486]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T19:28:49.441 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:28:48 smithi101 ceph-mon[128807]: pgmap v242: 1 pgs: 1 active+clean; 577 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-14T19:28:49.442 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:28:48 smithi101 ceph-mon[128807]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' 2023-07-14T19:28:49.442 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:28:48 smithi101 ceph-mon[128807]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' 2023-07-14T19:28:49.442 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:28:48 smithi101 ceph-mon[128807]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' 2023-07-14T19:28:49.442 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:28:48 smithi101 ceph-mon[128807]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-07-14T19:28:49.442 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:28:48 smithi101 ceph-mon[128807]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-07-14T19:28:49.443 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:28:48 smithi101 ceph-mon[128807]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T19:28:49.658 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-07-14T19:28:49.791 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:49.790+0000 7fbe0a801700 1 -- 172.21.15.98:0/2979661923 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbe04100e20 msgr2=0x7fbe04101270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:28:49.792 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:49.790+0000 7fbe0a801700 1 --2- 172.21.15.98:0/2979661923 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbe04100e20 0x7fbe04101270 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7fbdf80098f0 tx=0x7fbdf8009c00 comp rx=0 tx=0).stop 2023-07-14T19:28:49.792 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:49.790+0000 7fbe0a801700 1 -- 172.21.15.98:0/2979661923 shutdown_connections 2023-07-14T19:28:49.792 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:49.790+0000 7fbe0a801700 1 --2- 172.21.15.98:0/2979661923 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fbe041017b0 0x7fbe04108070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:28:49.792 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:49.790+0000 7fbe0a801700 1 --2- 172.21.15.98:0/2979661923 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7fbe040ffc30 0x7fbe04100040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:28:49.792 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:49.790+0000 7fbe0a801700 1 --2- 172.21.15.98:0/2979661923 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbe04100e20 0x7fbe04101270 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:28:49.793 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:49.790+0000 7fbe0a801700 1 -- 172.21.15.98:0/2979661923 >> 172.21.15.98:0/2979661923 conn(0x7fbe040fb230 msgr2=0x7fbe040fd680 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:28:49.793 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:49.790+0000 7fbe0a801700 1 -- 172.21.15.98:0/2979661923 shutdown_connections 2023-07-14T19:28:49.793 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:49.791+0000 7fbe0a801700 1 -- 172.21.15.98:0/2979661923 wait complete. 2023-07-14T19:28:49.793 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:49.791+0000 7fbe0a801700 1 Processor -- start 2023-07-14T19:28:49.793 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:49.791+0000 7fbe0a801700 1 -- start start 2023-07-14T19:28:49.795 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:49.792+0000 7fbe0a801700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fbe040ffc30 0x7fbe04195970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:28:49.795 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:49.792+0000 7fbe0a801700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbe04100e20 0x7fbe04195eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:28:49.795 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:49.792+0000 7fbe0a801700 1 --2- >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7fbe041017b0 0x7fbe0419cf20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:28:49.796 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:49.792+0000 7fbe0a801700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbe0419d4f0 con 0x7fbe04100e20 2023-07-14T19:28:49.796 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:49.792+0000 7fbe0a801700 1 -- --> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbe0419d630 con 0x7fbe041017b0 2023-07-14T19:28:49.796 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:49.792+0000 7fbe0a801700 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbe0419d7c0 con 0x7fbe040ffc30 2023-07-14T19:28:49.796 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:49.792+0000 7fbe037fe700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbe04100e20 0x7fbe04195eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:28:49.796 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:49.792+0000 7fbe037fe700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbe04100e20 0x7fbe04195eb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:45492/0 (socket says 172.21.15.98:45492) 2023-07-14T19:28:49.796 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:49.792+0000 7fbe037fe700 1 -- 172.21.15.98:0/2349070787 learned_addr learned my addr 172.21.15.98:0/2349070787 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-07-14T19:28:49.796 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:49.792+0000 7fbe03fff700 1 --2- 172.21.15.98:0/2349070787 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fbe040ffc30 0x7fbe04195970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:28:49.797 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:49.792+0000 7fbe08d9e700 1 --2- 172.21.15.98:0/2349070787 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7fbe041017b0 0x7fbe0419cf20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:28:49.797 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:49.792+0000 7fbe037fe700 1 -- 172.21.15.98:0/2349070787 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fbe040ffc30 msgr2=0x7fbe04195970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:28:49.797 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:49.792+0000 7fbe037fe700 1 --2- 172.21.15.98:0/2349070787 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fbe040ffc30 0x7fbe04195970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:28:49.797 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:49.792+0000 7fbe037fe700 1 -- 172.21.15.98:0/2349070787 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7fbe041017b0 msgr2=0x7fbe0419cf20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:28:49.797 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:49.792+0000 7fbe037fe700 1 --2- 172.21.15.98:0/2349070787 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7fbe041017b0 0x7fbe0419cf20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:28:49.797 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:49.792+0000 7fbe037fe700 1 -- 172.21.15.98:0/2349070787 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbdf8009520 con 0x7fbe04100e20 2023-07-14T19:28:49.797 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:49.792+0000 7fbe03fff700 1 --2- 172.21.15.98:0/2349070787 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fbe040ffc30 0x7fbe04195970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-07-14T19:28:49.797 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:49.792+0000 7fbe037fe700 1 --2- 172.21.15.98:0/2349070787 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbe04100e20 0x7fbe04195eb0 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7fbdf8005e90 tx=0x7fbdf800b7e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:28:49.798 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:49.792+0000 7fbe017fa700 1 -- 172.21.15.98:0/2349070787 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbdf801c070 con 0x7fbe04100e20 2023-07-14T19:28:49.798 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:49.792+0000 7fbe0a801700 1 -- 172.21.15.98:0/2349070787 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbe0419da40 con 0x7fbe04100e20 2023-07-14T19:28:49.798 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:49.792+0000 7fbe017fa700 1 -- 172.21.15.98:0/2349070787 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fbdf800f7f0 con 0x7fbe04100e20 2023-07-14T19:28:49.798 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:49.792+0000 7fbe017fa700 1 -- 172.21.15.98:0/2349070787 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbdf8004bd0 con 0x7fbe04100e20 2023-07-14T19:28:49.798 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:49.792+0000 7fbe0a801700 1 -- 172.21.15.98:0/2349070787 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbe0419dec0 con 0x7fbe04100e20 2023-07-14T19:28:49.798 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:49.793+0000 7fbe0a801700 1 -- 172.21.15.98:0/2349070787 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbe0410b450 con 0x7fbe04100e20 2023-07-14T19:28:49.798 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:49.794+0000 7fbe017fa700 1 -- 172.21.15.98:0/2349070787 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 16) v1 ==== 135715+0+0 (secure 0 0 0) 0x7fbdf8020070 con 0x7fbe04100e20 2023-07-14T19:28:49.798 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:49.795+0000 7fbe017fa700 1 --2- 172.21.15.98:0/2349070787 >> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] conn(0x7fbdec0a11a0 0x7fbdec0a3650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:28:49.799 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:49.795+0000 7fbe03fff700 1 --2- 172.21.15.98:0/2349070787 >> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] conn(0x7fbdec0a11a0 0x7fbdec0a3650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:28:49.799 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:49.795+0000 7fbe017fa700 1 -- 172.21.15.98:0/2349070787 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(79..79 src has 1..79) v4 ==== 7127+0+0 (secure 0 0 0) 0x7fbdf80cda50 con 0x7fbe04100e20 2023-07-14T19:28:49.799 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:49.795+0000 7fbe03fff700 1 --2- 172.21.15.98:0/2349070787 >> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] conn(0x7fbdec0a11a0 0x7fbdec0a3650 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7fbe04100c80 tx=0x7fbdf4009310 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:28:49.800 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:49.798+0000 7fbe017fa700 1 -- 172.21.15.98:0/2349070787 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7fbdf80976d0 con 0x7fbe04100e20 2023-07-14T19:28:49.999 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:28:49 smithi098 ceph-mon[127486]: Deploying daemon mds.g on smithi098 2023-07-14T19:28:50.000 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:28:49 smithi098 ceph-mon[127486]: mds.? [v2:172.21.15.139:6838/1258854143,v1:172.21.15.139:6839/1258854143] up:boot 2023-07-14T19:28:50.000 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:28:49 smithi098 ceph-mon[127486]: fsmap 9 up:standby 2023-07-14T19:28:50.000 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:28:49 smithi098 ceph-mon[127486]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2023-07-14T19:28:50.054 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:50.054+0000 7fbe0a801700 1 -- 172.21.15.98:0/2349070787 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fbe04106b10 con 0x7fbe04100e20 2023-07-14T19:28:50.055 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:50.054+0000 7fbe017fa700 1 -- 172.21.15.98:0/2349070787 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) v1 ==== 94+0+7327 (secure 0 0 0) 0x7fbdf809acf0 con 0x7fbe04100e20 2023-07-14T19:28:50.055 INFO:teuthology.orchestra.run.smithi098.stdout: 2023-07-14T19:28:50.055 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":10,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14565,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6835/586280675","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6834","nonce":586280675},{"type":"v1","addr":"172.21.15.98:6835","nonce":586280675}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":4},{"gid":14577,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6837/487577257","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6836","nonce":487577257},{"type":"v1","addr":"172.21.15.98:6837","nonce":487577257}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":5},{"gid":24421,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.139:6833/3505477820","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6832","nonce":3505477820},{"type":"v1","addr":"172.21.15.139:6833","nonce":3505477820}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":24437,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.101:6833/3474460171","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.101:6832","nonce":3474460171},{"type":"v1","addr":"172.21.15.101:6833","nonce":3474460171}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":24455,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.101:6835/1583264648","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.101:6834","nonce":1583264648},{"type":"v1","addr":"172.21.15.101:6835","nonce":1583264648}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":6},{"gid":24460,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.139:6835/2315281724","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6834","nonce":2315281724},{"type":"v1","addr":"172.21.15.139:6835","nonce":2315281724}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":7},{"gid":24469,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.139:6837/3597660552","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6836","nonce":3597660552},{"type":"v1","addr":"172.21.15.139:6837","nonce":3597660552}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":8},{"gid":24479,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.101:6837/485741528","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.101:6836","nonce":485741528},{"type":"v1","addr":"172.21.15.101:6837","nonce":485741528}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":9},{"gid":24493,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.139:6839/1258854143","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6838","nonce":1258854143},{"type":"v1","addr":"172.21.15.139:6839","nonce":1258854143}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10}],"filesystems":[]} 2023-07-14T19:28:50.058 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:50.056+0000 7fbe0a801700 1 -- 172.21.15.98:0/2349070787 >> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] conn(0x7fbdec0a11a0 msgr2=0x7fbdec0a3650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:28:50.058 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:50.056+0000 7fbe0a801700 1 --2- 172.21.15.98:0/2349070787 >> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] conn(0x7fbdec0a11a0 0x7fbdec0a3650 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7fbe04100c80 tx=0x7fbdf4009310 comp rx=0 tx=0).stop 2023-07-14T19:28:50.059 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:50.056+0000 7fbe0a801700 1 -- 172.21.15.98:0/2349070787 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbe04100e20 msgr2=0x7fbe04195eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:28:50.059 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:50.056+0000 7fbe0a801700 1 --2- 172.21.15.98:0/2349070787 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbe04100e20 0x7fbe04195eb0 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7fbdf8005e90 tx=0x7fbdf800b7e0 comp rx=0 tx=0).stop 2023-07-14T19:28:50.059 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:50.056+0000 7fbe0a801700 1 -- 172.21.15.98:0/2349070787 shutdown_connections 2023-07-14T19:28:50.060 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:50.056+0000 7fbe0a801700 1 --2- 172.21.15.98:0/2349070787 >> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] conn(0x7fbdec0a11a0 0x7fbdec0a3650 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:28:50.060 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:50.056+0000 7fbe0a801700 1 --2- 172.21.15.98:0/2349070787 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fbe040ffc30 0x7fbe04195970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:28:50.061 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:50.056+0000 7fbe0a801700 1 --2- 172.21.15.98:0/2349070787 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7fbe041017b0 0x7fbe0419cf20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:28:50.061 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:50.056+0000 7fbe0a801700 1 --2- 172.21.15.98:0/2349070787 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbe04100e20 0x7fbe04195eb0 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:28:50.061 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:50.056+0000 7fbe0a801700 1 -- 172.21.15.98:0/2349070787 >> 172.21.15.98:0/2349070787 conn(0x7fbe040fb230 msgr2=0x7fbe040fd440 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:28:50.062 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:50.056+0000 7fbe0a801700 1 -- 172.21.15.98:0/2349070787 shutdown_connections 2023-07-14T19:28:50.063 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:50.056+0000 7fbe0a801700 1 -- 172.21.15.98:0/2349070787 wait complete. 2023-07-14T19:28:50.063 INFO:teuthology.orchestra.run.smithi098.stderr:dumped fsmap epoch 10 2023-07-14T19:28:50.220 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:28:49 smithi139 ceph-mon[129293]: Deploying daemon mds.g on smithi098 2023-07-14T19:28:50.221 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:28:49 smithi139 ceph-mon[129293]: mds.? [v2:172.21.15.139:6838/1258854143,v1:172.21.15.139:6839/1258854143] up:boot 2023-07-14T19:28:50.221 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:28:49 smithi139 ceph-mon[129293]: fsmap 9 up:standby 2023-07-14T19:28:50.221 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:28:49 smithi139 ceph-mon[129293]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2023-07-14T19:28:50.281 INFO:journalctl@ceph.mds.g.smithi098.stdout:Jul 14 19:28:50 smithi098 systemd[1]: Starting Ceph mds.g for 9adea648-227b-11ee-9b33-001a4aab830c... 2023-07-14T19:28:50.441 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:28:49 smithi101 ceph-mon[128807]: Deploying daemon mds.g on smithi098 2023-07-14T19:28:50.442 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:28:49 smithi101 ceph-mon[128807]: mds.? [v2:172.21.15.139:6838/1258854143,v1:172.21.15.139:6839/1258854143] up:boot 2023-07-14T19:28:50.442 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:28:49 smithi101 ceph-mon[128807]: fsmap 9 up:standby 2023-07-14T19:28:50.442 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:28:49 smithi101 ceph-mon[128807]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2023-07-14T19:28:50.834 INFO:journalctl@ceph.mds.g.smithi098.stdout:Jul 14 19:28:50 smithi098 podman[161748]: 2023-07-14T19:28:51.220 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:28:50 smithi139 ceph-mon[129293]: pgmap v243: 1 pgs: 1 active+clean; 577 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-14T19:28:51.220 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:28:50 smithi139 ceph-mon[129293]: from='client.? 172.21.15.98:0/2349070787' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-07-14T19:28:51.281 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:28:50 smithi098 ceph-mon[127486]: pgmap v243: 1 pgs: 1 active+clean; 577 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-14T19:28:51.281 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:28:50 smithi098 ceph-mon[127486]: from='client.? 172.21.15.98:0/2349070787' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-07-14T19:28:51.282 INFO:journalctl@ceph.mds.g.smithi098.stdout:Jul 14 19:28:50 smithi098 ceph-mds[161789]: -- 172.21.15.98:0/575318308 <== mon.0 v2:172.21.15.98:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x5557766211e0 con 0x5557765f6800 2023-07-14T19:28:51.282 INFO:journalctl@ceph.mds.g.smithi098.stdout:Jul 14 19:28:50 smithi098 ceph-9adea648-227b-11ee-9b33-001a4aab830c-mds-g[161764]: starting mds.g at 2023-07-14T19:28:51.282 INFO:journalctl@ceph.mds.g.smithi098.stdout:Jul 14 19:28:50 smithi098 bash[161748]: 13afeea7158e5b73de7a6ac3904a47eca05bd3df55db3d32592cc75c0146f48c 2023-07-14T19:28:51.282 INFO:journalctl@ceph.mds.g.smithi098.stdout:Jul 14 19:28:50 smithi098 systemd[1]: Started Ceph mds.g for 9adea648-227b-11ee-9b33-001a4aab830c. 2023-07-14T19:28:51.441 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:28:50 smithi101 ceph-mon[128807]: pgmap v243: 1 pgs: 1 active+clean; 577 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-14T19:28:51.442 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:28:50 smithi101 ceph-mon[128807]: from='client.? 172.21.15.98:0/2349070787' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-07-14T19:28:52.072 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-07-14T19:28:52.173 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:28:51 smithi098 ceph-mon[127486]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' 2023-07-14T19:28:52.173 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:28:51 smithi098 ceph-mon[127486]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' 2023-07-14T19:28:52.174 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:28:51 smithi098 ceph-mon[127486]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' 2023-07-14T19:28:52.174 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:28:51 smithi098 ceph-mon[127486]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-07-14T19:28:52.174 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:28:51 smithi098 ceph-mon[127486]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-07-14T19:28:52.176 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:28:51 smithi098 ceph-mon[127486]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T19:28:52.177 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:28:51 smithi098 ceph-mon[127486]: mds.? [v2:172.21.15.98:6838/4085731209,v1:172.21.15.98:6839/4085731209] up:boot 2023-07-14T19:28:52.177 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:28:51 smithi098 ceph-mon[127486]: fsmap 10 up:standby 2023-07-14T19:28:52.177 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:28:51 smithi098 ceph-mon[127486]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2023-07-14T19:28:52.209 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:52.207+0000 7fb91637d700 1 -- 172.21.15.98:0/649548665 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb9100feb30 msgr2=0x7fb9100fef40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:28:52.209 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:52.207+0000 7fb91637d700 1 --2- 172.21.15.98:0/649548665 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb9100feb30 0x7fb9100fef40 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7fb900009990 tx=0x7fb900009ca0 comp rx=0 tx=0).stop 2023-07-14T19:28:52.210 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:52.207+0000 7fb91637d700 1 -- 172.21.15.98:0/649548665 shutdown_connections 2023-07-14T19:28:52.211 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:52.207+0000 7fb91637d700 1 --2- 172.21.15.98:0/649548665 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7fb9101006b0 0x7fb910106f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:28:52.211 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:52.207+0000 7fb91637d700 1 --2- 172.21.15.98:0/649548665 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb9100feb30 0x7fb9100fef40 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:28:52.211 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:52.207+0000 7fb91637d700 1 --2- 172.21.15.98:0/649548665 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fb9100ffd20 0x7fb910100170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:28:52.211 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:52.207+0000 7fb91637d700 1 -- 172.21.15.98:0/649548665 >> 172.21.15.98:0/649548665 conn(0x7fb9100fa110 msgr2=0x7fb9100fc580 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:28:52.211 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:52.207+0000 7fb91637d700 1 -- 172.21.15.98:0/649548665 shutdown_connections 2023-07-14T19:28:52.211 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:52.207+0000 7fb91637d700 1 -- 172.21.15.98:0/649548665 wait complete. 2023-07-14T19:28:52.212 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:52.209+0000 7fb91637d700 1 Processor -- start 2023-07-14T19:28:52.212 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:52.209+0000 7fb91637d700 1 -- start start 2023-07-14T19:28:52.215 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:52.209+0000 7fb91637d700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fb9100feb30 0x7fb910194800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:28:52.215 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:52.209+0000 7fb91637d700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb9100ffd20 0x7fb910194d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:28:52.215 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:52.209+0000 7fb91637d700 1 --2- >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7fb9101006b0 0x7fb91019bdb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:28:52.215 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:52.209+0000 7fb91637d700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb91019c380 con 0x7fb9100ffd20 2023-07-14T19:28:52.215 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:52.209+0000 7fb91637d700 1 -- --> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb91019c4c0 con 0x7fb9101006b0 2023-07-14T19:28:52.215 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:52.209+0000 7fb91637d700 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb91019c650 con 0x7fb9100feb30 2023-07-14T19:28:52.216 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:52.209+0000 7fb90f7fe700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb9100ffd20 0x7fb910194d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:28:52.216 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:52.209+0000 7fb90f7fe700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb9100ffd20 0x7fb910194d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:45518/0 (socket says 172.21.15.98:45518) 2023-07-14T19:28:52.216 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:52.209+0000 7fb90f7fe700 1 -- 172.21.15.98:0/3169276804 learned_addr learned my addr 172.21.15.98:0/3169276804 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-07-14T19:28:52.216 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:52.209+0000 7fb91491a700 1 --2- 172.21.15.98:0/3169276804 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7fb9101006b0 0x7fb91019bdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:28:52.216 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:52.209+0000 7fb90ffff700 1 --2- 172.21.15.98:0/3169276804 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fb9100feb30 0x7fb910194800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:28:52.216 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:52.210+0000 7fb90f7fe700 1 -- 172.21.15.98:0/3169276804 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fb9100feb30 msgr2=0x7fb910194800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:28:52.216 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:52.210+0000 7fb90f7fe700 1 --2- 172.21.15.98:0/3169276804 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fb9100feb30 0x7fb910194800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:28:52.217 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:52.210+0000 7fb90f7fe700 1 -- 172.21.15.98:0/3169276804 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7fb9101006b0 msgr2=0x7fb91019bdb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:28:52.217 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:52.210+0000 7fb90f7fe700 1 --2- 172.21.15.98:0/3169276804 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7fb9101006b0 0x7fb91019bdb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:28:52.217 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:52.210+0000 7fb90f7fe700 1 -- 172.21.15.98:0/3169276804 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb9000095f0 con 0x7fb9100ffd20 2023-07-14T19:28:52.217 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:52.210+0000 7fb90ffff700 1 --2- 172.21.15.98:0/3169276804 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fb9100feb30 0x7fb910194800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-07-14T19:28:52.218 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:52.210+0000 7fb90f7fe700 1 --2- 172.21.15.98:0/3169276804 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb9100ffd20 0x7fb910194d40 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7fb90400eab0 tx=0x7fb90400edc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:28:52.218 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:52.210+0000 7fb90d7fa700 1 -- 172.21.15.98:0/3169276804 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb904013070 con 0x7fb9100ffd20 2023-07-14T19:28:52.218 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:52.210+0000 7fb91637d700 1 -- 172.21.15.98:0/3169276804 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb910068810 con 0x7fb9100ffd20 2023-07-14T19:28:52.218 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:52.210+0000 7fb90d7fa700 1 -- 172.21.15.98:0/3169276804 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fb90400be80 con 0x7fb9100ffd20 2023-07-14T19:28:52.218 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:52.210+0000 7fb90d7fa700 1 -- 172.21.15.98:0/3169276804 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb9040163b0 con 0x7fb9100ffd20 2023-07-14T19:28:52.218 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:52.210+0000 7fb91637d700 1 -- 172.21.15.98:0/3169276804 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb910068d20 con 0x7fb9100ffd20 2023-07-14T19:28:52.218 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:52.211+0000 7fb91637d700 1 -- 172.21.15.98:0/3169276804 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb91010a300 con 0x7fb9100ffd20 2023-07-14T19:28:52.219 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:52.213+0000 7fb90d7fa700 1 -- 172.21.15.98:0/3169276804 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 16) v1 ==== 135715+0+0 (secure 0 0 0) 0x7fb904017580 con 0x7fb9100ffd20 2023-07-14T19:28:52.219 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:52.213+0000 7fb90d7fa700 1 --2- 172.21.15.98:0/3169276804 >> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] conn(0x7fb8f80a11a0 0x7fb8f80a3650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:28:52.219 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:52.213+0000 7fb90ffff700 1 --2- 172.21.15.98:0/3169276804 >> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] conn(0x7fb8f80a11a0 0x7fb8f80a3650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:28:52.219 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:52.213+0000 7fb90d7fa700 1 -- 172.21.15.98:0/3169276804 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(79..79 src has 1..79) v4 ==== 7127+0+0 (secure 0 0 0) 0x7fb9040cde00 con 0x7fb9100ffd20 2023-07-14T19:28:52.219 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:52.214+0000 7fb90ffff700 1 --2- 172.21.15.98:0/3169276804 >> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] conn(0x7fb8f80a11a0 0x7fb8f80a3650 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7fb90000b2e0 tx=0x7fb900005e80 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:28:52.220 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:52.216+0000 7fb90d7fa700 1 -- 172.21.15.98:0/3169276804 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7fb9040979f0 con 0x7fb9100ffd20 2023-07-14T19:28:52.441 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:28:51 smithi101 ceph-mon[128807]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' 2023-07-14T19:28:52.442 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:28:51 smithi101 ceph-mon[128807]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' 2023-07-14T19:28:52.442 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:28:51 smithi101 ceph-mon[128807]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' 2023-07-14T19:28:52.442 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:28:51 smithi101 ceph-mon[128807]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-07-14T19:28:52.442 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:28:51 smithi101 ceph-mon[128807]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-07-14T19:28:52.442 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:28:51 smithi101 ceph-mon[128807]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T19:28:52.442 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:28:51 smithi101 ceph-mon[128807]: mds.? [v2:172.21.15.98:6838/4085731209,v1:172.21.15.98:6839/4085731209] up:boot 2023-07-14T19:28:52.442 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:28:51 smithi101 ceph-mon[128807]: fsmap 10 up:standby 2023-07-14T19:28:52.443 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:28:51 smithi101 ceph-mon[128807]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2023-07-14T19:28:52.471 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:28:51 smithi139 ceph-mon[129293]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' 2023-07-14T19:28:52.471 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:28:51 smithi139 ceph-mon[129293]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' 2023-07-14T19:28:52.471 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:28:51 smithi139 ceph-mon[129293]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' 2023-07-14T19:28:52.471 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:28:51 smithi139 ceph-mon[129293]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-07-14T19:28:52.472 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:28:51 smithi139 ceph-mon[129293]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-07-14T19:28:52.472 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:28:51 smithi139 ceph-mon[129293]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T19:28:52.472 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:28:51 smithi139 ceph-mon[129293]: mds.? [v2:172.21.15.98:6838/4085731209,v1:172.21.15.98:6839/4085731209] up:boot 2023-07-14T19:28:52.472 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:28:51 smithi139 ceph-mon[129293]: fsmap 10 up:standby 2023-07-14T19:28:52.473 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:28:51 smithi139 ceph-mon[129293]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2023-07-14T19:28:52.475 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:52.474+0000 7fb91637d700 1 -- 172.21.15.98:0/3169276804 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb910105fe0 con 0x7fb9100ffd20 2023-07-14T19:28:52.475 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:52.475+0000 7fb90d7fa700 1 -- 172.21.15.98:0/3169276804 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 11 v11) v1 ==== 94+0+8085 (secure 0 0 0) 0x7fb90409b010 con 0x7fb9100ffd20 2023-07-14T19:28:52.476 INFO:teuthology.orchestra.run.smithi098.stdout: 2023-07-14T19:28:52.476 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":11,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14565,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6835/586280675","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6834","nonce":586280675},{"type":"v1","addr":"172.21.15.98:6835","nonce":586280675}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":4},{"gid":14577,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6837/487577257","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6836","nonce":487577257},{"type":"v1","addr":"172.21.15.98:6837","nonce":487577257}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":5},{"gid":14634,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6839/4085731209","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6838","nonce":4085731209},{"type":"v1","addr":"172.21.15.98:6839","nonce":4085731209}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":11},{"gid":24421,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.139:6833/3505477820","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6832","nonce":3505477820},{"type":"v1","addr":"172.21.15.139:6833","nonce":3505477820}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":24437,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.101:6833/3474460171","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.101:6832","nonce":3474460171},{"type":"v1","addr":"172.21.15.101:6833","nonce":3474460171}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":24455,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.101:6835/1583264648","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.101:6834","nonce":1583264648},{"type":"v1","addr":"172.21.15.101:6835","nonce":1583264648}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":6},{"gid":24460,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.139:6835/2315281724","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6834","nonce":2315281724},{"type":"v1","addr":"172.21.15.139:6835","nonce":2315281724}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":7},{"gid":24469,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.139:6837/3597660552","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6836","nonce":3597660552},{"type":"v1","addr":"172.21.15.139:6837","nonce":3597660552}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":8},{"gid":24479,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.101:6837/485741528","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.101:6836","nonce":485741528},{"type":"v1","addr":"172.21.15.101:6837","nonce":485741528}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":9},{"gid":24493,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.139:6839/1258854143","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6838","nonce":1258854143},{"type":"v1","addr":"172.21.15.139:6839","nonce":1258854143}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10}],"filesystems":[]} 2023-07-14T19:28:52.478 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:52.478+0000 7fb91637d700 1 -- 172.21.15.98:0/3169276804 >> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] conn(0x7fb8f80a11a0 msgr2=0x7fb8f80a3650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:28:52.478 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:52.478+0000 7fb91637d700 1 --2- 172.21.15.98:0/3169276804 >> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] conn(0x7fb8f80a11a0 0x7fb8f80a3650 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7fb90000b2e0 tx=0x7fb900005e80 comp rx=0 tx=0).stop 2023-07-14T19:28:52.479 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:52.478+0000 7fb91637d700 1 -- 172.21.15.98:0/3169276804 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb9100ffd20 msgr2=0x7fb910194d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:28:52.479 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:52.478+0000 7fb91637d700 1 --2- 172.21.15.98:0/3169276804 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb9100ffd20 0x7fb910194d40 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7fb90400eab0 tx=0x7fb90400edc0 comp rx=0 tx=0).stop 2023-07-14T19:28:52.479 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:52.478+0000 7fb91637d700 1 -- 172.21.15.98:0/3169276804 shutdown_connections 2023-07-14T19:28:52.479 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:52.478+0000 7fb91637d700 1 --2- 172.21.15.98:0/3169276804 >> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] conn(0x7fb8f80a11a0 0x7fb8f80a3650 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:28:52.480 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:52.478+0000 7fb91637d700 1 --2- 172.21.15.98:0/3169276804 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fb9100feb30 0x7fb910194800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:28:52.480 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:52.478+0000 7fb91637d700 1 --2- 172.21.15.98:0/3169276804 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7fb9101006b0 0x7fb91019bdb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:28:52.480 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:52.478+0000 7fb91637d700 1 --2- 172.21.15.98:0/3169276804 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb9100ffd20 0x7fb910194d40 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:28:52.480 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:52.478+0000 7fb91637d700 1 -- 172.21.15.98:0/3169276804 >> 172.21.15.98:0/3169276804 conn(0x7fb9100fa110 msgr2=0x7fb910103160 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:28:52.481 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:52.478+0000 7fb91637d700 1 -- 172.21.15.98:0/3169276804 shutdown_connections 2023-07-14T19:28:52.481 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:52.478+0000 7fb91637d700 1 -- 172.21.15.98:0/3169276804 wait complete. 2023-07-14T19:28:52.482 INFO:teuthology.orchestra.run.smithi098.stderr:dumped fsmap epoch 11 2023-07-14T19:28:52.495 INFO:journalctl@ceph.mds.a.smithi098.stdout:Jul 14 19:28:52 smithi098 systemd[1]: Starting Ceph mds.a for 9adea648-227b-11ee-9b33-001a4aab830c... 2023-07-14T19:28:52.781 INFO:journalctl@ceph.mds.a.smithi098.stdout:Jul 14 19:28:52 smithi098 podman[162132]: 2023-07-14T19:28:53.281 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:28:52 smithi098 ceph-mon[127486]: Deploying daemon mds.a on smithi098 2023-07-14T19:28:53.282 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:28:52 smithi098 ceph-mon[127486]: pgmap v244: 1 pgs: 1 active+clean; 577 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-14T19:28:53.282 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:28:52 smithi098 ceph-mon[127486]: from='client.? 172.21.15.98:0/3169276804' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-07-14T19:28:53.282 INFO:journalctl@ceph.mds.a.smithi098.stdout:Jul 14 19:28:52 smithi098 ceph-mds[162172]: -- 172.21.15.98:0/268515527 <== mon.0 v2:172.21.15.98:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55ca387011e0 con 0x55ca386d6800 2023-07-14T19:28:53.283 INFO:journalctl@ceph.mds.a.smithi098.stdout:Jul 14 19:28:52 smithi098 ceph-9adea648-227b-11ee-9b33-001a4aab830c-mds-a[162148]: starting mds.a at 2023-07-14T19:28:53.283 INFO:journalctl@ceph.mds.a.smithi098.stdout:Jul 14 19:28:53 smithi098 bash[162132]: 72b8a2739e10b0dcfcd78bad6223ad282a4a4c308ff307460f33d6ed74f46b28 2023-07-14T19:28:53.283 INFO:journalctl@ceph.mds.a.smithi098.stdout:Jul 14 19:28:53 smithi098 systemd[1]: Started Ceph mds.a for 9adea648-227b-11ee-9b33-001a4aab830c. 2023-07-14T19:28:53.441 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:28:52 smithi101 ceph-mon[128807]: Deploying daemon mds.a on smithi098 2023-07-14T19:28:53.442 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:28:52 smithi101 ceph-mon[128807]: pgmap v244: 1 pgs: 1 active+clean; 577 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-14T19:28:53.442 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:28:52 smithi101 ceph-mon[128807]: from='client.? 172.21.15.98:0/3169276804' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-07-14T19:28:53.470 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:28:52 smithi139 ceph-mon[129293]: Deploying daemon mds.a on smithi098 2023-07-14T19:28:53.470 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:28:52 smithi139 ceph-mon[129293]: pgmap v244: 1 pgs: 1 active+clean; 577 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-14T19:28:53.471 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:28:52 smithi139 ceph-mon[129293]: from='client.? 172.21.15.98:0/3169276804' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-07-14T19:28:54.442 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:28:54 smithi101 ceph-mon[128807]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' 2023-07-14T19:28:54.442 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:28:54 smithi101 ceph-mon[128807]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' 2023-07-14T19:28:54.442 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:28:54 smithi101 ceph-mon[128807]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' 2023-07-14T19:28:54.442 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:28:54 smithi101 ceph-mon[128807]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-07-14T19:28:54.443 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:28:54 smithi101 ceph-mon[128807]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-07-14T19:28:54.443 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:28:54 smithi101 ceph-mon[128807]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T19:28:54.443 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:28:54 smithi101 ceph-mon[128807]: Deploying daemon mds.b on smithi101 2023-07-14T19:28:54.443 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:28:54 smithi101 ceph-mon[128807]: pgmap v245: 1 pgs: 1 active+clean; 577 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-14T19:28:54.443 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:28:54 smithi101 ceph-mon[128807]: mds.? [v2:172.21.15.98:6840/1679318259,v1:172.21.15.98:6841/1679318259] up:boot 2023-07-14T19:28:54.443 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:28:54 smithi101 ceph-mon[128807]: fsmap 11 up:standby 2023-07-14T19:28:54.443 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:28:54 smithi101 ceph-mon[128807]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2023-07-14T19:28:54.470 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:28:54 smithi139 ceph-mon[129293]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' 2023-07-14T19:28:54.471 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:28:54 smithi139 ceph-mon[129293]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' 2023-07-14T19:28:54.471 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:28:54 smithi139 ceph-mon[129293]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' 2023-07-14T19:28:54.471 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:28:54 smithi139 ceph-mon[129293]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-07-14T19:28:54.471 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:28:54 smithi139 ceph-mon[129293]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-07-14T19:28:54.471 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:28:54 smithi139 ceph-mon[129293]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T19:28:54.471 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:28:54 smithi139 ceph-mon[129293]: Deploying daemon mds.b on smithi101 2023-07-14T19:28:54.471 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:28:54 smithi139 ceph-mon[129293]: pgmap v245: 1 pgs: 1 active+clean; 577 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-14T19:28:54.472 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:28:54 smithi139 ceph-mon[129293]: mds.? [v2:172.21.15.98:6840/1679318259,v1:172.21.15.98:6841/1679318259] up:boot 2023-07-14T19:28:54.472 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:28:54 smithi139 ceph-mon[129293]: fsmap 11 up:standby 2023-07-14T19:28:54.472 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:28:54 smithi139 ceph-mon[129293]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2023-07-14T19:28:54.497 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-07-14T19:28:54.529 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:28:54 smithi098 ceph-mon[127486]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' 2023-07-14T19:28:54.530 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:28:54 smithi098 ceph-mon[127486]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' 2023-07-14T19:28:54.530 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:28:54 smithi098 ceph-mon[127486]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' 2023-07-14T19:28:54.530 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:28:54 smithi098 ceph-mon[127486]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-07-14T19:28:54.530 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:28:54 smithi098 ceph-mon[127486]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-07-14T19:28:54.530 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:28:54 smithi098 ceph-mon[127486]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T19:28:54.530 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:28:54 smithi098 ceph-mon[127486]: Deploying daemon mds.b on smithi101 2023-07-14T19:28:54.531 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:28:54 smithi098 ceph-mon[127486]: pgmap v245: 1 pgs: 1 active+clean; 577 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-14T19:28:54.531 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:28:54 smithi098 ceph-mon[127486]: mds.? [v2:172.21.15.98:6840/1679318259,v1:172.21.15.98:6841/1679318259] up:boot 2023-07-14T19:28:54.531 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:28:54 smithi098 ceph-mon[127486]: fsmap 11 up:standby 2023-07-14T19:28:54.531 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:28:54 smithi098 ceph-mon[127486]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2023-07-14T19:28:54.638 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:54.637+0000 7f5ecfe7c700 1 -- 172.21.15.98:0/2996011469 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5ec81017b0 msgr2=0x7f5ec8108070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:28:54.639 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:54.637+0000 7f5ecfe7c700 1 --2- 172.21.15.98:0/2996011469 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5ec81017b0 0x7f5ec8108070 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f5ec000b0a0 tx=0x7f5ec000b3b0 comp rx=0 tx=0).stop 2023-07-14T19:28:54.639 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:54.638+0000 7f5ecfe7c700 1 -- 172.21.15.98:0/2996011469 shutdown_connections 2023-07-14T19:28:54.639 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:54.638+0000 7f5ecfe7c700 1 --2- 172.21.15.98:0/2996011469 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5ec81017b0 0x7f5ec8108070 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:28:54.639 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:54.638+0000 7f5ecfe7c700 1 --2- 172.21.15.98:0/2996011469 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f5ec80ffc30 0x7f5ec8100040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:28:54.639 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:54.638+0000 7f5ecfe7c700 1 --2- 172.21.15.98:0/2996011469 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f5ec8100e20 0x7f5ec8101270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:28:54.640 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:54.638+0000 7f5ecfe7c700 1 -- 172.21.15.98:0/2996011469 >> 172.21.15.98:0/2996011469 conn(0x7f5ec80fb210 msgr2=0x7f5ec80fd680 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:28:54.640 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:54.638+0000 7f5ecfe7c700 1 -- 172.21.15.98:0/2996011469 shutdown_connections 2023-07-14T19:28:54.640 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:54.638+0000 7f5ecfe7c700 1 -- 172.21.15.98:0/2996011469 wait complete. 2023-07-14T19:28:54.640 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:54.638+0000 7f5ecfe7c700 1 Processor -- start 2023-07-14T19:28:54.640 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:54.638+0000 7f5ecfe7c700 1 -- start start 2023-07-14T19:28:54.642 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:54.638+0000 7f5ecfe7c700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5ec80ffc30 0x7f5ec80743f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:28:54.642 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:54.638+0000 7f5ecfe7c700 1 --2- >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f5ec8100e20 0x7f5ec8072a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:28:54.642 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:54.638+0000 7f5ecfe7c700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f5ec81017b0 0x7f5ec8072f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:28:54.643 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:54.638+0000 7f5ecfe7c700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5ec8073710 con 0x7f5ec80ffc30 2023-07-14T19:28:54.643 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:54.638+0000 7f5ecfe7c700 1 -- --> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5ec8073880 con 0x7f5ec8100e20 2023-07-14T19:28:54.643 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:54.638+0000 7f5ecfe7c700 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5ec819fb90 con 0x7f5ec81017b0 2023-07-14T19:28:54.643 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:54.639+0000 7f5ecdc18700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5ec80ffc30 0x7f5ec80743f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:28:54.643 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:54.639+0000 7f5ecdc18700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5ec80ffc30 0x7f5ec80743f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:45544/0 (socket says 172.21.15.98:45544) 2023-07-14T19:28:54.643 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:54.639+0000 7f5ecdc18700 1 -- 172.21.15.98:0/3769446478 learned_addr learned my addr 172.21.15.98:0/3769446478 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-07-14T19:28:54.643 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:54.639+0000 7f5ece419700 1 --2- 172.21.15.98:0/3769446478 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f5ec81017b0 0x7f5ec8072f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:28:54.644 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:54.639+0000 7f5ecdc18700 1 -- 172.21.15.98:0/3769446478 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f5ec81017b0 msgr2=0x7f5ec8072f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:28:54.644 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:54.639+0000 7f5ecdc18700 1 --2- 172.21.15.98:0/3769446478 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f5ec81017b0 0x7f5ec8072f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:28:54.644 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:54.639+0000 7f5ecdc18700 1 -- 172.21.15.98:0/3769446478 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f5ec8100e20 msgr2=0x7f5ec8072a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:28:54.644 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:54.639+0000 7f5ecdc18700 1 --2- 172.21.15.98:0/3769446478 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f5ec8100e20 0x7f5ec8072a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:28:54.644 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:54.639+0000 7f5ecdc18700 1 -- 172.21.15.98:0/3769446478 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5ec0009d00 con 0x7f5ec80ffc30 2023-07-14T19:28:54.644 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:54.639+0000 7f5ecdc18700 1 --2- 172.21.15.98:0/3769446478 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5ec80ffc30 0x7f5ec80743f0 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f5ec400b500 tx=0x7f5ec400b8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:28:54.644 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:54.639+0000 7f5ebeffd700 1 -- 172.21.15.98:0/3769446478 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ec4011830 con 0x7f5ec80ffc30 2023-07-14T19:28:54.645 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:54.639+0000 7f5ebeffd700 1 -- 172.21.15.98:0/3769446478 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f5ec4011e50 con 0x7f5ec80ffc30 2023-07-14T19:28:54.645 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:54.639+0000 7f5ecfe7c700 1 -- 172.21.15.98:0/3769446478 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5ec819fe70 con 0x7f5ec80ffc30 2023-07-14T19:28:54.645 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:54.639+0000 7f5ebeffd700 1 -- 172.21.15.98:0/3769446478 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ec400f2e0 con 0x7f5ec80ffc30 2023-07-14T19:28:54.645 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:54.639+0000 7f5ecfe7c700 1 -- 172.21.15.98:0/3769446478 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5ec81a0370 con 0x7f5ec80ffc30 2023-07-14T19:28:54.645 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:54.640+0000 7f5ecfe7c700 1 -- 172.21.15.98:0/3769446478 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5ec81070e0 con 0x7f5ec80ffc30 2023-07-14T19:28:54.645 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:54.641+0000 7f5ebeffd700 1 -- 172.21.15.98:0/3769446478 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 16) v1 ==== 135715+0+0 (secure 0 0 0) 0x7f5ec4010580 con 0x7f5ec80ffc30 2023-07-14T19:28:54.645 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:54.642+0000 7f5ebeffd700 1 --2- 172.21.15.98:0/3769446478 >> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] conn(0x7f5eb40a1220 0x7f5eb40a36d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:28:54.645 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:54.642+0000 7f5ebeffd700 1 -- 172.21.15.98:0/3769446478 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(79..79 src has 1..79) v4 ==== 7127+0+0 (secure 0 0 0) 0x7f5ec40cc490 con 0x7f5ec80ffc30 2023-07-14T19:28:54.646 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:54.642+0000 7f5ecd417700 1 --2- 172.21.15.98:0/3769446478 >> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] conn(0x7f5eb40a1220 0x7f5eb40a36d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:28:54.646 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:54.642+0000 7f5ecd417700 1 --2- 172.21.15.98:0/3769446478 >> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] conn(0x7f5eb40a1220 0x7f5eb40a36d0 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f5eb80097b0 tx=0x7f5eb8006d20 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:28:54.646 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:54.645+0000 7f5ebeffd700 1 -- 172.21.15.98:0/3769446478 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f5ec4096080 con 0x7f5ec80ffc30 2023-07-14T19:28:54.728 INFO:journalctl@ceph.mds.b.smithi101.stdout:Jul 14 19:28:54 smithi101 systemd[1]: Starting Ceph mds.b for 9adea648-227b-11ee-9b33-001a4aab830c... 2023-07-14T19:28:54.881 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:54.878+0000 7f5ecfe7c700 1 -- 172.21.15.98:0/3769446478 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f5ec802c8e0 con 0x7f5ec80ffc30 2023-07-14T19:28:54.881 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:54.878+0000 7f5ebeffd700 1 -- 172.21.15.98:0/3769446478 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 12 v12) v1 ==== 94+0+8843 (secure 0 0 0) 0x7f5ec40996a0 con 0x7f5ec80ffc30 2023-07-14T19:28:54.882 INFO:teuthology.orchestra.run.smithi098.stdout: 2023-07-14T19:28:54.882 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":12,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14565,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6835/586280675","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6834","nonce":586280675},{"type":"v1","addr":"172.21.15.98:6835","nonce":586280675}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":4},{"gid":14577,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6837/487577257","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6836","nonce":487577257},{"type":"v1","addr":"172.21.15.98:6837","nonce":487577257}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":5},{"gid":14634,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6839/4085731209","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6838","nonce":4085731209},{"type":"v1","addr":"172.21.15.98:6839","nonce":4085731209}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":11},{"gid":14646,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6841/1679318259","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6840","nonce":1679318259},{"type":"v1","addr":"172.21.15.98:6841","nonce":1679318259}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":12},{"gid":24421,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.139:6833/3505477820","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6832","nonce":3505477820},{"type":"v1","addr":"172.21.15.139:6833","nonce":3505477820}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":24437,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.101:6833/3474460171","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.101:6832","nonce":3474460171},{"type":"v1","addr":"172.21.15.101:6833","nonce":3474460171}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":24455,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.101:6835/1583264648","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.101:6834","nonce":1583264648},{"type":"v1","addr":"172.21.15.101:6835","nonce":1583264648}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":6},{"gid":24460,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.139:6835/2315281724","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6834","nonce":2315281724},{"type":"v1","addr":"172.21.15.139:6835","nonce":2315281724}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":7},{"gid":24469,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.139:6837/3597660552","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6836","nonce":3597660552},{"type":"v1","addr":"172.21.15.139:6837","nonce":3597660552}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":8},{"gid":24479,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.101:6837/485741528","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.101:6836","nonce":485741528},{"type":"v1","addr":"172.21.15.101:6837","nonce":485741528}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":9},{"gid":24493,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.139:6839/1258854143","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6838","nonce":1258854143},{"type":"v1","addr":"172.21.15.139:6839","nonce":1258854143}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10}],"filesystems":[]} 2023-07-14T19:28:54.883 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:54.880+0000 7f5ecfe7c700 1 -- 172.21.15.98:0/3769446478 >> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] conn(0x7f5eb40a1220 msgr2=0x7f5eb40a36d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:28:54.883 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:54.880+0000 7f5ecfe7c700 1 --2- 172.21.15.98:0/3769446478 >> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] conn(0x7f5eb40a1220 0x7f5eb40a36d0 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f5eb80097b0 tx=0x7f5eb8006d20 comp rx=0 tx=0).stop 2023-07-14T19:28:54.883 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:54.880+0000 7f5ecfe7c700 1 -- 172.21.15.98:0/3769446478 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5ec80ffc30 msgr2=0x7f5ec80743f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:28:54.883 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:54.880+0000 7f5ecfe7c700 1 --2- 172.21.15.98:0/3769446478 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5ec80ffc30 0x7f5ec80743f0 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f5ec400b500 tx=0x7f5ec400b8c0 comp rx=0 tx=0).stop 2023-07-14T19:28:54.884 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:54.881+0000 7f5ecfe7c700 1 -- 172.21.15.98:0/3769446478 shutdown_connections 2023-07-14T19:28:54.884 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:54.881+0000 7f5ecfe7c700 1 --2- 172.21.15.98:0/3769446478 >> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] conn(0x7f5eb40a1220 0x7f5eb40a36d0 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:28:54.884 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:54.881+0000 7f5ecfe7c700 1 --2- 172.21.15.98:0/3769446478 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f5ec81017b0 0x7f5ec8072f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:28:54.884 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:54.881+0000 7f5ecfe7c700 1 --2- 172.21.15.98:0/3769446478 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5ec80ffc30 0x7f5ec80743f0 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:28:54.884 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:54.881+0000 7f5ecfe7c700 1 --2- 172.21.15.98:0/3769446478 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f5ec8100e20 0x7f5ec8072a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:28:54.884 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:54.881+0000 7f5ecfe7c700 1 -- 172.21.15.98:0/3769446478 >> 172.21.15.98:0/3769446478 conn(0x7f5ec80fb210 msgr2=0x7f5ec8104220 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:28:54.885 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:54.881+0000 7f5ecfe7c700 1 -- 172.21.15.98:0/3769446478 shutdown_connections 2023-07-14T19:28:54.885 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:54.881+0000 7f5ecfe7c700 1 -- 172.21.15.98:0/3769446478 wait complete. 2023-07-14T19:28:54.885 INFO:teuthology.orchestra.run.smithi098.stderr:dumped fsmap epoch 12 2023-07-14T19:28:55.290 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:28:55 smithi101 ceph-mon[128807]: from='client.? 172.21.15.98:0/3769446478' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-07-14T19:28:55.291 INFO:journalctl@ceph.mds.b.smithi101.stdout:Jul 14 19:28:55 smithi101 podman[154053]: 2023-07-14T19:28:55.470 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:28:55 smithi139 ceph-mon[129293]: from='client.? 172.21.15.98:0/3769446478' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-07-14T19:28:55.531 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:28:55 smithi098 ceph-mon[127486]: from='client.? 172.21.15.98:0/3769446478' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-07-14T19:28:55.691 INFO:journalctl@ceph.mds.b.smithi101.stdout:Jul 14 19:28:55 smithi101 ceph-mds[154093]: -- 172.21.15.101:0/1911141047 <== mon.2 v2:172.21.15.101:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x558bb6a831e0 con 0x558bb6a59000 2023-07-14T19:28:55.692 INFO:journalctl@ceph.mds.b.smithi101.stdout:Jul 14 19:28:55 smithi101 ceph-9adea648-227b-11ee-9b33-001a4aab830c-mds-b[154069]: starting mds.b at 2023-07-14T19:28:55.692 INFO:journalctl@ceph.mds.b.smithi101.stdout:Jul 14 19:28:55 smithi101 bash[154053]: 3ff95e44380691b5e19f4729a03495a2b0971fda2f7c33d604dc1c3d4ab3632a 2023-07-14T19:28:55.692 INFO:journalctl@ceph.mds.b.smithi101.stdout:Jul 14 19:28:55 smithi101 systemd[1]: Started Ceph mds.b for 9adea648-227b-11ee-9b33-001a4aab830c. 2023-07-14T19:28:56.691 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:28:56 smithi101 ceph-mon[128807]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' 2023-07-14T19:28:56.691 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:28:56 smithi101 ceph-mon[128807]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' 2023-07-14T19:28:56.692 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:28:56 smithi101 ceph-mon[128807]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' 2023-07-14T19:28:56.692 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:28:56 smithi101 ceph-mon[128807]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' 2023-07-14T19:28:56.692 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:28:56 smithi101 ceph-mon[128807]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' 2023-07-14T19:28:56.692 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:28:56 smithi101 ceph-mon[128807]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-14T19:28:56.692 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:28:56 smithi101 ceph-mon[128807]: pgmap v246: 1 pgs: 1 active+clean; 577 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-14T19:28:56.693 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:28:56 smithi101 ceph-mon[128807]: mds.? [v2:172.21.15.101:6838/48393381,v1:172.21.15.101:6839/48393381] up:boot 2023-07-14T19:28:56.693 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:28:56 smithi101 ceph-mon[128807]: fsmap 12 up:standby 2023-07-14T19:28:56.693 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:28:56 smithi101 ceph-mon[128807]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2023-07-14T19:28:56.721 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:28:56 smithi139 ceph-mon[129293]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' 2023-07-14T19:28:56.721 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:28:56 smithi139 ceph-mon[129293]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' 2023-07-14T19:28:56.721 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:28:56 smithi139 ceph-mon[129293]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' 2023-07-14T19:28:56.721 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:28:56 smithi139 ceph-mon[129293]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' 2023-07-14T19:28:56.722 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:28:56 smithi139 ceph-mon[129293]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' 2023-07-14T19:28:56.722 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:28:56 smithi139 ceph-mon[129293]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-14T19:28:56.722 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:28:56 smithi139 ceph-mon[129293]: pgmap v246: 1 pgs: 1 active+clean; 577 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-14T19:28:56.722 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:28:56 smithi139 ceph-mon[129293]: mds.? [v2:172.21.15.101:6838/48393381,v1:172.21.15.101:6839/48393381] up:boot 2023-07-14T19:28:56.723 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:28:56 smithi139 ceph-mon[129293]: fsmap 12 up:standby 2023-07-14T19:28:56.723 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:28:56 smithi139 ceph-mon[129293]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2023-07-14T19:28:56.781 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:28:56 smithi098 ceph-mon[127486]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' 2023-07-14T19:28:56.781 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:28:56 smithi098 ceph-mon[127486]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' 2023-07-14T19:28:56.782 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:28:56 smithi098 ceph-mon[127486]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' 2023-07-14T19:28:56.782 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:28:56 smithi098 ceph-mon[127486]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' 2023-07-14T19:28:56.782 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:28:56 smithi098 ceph-mon[127486]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' 2023-07-14T19:28:56.782 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:28:56 smithi098 ceph-mon[127486]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-14T19:28:56.782 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:28:56 smithi098 ceph-mon[127486]: pgmap v246: 1 pgs: 1 active+clean; 577 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-14T19:28:56.782 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:28:56 smithi098 ceph-mon[127486]: mds.? [v2:172.21.15.101:6838/48393381,v1:172.21.15.101:6839/48393381] up:boot 2023-07-14T19:28:56.783 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:28:56 smithi098 ceph-mon[127486]: fsmap 12 up:standby 2023-07-14T19:28:56.783 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:28:56 smithi098 ceph-mon[127486]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2023-07-14T19:28:56.894 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-07-14T19:28:57.029 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:57.027+0000 7f4b90b48700 1 -- 172.21.15.98:0/1920210417 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4b8c1006a0 msgr2=0x7f4b8c106f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:28:57.030 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:57.027+0000 7f4b90b48700 1 --2- 172.21.15.98:0/1920210417 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4b8c1006a0 0x7f4b8c106f60 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f4b8400b0a0 tx=0x7f4b8400b3b0 comp rx=0 tx=0).stop 2023-07-14T19:28:57.030 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:57.027+0000 7f4b90b48700 1 -- 172.21.15.98:0/1920210417 shutdown_connections 2023-07-14T19:28:57.030 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:57.027+0000 7f4b90b48700 1 --2- 172.21.15.98:0/1920210417 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4b8c1006a0 0x7f4b8c106f60 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:28:57.030 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:57.027+0000 7f4b90b48700 1 --2- 172.21.15.98:0/1920210417 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f4b8c0feb20 0x7f4b8c0fef30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:28:57.030 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:57.027+0000 7f4b90b48700 1 --2- 172.21.15.98:0/1920210417 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f4b8c0ffd10 0x7f4b8c100160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:28:57.031 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:57.027+0000 7f4b90b48700 1 -- 172.21.15.98:0/1920210417 >> 172.21.15.98:0/1920210417 conn(0x7f4b8c0fa120 msgr2=0x7f4b8c0fc570 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:28:57.031 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:57.027+0000 7f4b90b48700 1 -- 172.21.15.98:0/1920210417 shutdown_connections 2023-07-14T19:28:57.031 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:57.027+0000 7f4b90b48700 1 -- 172.21.15.98:0/1920210417 wait complete. 2023-07-14T19:28:57.031 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:57.028+0000 7f4b90b48700 1 Processor -- start 2023-07-14T19:28:57.031 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:57.028+0000 7f4b90b48700 1 -- start start 2023-07-14T19:28:57.032 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:57.028+0000 7f4b90b48700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4b8c0feb20 0x7f4b8c1948e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:28:57.032 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:57.028+0000 7f4b90b48700 1 --2- >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f4b8c0ffd10 0x7f4b8c194e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:28:57.032 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:57.028+0000 7f4b90b48700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f4b8c1006a0 0x7f4b8c19be90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:28:57.032 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:57.028+0000 7f4b90b48700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4b8c19c460 con 0x7f4b8c0feb20 2023-07-14T19:28:57.033 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:57.028+0000 7f4b90b48700 1 -- --> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4b8c19c5a0 con 0x7f4b8c0ffd10 2023-07-14T19:28:57.033 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:57.028+0000 7f4b90b48700 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4b8c19c730 con 0x7f4b8c1006a0 2023-07-14T19:28:57.033 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:57.028+0000 7f4b8a59c700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4b8c0feb20 0x7f4b8c1948e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:28:57.033 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:57.028+0000 7f4b8a59c700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4b8c0feb20 0x7f4b8c1948e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:45558/0 (socket says 172.21.15.98:45558) 2023-07-14T19:28:57.033 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:57.028+0000 7f4b89d9b700 1 --2- >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f4b8c0ffd10 0x7f4b8c194e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:28:57.034 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:57.028+0000 7f4b8a59c700 1 -- 172.21.15.98:0/3238159607 learned_addr learned my addr 172.21.15.98:0/3238159607 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-07-14T19:28:57.034 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:57.028+0000 7f4b8ad9d700 1 --2- 172.21.15.98:0/3238159607 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f4b8c1006a0 0x7f4b8c19be90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:28:57.034 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:57.028+0000 7f4b8a59c700 1 -- 172.21.15.98:0/3238159607 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f4b8c1006a0 msgr2=0x7f4b8c19be90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:28:57.034 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:57.028+0000 7f4b8a59c700 1 --2- 172.21.15.98:0/3238159607 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f4b8c1006a0 0x7f4b8c19be90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:28:57.035 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:57.028+0000 7f4b8a59c700 1 -- 172.21.15.98:0/3238159607 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f4b8c0ffd10 msgr2=0x7f4b8c194e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:28:57.035 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:57.028+0000 7f4b8a59c700 1 --2- 172.21.15.98:0/3238159607 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f4b8c0ffd10 0x7f4b8c194e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:28:57.035 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:57.028+0000 7f4b8a59c700 1 -- 172.21.15.98:0/3238159607 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4b84009d00 con 0x7f4b8c0feb20 2023-07-14T19:28:57.035 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:57.028+0000 7f4b8a59c700 1 --2- 172.21.15.98:0/3238159607 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4b8c0feb20 0x7f4b8c1948e0 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f4b7c00e970 tx=0x7f4b7c00ec80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:28:57.035 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:57.029+0000 7f4b7b7fe700 1 -- 172.21.15.98:0/3238159607 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4b7c009790 con 0x7f4b8c0feb20 2023-07-14T19:28:57.036 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:57.029+0000 7f4b7b7fe700 1 -- 172.21.15.98:0/3238159607 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f4b7c00c7e0 con 0x7f4b8c0feb20 2023-07-14T19:28:57.036 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:57.029+0000 7f4b90b48700 1 -- 172.21.15.98:0/3238159607 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4b8c19ca10 con 0x7f4b8c0feb20 2023-07-14T19:28:57.036 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:57.029+0000 7f4b7b7fe700 1 -- 172.21.15.98:0/3238159607 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4b7c003c70 con 0x7f4b8c0feb20 2023-07-14T19:28:57.036 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:57.029+0000 7f4b90b48700 1 -- 172.21.15.98:0/3238159607 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4b8c19cf20 con 0x7f4b8c0feb20 2023-07-14T19:28:57.038 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:57.030+0000 7f4b90b48700 1 -- 172.21.15.98:0/3238159607 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4b8c10a300 con 0x7f4b8c0feb20 2023-07-14T19:28:57.039 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:57.030+0000 7f4b7b7fe700 1 -- 172.21.15.98:0/3238159607 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 16) v1 ==== 135715+0+0 (secure 0 0 0) 0x7f4b7c00cc50 con 0x7f4b8c0feb20 2023-07-14T19:28:57.039 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:57.031+0000 7f4b7b7fe700 1 --2- 172.21.15.98:0/3238159607 >> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] conn(0x7f4b740a11a0 0x7f4b740a3650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:28:57.039 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:57.031+0000 7f4b7b7fe700 1 -- 172.21.15.98:0/3238159607 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(79..79 src has 1..79) v4 ==== 7127+0+0 (secure 0 0 0) 0x7f4b7c014070 con 0x7f4b8c0feb20 2023-07-14T19:28:57.039 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:57.031+0000 7f4b89d9b700 1 --2- 172.21.15.98:0/3238159607 >> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] conn(0x7f4b740a11a0 0x7f4b740a3650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:28:57.039 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:57.031+0000 7f4b89d9b700 1 --2- 172.21.15.98:0/3238159607 >> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] conn(0x7f4b740a11a0 0x7f4b740a3650 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f4b80005950 tx=0x7f4b8000b410 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:28:57.039 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:57.035+0000 7f4b7b7fe700 1 -- 172.21.15.98:0/3238159607 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f4b7c096210 con 0x7f4b8c0feb20 2023-07-14T19:28:57.281 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:57.280+0000 7f4b90b48700 1 -- 172.21.15.98:0/3238159607 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f4b8c10a520 con 0x7f4b8c0feb20 2023-07-14T19:28:57.281 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:57.281+0000 7f4b7b7fe700 1 -- 172.21.15.98:0/3238159607 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 13 v13) v1 ==== 94+0+9598 (secure 0 0 0) 0x7f4b7c003dd0 con 0x7f4b8c0feb20 2023-07-14T19:28:57.283 INFO:teuthology.orchestra.run.smithi098.stdout: 2023-07-14T19:28:57.283 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":13,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14565,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6835/586280675","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6834","nonce":586280675},{"type":"v1","addr":"172.21.15.98:6835","nonce":586280675}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":4},{"gid":14577,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6837/487577257","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6836","nonce":487577257},{"type":"v1","addr":"172.21.15.98:6837","nonce":487577257}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":5},{"gid":14634,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6839/4085731209","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6838","nonce":4085731209},{"type":"v1","addr":"172.21.15.98:6839","nonce":4085731209}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":11},{"gid":14646,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6841/1679318259","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6840","nonce":1679318259},{"type":"v1","addr":"172.21.15.98:6841","nonce":1679318259}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":12},{"gid":24421,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.139:6833/3505477820","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6832","nonce":3505477820},{"type":"v1","addr":"172.21.15.139:6833","nonce":3505477820}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":24437,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.101:6833/3474460171","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.101:6832","nonce":3474460171},{"type":"v1","addr":"172.21.15.101:6833","nonce":3474460171}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":24455,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.101:6835/1583264648","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.101:6834","nonce":1583264648},{"type":"v1","addr":"172.21.15.101:6835","nonce":1583264648}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":6},{"gid":24460,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.139:6835/2315281724","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6834","nonce":2315281724},{"type":"v1","addr":"172.21.15.139:6835","nonce":2315281724}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":7},{"gid":24469,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.139:6837/3597660552","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6836","nonce":3597660552},{"type":"v1","addr":"172.21.15.139:6837","nonce":3597660552}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":8},{"gid":24479,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.101:6837/485741528","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.101:6836","nonce":485741528},{"type":"v1","addr":"172.21.15.101:6837","nonce":485741528}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":9},{"gid":24493,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.139:6839/1258854143","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6838","nonce":1258854143},{"type":"v1","addr":"172.21.15.139:6839","nonce":1258854143}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10},{"gid":24521,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.101:6839/48393381","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.101:6838","nonce":48393381},{"type":"v1","addr":"172.21.15.101:6839","nonce":48393381}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":13}],"filesystems":[]} 2023-07-14T19:28:57.284 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:57.283+0000 7f4b90b48700 1 -- 172.21.15.98:0/3238159607 >> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] conn(0x7f4b740a11a0 msgr2=0x7f4b740a3650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:28:57.284 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:57.283+0000 7f4b90b48700 1 --2- 172.21.15.98:0/3238159607 >> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] conn(0x7f4b740a11a0 0x7f4b740a3650 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f4b80005950 tx=0x7f4b8000b410 comp rx=0 tx=0).stop 2023-07-14T19:28:57.285 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:57.283+0000 7f4b90b48700 1 -- 172.21.15.98:0/3238159607 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4b8c0feb20 msgr2=0x7f4b8c1948e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:28:57.285 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:57.283+0000 7f4b90b48700 1 --2- 172.21.15.98:0/3238159607 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4b8c0feb20 0x7f4b8c1948e0 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f4b7c00e970 tx=0x7f4b7c00ec80 comp rx=0 tx=0).stop 2023-07-14T19:28:57.285 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:57.283+0000 7f4b90b48700 1 -- 172.21.15.98:0/3238159607 shutdown_connections 2023-07-14T19:28:57.285 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:57.283+0000 7f4b90b48700 1 --2- 172.21.15.98:0/3238159607 >> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] conn(0x7f4b740a11a0 0x7f4b740a3650 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:28:57.285 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:57.283+0000 7f4b90b48700 1 --2- 172.21.15.98:0/3238159607 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f4b8c1006a0 0x7f4b8c19be90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:28:57.285 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:57.283+0000 7f4b90b48700 1 --2- 172.21.15.98:0/3238159607 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4b8c0feb20 0x7f4b8c1948e0 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:28:57.285 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:57.283+0000 7f4b90b48700 1 --2- 172.21.15.98:0/3238159607 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f4b8c0ffd10 0x7f4b8c194e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:28:57.286 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:57.283+0000 7f4b90b48700 1 -- 172.21.15.98:0/3238159607 >> 172.21.15.98:0/3238159607 conn(0x7f4b8c0fa120 msgr2=0x7f4b8c0fc250 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:28:57.286 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:57.283+0000 7f4b90b48700 1 -- 172.21.15.98:0/3238159607 shutdown_connections 2023-07-14T19:28:57.286 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:57.283+0000 7f4b90b48700 1 -- 172.21.15.98:0/3238159607 wait complete. 2023-07-14T19:28:57.286 INFO:teuthology.orchestra.run.smithi098.stderr:dumped fsmap epoch 13 2023-07-14T19:28:57.297 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-07-14T19:28:57.426 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:57.424+0000 7fe7af0f2700 1 -- 172.21.15.98:0/2310522427 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe7a80ffcf0 msgr2=0x7fe7a8100140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:28:57.427 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:57.424+0000 7fe7af0f2700 1 --2- 172.21.15.98:0/2310522427 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe7a80ffcf0 0x7fe7a8100140 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7fe79800b0a0 tx=0x7fe79800b3b0 comp rx=0 tx=0).stop 2023-07-14T19:28:57.427 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:57.424+0000 7fe7af0f2700 1 -- 172.21.15.98:0/2310522427 shutdown_connections 2023-07-14T19:28:57.427 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:57.424+0000 7fe7af0f2700 1 --2- 172.21.15.98:0/2310522427 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fe7a8100680 0x7fe7a8106f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:28:57.427 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:57.424+0000 7fe7af0f2700 1 --2- 172.21.15.98:0/2310522427 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7fe7a80feb00 0x7fe7a80fef10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:28:57.428 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:57.424+0000 7fe7af0f2700 1 --2- 172.21.15.98:0/2310522427 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe7a80ffcf0 0x7fe7a8100140 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:28:57.428 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:57.424+0000 7fe7af0f2700 1 -- 172.21.15.98:0/2310522427 >> 172.21.15.98:0/2310522427 conn(0x7fe7a80fa100 msgr2=0x7fe7a80fc550 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:28:57.428 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:57.424+0000 7fe7af0f2700 1 -- 172.21.15.98:0/2310522427 shutdown_connections 2023-07-14T19:28:57.428 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:57.424+0000 7fe7af0f2700 1 -- 172.21.15.98:0/2310522427 wait complete. 2023-07-14T19:28:57.428 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:57.424+0000 7fe7af0f2700 1 Processor -- start 2023-07-14T19:28:57.428 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:57.424+0000 7fe7af0f2700 1 -- start start 2023-07-14T19:28:57.428 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:57.425+0000 7fe7af0f2700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe7a80feb00 0x7fe7a8068310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:28:57.429 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:57.425+0000 7fe7af0f2700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fe7a80ffcf0 0x7fe7a819a780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:28:57.429 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:57.425+0000 7fe7af0f2700 1 --2- >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7fe7a8100680 0x7fe7a819cdc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:28:57.429 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:57.425+0000 7fe7af0f2700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe7a819d560 con 0x7fe7a80feb00 2023-07-14T19:28:57.429 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:57.425+0000 7fe7af0f2700 1 -- --> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe7a819d6d0 con 0x7fe7a8100680 2023-07-14T19:28:57.429 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:57.425+0000 7fe7af0f2700 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe7a819d890 con 0x7fe7a80ffcf0 2023-07-14T19:28:57.429 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:57.425+0000 7fe7ace8e700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe7a80feb00 0x7fe7a8068310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:28:57.429 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:57.425+0000 7fe7ace8e700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe7a80feb00 0x7fe7a8068310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:45572/0 (socket says 172.21.15.98:45572) 2023-07-14T19:28:57.429 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:57.425+0000 7fe7ace8e700 1 -- 172.21.15.98:0/1198522029 learned_addr learned my addr 172.21.15.98:0/1198522029 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-07-14T19:28:57.430 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:57.425+0000 7fe7ad68f700 1 --2- 172.21.15.98:0/1198522029 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7fe7a8100680 0x7fe7a819cdc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:28:57.430 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:57.425+0000 7fe7a7fff700 1 --2- 172.21.15.98:0/1198522029 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fe7a80ffcf0 0x7fe7a819a780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:28:57.430 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:57.425+0000 7fe7ace8e700 1 -- 172.21.15.98:0/1198522029 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fe7a80ffcf0 msgr2=0x7fe7a819a780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:28:57.430 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:57.425+0000 7fe7ace8e700 1 --2- 172.21.15.98:0/1198522029 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fe7a80ffcf0 0x7fe7a819a780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:28:57.430 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:57.425+0000 7fe7ace8e700 1 -- 172.21.15.98:0/1198522029 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7fe7a8100680 msgr2=0x7fe7a819cdc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:28:57.431 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:57.425+0000 7fe7ace8e700 1 --2- 172.21.15.98:0/1198522029 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7fe7a8100680 0x7fe7a819cdc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:28:57.431 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:57.425+0000 7fe7ace8e700 1 -- 172.21.15.98:0/1198522029 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe798009d00 con 0x7fe7a80feb00 2023-07-14T19:28:57.431 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:57.425+0000 7fe7ace8e700 1 --2- 172.21.15.98:0/1198522029 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe7a80feb00 0x7fe7a8068310 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7fe7a000e970 tx=0x7fe7a000ec80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:28:57.431 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:57.425+0000 7fe7a5ffb700 1 -- 172.21.15.98:0/1198522029 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe7a0009790 con 0x7fe7a80feb00 2023-07-14T19:28:57.431 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:57.425+0000 7fe7a5ffb700 1 -- 172.21.15.98:0/1198522029 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fe7a000eec0 con 0x7fe7a80feb00 2023-07-14T19:28:57.431 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:57.425+0000 7fe7af0f2700 1 -- 172.21.15.98:0/1198522029 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe7a819db70 con 0x7fe7a80feb00 2023-07-14T19:28:57.431 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:57.425+0000 7fe7a5ffb700 1 -- 172.21.15.98:0/1198522029 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe7a0003cf0 con 0x7fe7a80feb00 2023-07-14T19:28:57.431 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:57.426+0000 7fe7af0f2700 1 -- 172.21.15.98:0/1198522029 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe7a819e080 con 0x7fe7a80feb00 2023-07-14T19:28:57.432 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:57.426+0000 7fe7af0f2700 1 -- 172.21.15.98:0/1198522029 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe7a8105990 con 0x7fe7a80feb00 2023-07-14T19:28:57.433 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:57.427+0000 7fe7a5ffb700 1 -- 172.21.15.98:0/1198522029 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 16) v1 ==== 135715+0+0 (secure 0 0 0) 0x7fe7a000ccd0 con 0x7fe7a80feb00 2023-07-14T19:28:57.433 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:57.428+0000 7fe7a5ffb700 1 --2- 172.21.15.98:0/1198522029 >> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] conn(0x7fe7940a11a0 0x7fe7940a3650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:28:57.433 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:57.428+0000 7fe7a7fff700 1 --2- 172.21.15.98:0/1198522029 >> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] conn(0x7fe7940a11a0 0x7fe7940a3650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:28:57.433 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:57.428+0000 7fe7a5ffb700 1 -- 172.21.15.98:0/1198522029 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(79..79 src has 1..79) v4 ==== 7127+0+0 (secure 0 0 0) 0x7fe7a0014070 con 0x7fe7a80feb00 2023-07-14T19:28:57.433 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:57.429+0000 7fe7a7fff700 1 --2- 172.21.15.98:0/1198522029 >> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] conn(0x7fe7940a11a0 0x7fe7940a3650 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7fe798006130 tx=0x7fe7980060a0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:28:57.433 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:57.432+0000 7fe7a5ffb700 1 -- 172.21.15.98:0/1198522029 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7fe7a00962b0 con 0x7fe7a80feb00 2023-07-14T19:28:57.631 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:57.630+0000 7fe7af0f2700 1 -- 172.21.15.98:0/1198522029 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fe7a802cb20 con 0x7fe7a80feb00 2023-07-14T19:28:57.632 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:57.631+0000 7fe7a5ffb700 1 -- 172.21.15.98:0/1198522029 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v79) v1 ==== 74+0+18420 (secure 0 0 0) 0x7fe7a00998d0 con 0x7fe7a80feb00 2023-07-14T19:28:57.632 INFO:teuthology.orchestra.run.smithi098.stdout: 2023-07-14T19:28:57.633 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":79,"fsid":"9adea648-227b-11ee-9b33-001a4aab830c","created":"2023-07-14T19:22:07.178854+0000","modified":"2023-07-14T19:28:20.933203+0000","last_up_change":"2023-07-14T19:28:19.646827+0000","last_in_change":"2023-07-14T19:28:02.018325+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-07-14T19:25:09.845827+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"a3797e89-1754-41cf-9517-227e06ffdd72","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6802","nonce":120507228},{"type":"v1","addr":"172.21.15.98:6803","nonce":120507228}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6804","nonce":120507228},{"type":"v1","addr":"172.21.15.98:6805","nonce":120507228}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6808","nonce":120507228},{"type":"v1","addr":"172.21.15.98:6809","nonce":120507228}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6806","nonce":120507228},{"type":"v1","addr":"172.21.15.98:6807","nonce":120507228}]},"public_addr":"172.21.15.98:6803/120507228","cluster_addr":"172.21.15.98:6805/120507228","heartbeat_back_addr":"172.21.15.98:6809/120507228","heartbeat_front_addr":"172.21.15.98:6807/120507228","state":["exists","up"]},{"osd":1,"uuid":"2b001add-1aaa-4496-a403-7ef9725b80a0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.101:6800","nonce":2450584584},{"type":"v1","addr":"172.21.15.101:6801","nonce":2450584584}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.101:6802","nonce":2450584584},{"type":"v1","addr":"172.21.15.101:6803","nonce":2450584584}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.101:6806","nonce":2450584584},{"type":"v1","addr":"172.21.15.101:6807","nonce":2450584584}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.101:6804","nonce":2450584584},{"type":"v1","addr":"172.21.15.101:6805","nonce":2450584584}]},"public_addr":"172.21.15.101:6801/2450584584","cluster_addr":"172.21.15.101:6803/2450584584","heartbeat_back_addr":"172.21.15.101:6807/2450584584","heartbeat_front_addr":"172.21.15.101:6805/2450584584","state":["exists","up"]},{"osd":2,"uuid":"48e6d499-3ca9-424b-b0fb-3ba5f33b9fc9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":945038995},{"type":"v1","addr":"172.21.15.139:6801","nonce":945038995}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":945038995},{"type":"v1","addr":"172.21.15.139:6803","nonce":945038995}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":945038995},{"type":"v1","addr":"172.21.15.139:6807","nonce":945038995}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":945038995},{"type":"v1","addr":"172.21.15.139:6805","nonce":945038995}]},"public_addr":"172.21.15.139:6801/945038995","cluster_addr":"172.21.15.139:6803/945038995","heartbeat_back_addr":"172.21.15.139:6807/945038995","heartbeat_front_addr":"172.21.15.139:6805/945038995","state":["exists","up"]},{"osd":3,"uuid":"5530b265-31a5-447b-a1ea-17ebae9cfddf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6810","nonce":3095249665},{"type":"v1","addr":"172.21.15.98:6811","nonce":3095249665}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6812","nonce":3095249665},{"type":"v1","addr":"172.21.15.98:6813","nonce":3095249665}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6816","nonce":3095249665},{"type":"v1","addr":"172.21.15.98:6817","nonce":3095249665}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6814","nonce":3095249665},{"type":"v1","addr":"172.21.15.98:6815","nonce":3095249665}]},"public_addr":"172.21.15.98:6811/3095249665","cluster_addr":"172.21.15.98:6813/3095249665","heartbeat_back_addr":"172.21.15.98:6817/3095249665","heartbeat_front_addr":"172.21.15.98:6815/3095249665","state":["exists","up"]},{"osd":4,"uuid":"615d4642-a997-4f95-9e38-9fbdaa831400","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":32,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.101:6808","nonce":2844761406},{"type":"v1","addr":"172.21.15.101:6809","nonce":2844761406}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.101:6810","nonce":2844761406},{"type":"v1","addr":"172.21.15.101:6811","nonce":2844761406}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.101:6814","nonce":2844761406},{"type":"v1","addr":"172.21.15.101:6815","nonce":2844761406}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.101:6812","nonce":2844761406},{"type":"v1","addr":"172.21.15.101:6813","nonce":2844761406}]},"public_addr":"172.21.15.101:6809/2844761406","cluster_addr":"172.21.15.101:6811/2844761406","heartbeat_back_addr":"172.21.15.101:6815/2844761406","heartbeat_front_addr":"172.21.15.101:6813/2844761406","state":["exists","up"]},{"osd":5,"uuid":"415de5fa-607a-45a8-b732-28daf3be6dd3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":37,"up_thru":52,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":3532263042},{"type":"v1","addr":"172.21.15.139:6809","nonce":3532263042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6810","nonce":3532263042},{"type":"v1","addr":"172.21.15.139:6811","nonce":3532263042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6814","nonce":3532263042},{"type":"v1","addr":"172.21.15.139:6815","nonce":3532263042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":3532263042},{"type":"v1","addr":"172.21.15.139:6813","nonce":3532263042}]},"public_addr":"172.21.15.139:6809/3532263042","cluster_addr":"172.21.15.139:6811/3532263042","heartbeat_back_addr":"172.21.15.139:6815/3532263042","heartbeat_front_addr":"172.21.15.139:6813/3532263042","state":["exists","up"]},{"osd":6,"uuid":"d7f9d4ba-c309-4b75-a836-5a36384b528e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":45,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6818","nonce":658948743},{"type":"v1","addr":"172.21.15.98:6819","nonce":658948743}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6820","nonce":658948743},{"type":"v1","addr":"172.21.15.98:6821","nonce":658948743}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6824","nonce":658948743},{"type":"v1","addr":"172.21.15.98:6825","nonce":658948743}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6822","nonce":658948743},{"type":"v1","addr":"172.21.15.98:6823","nonce":658948743}]},"public_addr":"172.21.15.98:6819/658948743","cluster_addr":"172.21.15.98:6821/658948743","heartbeat_back_addr":"172.21.15.98:6825/658948743","heartbeat_front_addr":"172.21.15.98:6823/658948743","state":["exists","up"]},{"osd":7,"uuid":"eedb94d9-c5cd-4a93-8a8c-2648804b7195","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":50,"up_thru":73,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.101:6816","nonce":2881214066},{"type":"v1","addr":"172.21.15.101:6817","nonce":2881214066}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.101:6818","nonce":2881214066},{"type":"v1","addr":"172.21.15.101:6819","nonce":2881214066}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.101:6822","nonce":2881214066},{"type":"v1","addr":"172.21.15.101:6823","nonce":2881214066}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.101:6820","nonce":2881214066},{"type":"v1","addr":"172.21.15.101:6821","nonce":2881214066}]},"public_addr":"172.21.15.101:6817/2881214066","cluster_addr":"172.21.15.101:6819/2881214066","heartbeat_back_addr":"172.21.15.101:6823/2881214066","heartbeat_front_addr":"172.21.15.101:6821/2881214066","state":["exists","up"]},{"osd":8,"uuid":"6bfd77c2-5770-466e-aa6f-84b50c74ffb6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":57,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":3669514524},{"type":"v1","addr":"172.21.15.139:6817","nonce":3669514524}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":3669514524},{"type":"v1","addr":"172.21.15.139:6819","nonce":3669514524}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":3669514524},{"type":"v1","addr":"172.21.15.139:6823","nonce":3669514524}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6820","nonce":3669514524},{"type":"v1","addr":"172.21.15.139:6821","nonce":3669514524}]},"public_addr":"172.21.15.139:6817/3669514524","cluster_addr":"172.21.15.139:6819/3669514524","heartbeat_back_addr":"172.21.15.139:6823/3669514524","heartbeat_front_addr":"172.21.15.139:6821/3669514524","state":["exists","up"]},{"osd":9,"uuid":"19c5b54a-2242-4b23-b721-63be9559653d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":65,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6826","nonce":2893994080},{"type":"v1","addr":"172.21.15.98:6827","nonce":2893994080}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6828","nonce":2893994080},{"type":"v1","addr":"172.21.15.98:6829","nonce":2893994080}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6832","nonce":2893994080},{"type":"v1","addr":"172.21.15.98:6833","nonce":2893994080}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6830","nonce":2893994080},{"type":"v1","addr":"172.21.15.98:6831","nonce":2893994080}]},"public_addr":"172.21.15.98:6827/2893994080","cluster_addr":"172.21.15.98:6829/2893994080","heartbeat_back_addr":"172.21.15.98:6833/2893994080","heartbeat_front_addr":"172.21.15.98:6831/2893994080","state":["exists","up"]},{"osd":10,"uuid":"326f0209-ba9d-4b59-97cc-d61a758a8f29","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":70,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.101:6824","nonce":3545244225},{"type":"v1","addr":"172.21.15.101:6825","nonce":3545244225}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.101:6826","nonce":3545244225},{"type":"v1","addr":"172.21.15.101:6827","nonce":3545244225}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.101:6830","nonce":3545244225},{"type":"v1","addr":"172.21.15.101:6831","nonce":3545244225}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.101:6828","nonce":3545244225},{"type":"v1","addr":"172.21.15.101:6829","nonce":3545244225}]},"public_addr":"172.21.15.101:6825/3545244225","cluster_addr":"172.21.15.101:6827/3545244225","heartbeat_back_addr":"172.21.15.101:6831/3545244225","heartbeat_front_addr":"172.21.15.101:6829/3545244225","state":["exists","up"]},{"osd":11,"uuid":"c61c0320-1057-4f17-9bf1-7ad306ed7ed3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":78,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":2630023454},{"type":"v1","addr":"172.21.15.139:6825","nonce":2630023454}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6826","nonce":2630023454},{"type":"v1","addr":"172.21.15.139:6827","nonce":2630023454}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":2630023454},{"type":"v1","addr":"172.21.15.139:6831","nonce":2630023454}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":2630023454},{"type":"v1","addr":"172.21.15.139:6829","nonce":2630023454}]},"public_addr":"172.21.15.139:6825/2630023454","cluster_addr":"172.21.15.139:6827/2630023454","heartbeat_back_addr":"172.21.15.139:6831/2630023454","heartbeat_front_addr":"172.21.15.139:6829/2630023454","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T19:24:23.620028+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T19:24:45.779846+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T19:25:06.877949+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T19:25:27.994199+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T19:25:49.461139+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T19:26:10.409282+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T19:26:32.116390+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T19:26:53.536474+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T19:27:14.384839+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T19:27:35.635921+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T19:27:56.858744+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T19:28:18.342519+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.98:0/2164590253":"2023-07-15T19:23:01.520302+0000","172.21.15.98:0/2094734896":"2023-07-15T19:23:01.520302+0000","172.21.15.98:0/784745009":"2023-07-15T19:22:39.481502+0000","172.21.15.98:0/3657190818":"2023-07-15T19:22:39.481502+0000","172.21.15.98:6800/1732769087":"2023-07-15T19:23:01.520302+0000","172.21.15.98:0/3393542419":"2023-07-15T19:22:39.481502+0000","172.21.15.98:6800/749234665":"2023-07-15T19:22:39.481502+0000","172.21.15.98:6801/1732769087":"2023-07-15T19:23:01.520302+0000","172.21.15.98:6801/749234665":"2023-07-15T19:22:39.481502+0000","172.21.15.98:0/187822873":"2023-07-15T19:23:01.520302+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-07-14T19:28:57.636 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:57.634+0000 7fe7af0f2700 1 -- 172.21.15.98:0/1198522029 >> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] conn(0x7fe7940a11a0 msgr2=0x7fe7940a3650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:28:57.636 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:57.634+0000 7fe7af0f2700 1 --2- 172.21.15.98:0/1198522029 >> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] conn(0x7fe7940a11a0 0x7fe7940a3650 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7fe798006130 tx=0x7fe7980060a0 comp rx=0 tx=0).stop 2023-07-14T19:28:57.636 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:57.634+0000 7fe7af0f2700 1 -- 172.21.15.98:0/1198522029 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe7a80feb00 msgr2=0x7fe7a8068310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:28:57.636 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:57.634+0000 7fe7af0f2700 1 --2- 172.21.15.98:0/1198522029 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe7a80feb00 0x7fe7a8068310 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7fe7a000e970 tx=0x7fe7a000ec80 comp rx=0 tx=0).stop 2023-07-14T19:28:57.636 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:57.634+0000 7fe7af0f2700 1 -- 172.21.15.98:0/1198522029 shutdown_connections 2023-07-14T19:28:57.637 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:57.634+0000 7fe7af0f2700 1 --2- 172.21.15.98:0/1198522029 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7fe7a8100680 0x7fe7a819cdc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:28:57.637 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:57.634+0000 7fe7af0f2700 1 --2- 172.21.15.98:0/1198522029 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe7a80feb00 0x7fe7a8068310 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:28:57.637 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:57.634+0000 7fe7af0f2700 1 --2- 172.21.15.98:0/1198522029 >> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] conn(0x7fe7940a11a0 0x7fe7940a3650 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:28:57.637 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:57.634+0000 7fe7af0f2700 1 --2- 172.21.15.98:0/1198522029 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fe7a80ffcf0 0x7fe7a819a780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:28:57.637 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:57.634+0000 7fe7af0f2700 1 -- 172.21.15.98:0/1198522029 >> 172.21.15.98:0/1198522029 conn(0x7fe7a80fa100 msgr2=0x7fe7a80fc2c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:28:57.637 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:57.634+0000 7fe7af0f2700 1 -- 172.21.15.98:0/1198522029 shutdown_connections 2023-07-14T19:28:57.637 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:57.634+0000 7fe7af0f2700 1 -- 172.21.15.98:0/1198522029 wait complete. 2023-07-14T19:28:57.648 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-07-14T19:25:09.845827+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-07-14T19:28:57.648 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-07-14T19:28:57.775 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:57.774+0000 7f280122a700 1 -- 172.21.15.98:0/298675416 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f27fc100690 msgr2=0x7f27fc106f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:28:57.775 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:57.774+0000 7f280122a700 1 --2- 172.21.15.98:0/298675416 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f27fc100690 0x7f27fc106f50 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f27f400b0a0 tx=0x7f27f400b3b0 comp rx=0 tx=0).stop 2023-07-14T19:28:57.775 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:57.774+0000 7f280122a700 1 -- 172.21.15.98:0/298675416 shutdown_connections 2023-07-14T19:28:57.776 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:57.774+0000 7f280122a700 1 --2- 172.21.15.98:0/298675416 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f27fc0feb10 0x7f27fc0fef20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:28:57.776 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:57.774+0000 7f280122a700 1 --2- 172.21.15.98:0/298675416 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f27fc100690 0x7f27fc106f50 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:28:57.776 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:57.774+0000 7f280122a700 1 --2- 172.21.15.98:0/298675416 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f27fc0ffd00 0x7f27fc100150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:28:57.776 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:57.774+0000 7f280122a700 1 -- 172.21.15.98:0/298675416 >> 172.21.15.98:0/298675416 conn(0x7f27fc0fa130 msgr2=0x7f27fc0fc560 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:28:57.776 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:57.774+0000 7f280122a700 1 -- 172.21.15.98:0/298675416 shutdown_connections 2023-07-14T19:28:57.776 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:57.774+0000 7f280122a700 1 -- 172.21.15.98:0/298675416 wait complete. 2023-07-14T19:28:57.777 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:57.775+0000 7f280122a700 1 Processor -- start 2023-07-14T19:28:57.779 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:57.775+0000 7f280122a700 1 -- start start 2023-07-14T19:28:57.780 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:57.775+0000 7f280122a700 1 --2- >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f27fc0feb10 0x7f27fc068370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:28:57.780 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:57.775+0000 7f280122a700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f27fc0ffd00 0x7f27fc19a800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:28:57.780 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:57.775+0000 7f280122a700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f27fc100690 0x7f27fc19ce90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:28:57.780 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:57.775+0000 7f280122a700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f27fc19d630 con 0x7f27fc100690 2023-07-14T19:28:57.780 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:57.775+0000 7f280122a700 1 -- --> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f27fc19d7a0 con 0x7f27fc0feb10 2023-07-14T19:28:57.780 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:57.775+0000 7f280122a700 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f27fc19d960 con 0x7f27fc0ffd00 2023-07-14T19:28:57.781 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:57.775+0000 7f27fb59e700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f27fc100690 0x7f27fc19ce90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:28:57.781 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:57.775+0000 7f27fb59e700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f27fc100690 0x7f27fc19ce90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:45590/0 (socket says 172.21.15.98:45590) 2023-07-14T19:28:57.781 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:57.775+0000 7f27fa59c700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f27fc0ffd00 0x7f27fc19a800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:28:57.781 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:57.775+0000 7f27fb59e700 1 -- 172.21.15.98:0/3064899268 learned_addr learned my addr 172.21.15.98:0/3064899268 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-07-14T19:28:57.781 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:57.776+0000 7f27fb59e700 1 -- 172.21.15.98:0/3064899268 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f27fc0ffd00 msgr2=0x7f27fc19a800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:28:57.781 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:57.776+0000 7f27fb59e700 1 --2- 172.21.15.98:0/3064899268 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f27fc0ffd00 0x7f27fc19a800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:28:57.781 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:57.776+0000 7f27fb59e700 1 -- 172.21.15.98:0/3064899268 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f27fc0feb10 msgr2=0x7f27fc068370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:28:57.782 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:57.776+0000 7f27fb59e700 1 --2- 172.21.15.98:0/3064899268 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f27fc0feb10 0x7f27fc068370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:28:57.782 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:57.776+0000 7f27fb59e700 1 -- 172.21.15.98:0/3064899268 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f27f4009d00 con 0x7f27fc100690 2023-07-14T19:28:57.782 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:57.776+0000 7f27fb59e700 1 --2- 172.21.15.98:0/3064899268 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f27fc100690 0x7f27fc19ce90 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f27f400d010 tx=0x7f27f400bc80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:28:57.782 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:57.776+0000 7f27e3fff700 1 -- 172.21.15.98:0/3064899268 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f27f400e040 con 0x7f27fc100690 2023-07-14T19:28:57.782 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:57.776+0000 7f27e3fff700 1 -- 172.21.15.98:0/3064899268 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f27f4003e10 con 0x7f27fc100690 2023-07-14T19:28:57.782 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:57.776+0000 7f280122a700 1 -- 172.21.15.98:0/3064899268 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f27fc19dbe0 con 0x7f27fc100690 2023-07-14T19:28:57.782 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:57.776+0000 7f27e3fff700 1 -- 172.21.15.98:0/3064899268 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f27f4005260 con 0x7f27fc100690 2023-07-14T19:28:57.782 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:57.776+0000 7f280122a700 1 -- 172.21.15.98:0/3064899268 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f27fc19e000 con 0x7f27fc100690 2023-07-14T19:28:57.783 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:57.777+0000 7f280122a700 1 -- 172.21.15.98:0/3064899268 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f27fc106030 con 0x7f27fc100690 2023-07-14T19:28:57.783 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:57.778+0000 7f27e3fff700 1 -- 172.21.15.98:0/3064899268 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 16) v1 ==== 135715+0+0 (secure 0 0 0) 0x7f27f4005420 con 0x7f27fc100690 2023-07-14T19:28:57.783 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:57.778+0000 7f27e3fff700 1 --2- 172.21.15.98:0/3064899268 >> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] conn(0x7f27e40a1150 0x7f27e40a3600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:28:57.783 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:57.778+0000 7f27fad9d700 1 --2- 172.21.15.98:0/3064899268 >> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] conn(0x7f27e40a1150 0x7f27e40a3600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:28:57.783 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:57.779+0000 7f27e3fff700 1 -- 172.21.15.98:0/3064899268 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(79..79 src has 1..79) v4 ==== 7127+0+0 (secure 0 0 0) 0x7f27f40cd7d0 con 0x7f27fc100690 2023-07-14T19:28:57.783 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:57.779+0000 7f27fad9d700 1 --2- 172.21.15.98:0/3064899268 >> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] conn(0x7f27e40a1150 0x7f27e40a3600 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f27fc0ffb60 tx=0x7f27f0006d20 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:28:57.784 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:57.783+0000 7f27e3fff700 1 -- 172.21.15.98:0/3064899268 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f27f40973b0 con 0x7f27fc100690 2023-07-14T19:28:57.941 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:28:57 smithi101 ceph-mon[128807]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' 2023-07-14T19:28:57.942 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:28:57 smithi101 ceph-mon[128807]: from='client.? 172.21.15.98:0/3238159607' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-07-14T19:28:57.970 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:28:57 smithi139 ceph-mon[129293]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' 2023-07-14T19:28:57.971 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:28:57 smithi139 ceph-mon[129293]: from='client.? 172.21.15.98:0/3238159607' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-07-14T19:28:57.996 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:57.995+0000 7f280122a700 1 -- 172.21.15.98:0/3064899268 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f27fc061250 con 0x7f27fc100690 2023-07-14T19:28:57.996 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:57.995+0000 7f27e3fff700 1 -- 172.21.15.98:0/3064899268 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v79) v1 ==== 93+0+10 (secure 0 0 0) 0x7f27f409a9d0 con 0x7f27fc100690 2023-07-14T19:28:57.997 INFO:teuthology.orchestra.run.smithi098.stdout:pg_num: 1 2023-07-14T19:28:57.998 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:57.997+0000 7f280122a700 1 -- 172.21.15.98:0/3064899268 >> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] conn(0x7f27e40a1150 msgr2=0x7f27e40a3600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:28:57.999 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:57.997+0000 7f280122a700 1 --2- 172.21.15.98:0/3064899268 >> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] conn(0x7f27e40a1150 0x7f27e40a3600 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f27fc0ffb60 tx=0x7f27f0006d20 comp rx=0 tx=0).stop 2023-07-14T19:28:57.999 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:57.997+0000 7f280122a700 1 -- 172.21.15.98:0/3064899268 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f27fc100690 msgr2=0x7f27fc19ce90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:28:57.999 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:57.997+0000 7f280122a700 1 --2- 172.21.15.98:0/3064899268 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f27fc100690 0x7f27fc19ce90 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f27f400d010 tx=0x7f27f400bc80 comp rx=0 tx=0).stop 2023-07-14T19:28:58.000 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:57.997+0000 7f280122a700 1 -- 172.21.15.98:0/3064899268 shutdown_connections 2023-07-14T19:28:58.000 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:57.997+0000 7f280122a700 1 --2- 172.21.15.98:0/3064899268 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f27fc100690 0x7f27fc19ce90 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:28:58.001 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:57.997+0000 7f280122a700 1 --2- 172.21.15.98:0/3064899268 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f27fc0feb10 0x7f27fc068370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:28:58.001 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:57.997+0000 7f280122a700 1 --2- 172.21.15.98:0/3064899268 >> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] conn(0x7f27e40a1150 0x7f27e40a3600 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:28:58.001 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:57.997+0000 7f280122a700 1 --2- 172.21.15.98:0/3064899268 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f27fc0ffd00 0x7f27fc19a800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:28:58.002 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:57.997+0000 7f280122a700 1 -- 172.21.15.98:0/3064899268 >> 172.21.15.98:0/3064899268 conn(0x7f27fc0fa130 msgr2=0x7f27fc103170 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:28:58.002 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:57.998+0000 7f280122a700 1 -- 172.21.15.98:0/3064899268 shutdown_connections 2023-07-14T19:28:58.002 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:57.998+0000 7f280122a700 1 -- 172.21.15.98:0/3064899268 wait complete. 2023-07-14T19:28:58.009 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:28:57 smithi098 ceph-mon[127486]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' 2023-07-14T19:28:58.010 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:28:57 smithi098 ceph-mon[127486]: from='client.? 172.21.15.98:0/3238159607' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-07-14T19:28:58.011 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2023-07-14T19:28:58.136 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:58.135+0000 7f375d75e700 1 -- 172.21.15.98:0/3891472611 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f37580feb50 msgr2=0x7f37580fef60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:28:58.136 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:58.135+0000 7f375d75e700 1 --2- 172.21.15.98:0/3891472611 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f37580feb50 0x7f37580fef60 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f3748009990 tx=0x7f3748009ca0 comp rx=0 tx=0).stop 2023-07-14T19:28:58.137 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:58.135+0000 7f375d75e700 1 -- 172.21.15.98:0/3891472611 shutdown_connections 2023-07-14T19:28:58.137 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:58.135+0000 7f375d75e700 1 --2- 172.21.15.98:0/3891472611 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f37581006d0 0x7f3758106f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:28:58.137 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:58.135+0000 7f375d75e700 1 --2- 172.21.15.98:0/3891472611 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f37580feb50 0x7f37580fef60 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:28:58.137 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:58.135+0000 7f375d75e700 1 --2- 172.21.15.98:0/3891472611 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f37580ffd40 0x7f3758100190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:28:58.137 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:58.135+0000 7f375d75e700 1 -- 172.21.15.98:0/3891472611 >> 172.21.15.98:0/3891472611 conn(0x7f37580fa150 msgr2=0x7f37580fc5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:28:58.137 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:58.135+0000 7f375d75e700 1 -- 172.21.15.98:0/3891472611 shutdown_connections 2023-07-14T19:28:58.137 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:58.135+0000 7f375d75e700 1 -- 172.21.15.98:0/3891472611 wait complete. 2023-07-14T19:28:58.137 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:58.136+0000 7f375d75e700 1 Processor -- start 2023-07-14T19:28:58.139 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:58.136+0000 7f375d75e700 1 -- start start 2023-07-14T19:28:58.140 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:58.136+0000 7f375d75e700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f37580feb50 0x7f37581947e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:28:58.140 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:58.136+0000 7f375d75e700 1 --2- >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f37580ffd40 0x7f3758194d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:28:58.140 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:58.136+0000 7f375d75e700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f37581006d0 0x7f375819bd90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:28:58.140 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:58.136+0000 7f375d75e700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f375819c360 con 0x7f37581006d0 2023-07-14T19:28:58.140 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:58.136+0000 7f375d75e700 1 -- --> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f375819c4a0 con 0x7f37580ffd40 2023-07-14T19:28:58.140 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:58.136+0000 7f375d75e700 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f375819c630 con 0x7f37580feb50 2023-07-14T19:28:58.141 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:58.136+0000 7f37577fe700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f37581006d0 0x7f375819bd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:28:58.141 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:58.136+0000 7f37577fe700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f37581006d0 0x7f375819bd90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:45604/0 (socket says 172.21.15.98:45604) 2023-07-14T19:28:58.141 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:58.136+0000 7f37577fe700 1 -- 172.21.15.98:0/1629005833 learned_addr learned my addr 172.21.15.98:0/1629005833 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-07-14T19:28:58.141 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:58.136+0000 7f37567fc700 1 --2- 172.21.15.98:0/1629005833 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f37580ffd40 0x7f3758194d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:28:58.141 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:58.137+0000 7f3756ffd700 1 --2- 172.21.15.98:0/1629005833 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f37580feb50 0x7f37581947e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:28:58.141 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:58.137+0000 7f37577fe700 1 -- 172.21.15.98:0/1629005833 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f37580feb50 msgr2=0x7f37581947e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:28:58.141 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:58.137+0000 7f37577fe700 1 --2- 172.21.15.98:0/1629005833 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f37580feb50 0x7f37581947e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:28:58.142 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:58.137+0000 7f37577fe700 1 -- 172.21.15.98:0/1629005833 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f37580ffd40 msgr2=0x7f3758194d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:28:58.142 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:58.137+0000 7f37577fe700 1 --2- 172.21.15.98:0/1629005833 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f37580ffd40 0x7f3758194d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:28:58.142 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:58.137+0000 7f37577fe700 1 -- 172.21.15.98:0/1629005833 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f37480095f0 con 0x7f37581006d0 2023-07-14T19:28:58.142 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:58.137+0000 7f37577fe700 1 --2- 172.21.15.98:0/1629005833 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f37581006d0 0x7f375819bd90 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f3750007f10 tx=0x7f375000f5c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:28:58.142 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:58.137+0000 7f373ffff700 1 -- 172.21.15.98:0/1629005833 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f375000fde0 con 0x7f37581006d0 2023-07-14T19:28:58.142 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:58.137+0000 7f373ffff700 1 -- 172.21.15.98:0/1629005833 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f3750004bb0 con 0x7f37581006d0 2023-07-14T19:28:58.142 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:58.137+0000 7f373ffff700 1 -- 172.21.15.98:0/1629005833 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3750005020 con 0x7f37581006d0 2023-07-14T19:28:58.143 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:58.137+0000 7f375d75e700 1 -- 172.21.15.98:0/1629005833 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3758068a60 con 0x7f37581006d0 2023-07-14T19:28:58.143 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:58.137+0000 7f375d75e700 1 -- 172.21.15.98:0/1629005833 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3758068f20 con 0x7f37581006d0 2023-07-14T19:28:58.143 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:58.138+0000 7f375d75e700 1 -- 172.21.15.98:0/1629005833 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f375810a320 con 0x7f37581006d0 2023-07-14T19:28:58.144 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:58.140+0000 7f373ffff700 1 -- 172.21.15.98:0/1629005833 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 16) v1 ==== 135715+0+0 (secure 0 0 0) 0x7f3750004d20 con 0x7f37581006d0 2023-07-14T19:28:58.144 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:58.140+0000 7f373ffff700 1 --2- 172.21.15.98:0/1629005833 >> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] conn(0x7f37400a11a0 0x7f37400a3650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:28:58.144 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:58.140+0000 7f3756ffd700 1 --2- 172.21.15.98:0/1629005833 >> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] conn(0x7f37400a11a0 0x7f37400a3650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:28:58.144 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:58.140+0000 7f373ffff700 1 -- 172.21.15.98:0/1629005833 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(79..79 src has 1..79) v4 ==== 7127+0+0 (secure 0 0 0) 0x7f37500ceb50 con 0x7f37581006d0 2023-07-14T19:28:58.145 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:58.141+0000 7f3756ffd700 1 --2- 172.21.15.98:0/1629005833 >> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] conn(0x7f37400a11a0 0x7f37400a3650 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f3748006010 tx=0x7f374800b260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:28:58.145 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:58.143+0000 7f373ffff700 1 -- 172.21.15.98:0/1629005833 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f3750098740 con 0x7f37581006d0 2023-07-14T19:28:58.366 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:58.366+0000 7f375d75e700 1 -- 172.21.15.98:0/1629005833 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7f375810a510 con 0x7f37581006d0 2023-07-14T19:28:58.367 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:58.366+0000 7f373ffff700 1 -- 172.21.15.98:0/1629005833 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v79) v1 ==== 84+0+65 (secure 0 0 0) 0x7f375009bd60 con 0x7f37581006d0 2023-07-14T19:28:58.367 INFO:teuthology.orchestra.run.smithi098.stdout: 2023-07-14T19:28:58.367 INFO:teuthology.orchestra.run.smithi098.stdout:[ 2023-07-14T19:28:58.367 INFO:teuthology.orchestra.run.smithi098.stdout: { 2023-07-14T19:28:58.368 INFO:teuthology.orchestra.run.smithi098.stdout: "poolnum": 1, 2023-07-14T19:28:58.368 INFO:teuthology.orchestra.run.smithi098.stdout: "poolname": ".mgr" 2023-07-14T19:28:58.368 INFO:teuthology.orchestra.run.smithi098.stdout: } 2023-07-14T19:28:58.368 INFO:teuthology.orchestra.run.smithi098.stdout:] 2023-07-14T19:28:58.369 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:58.368+0000 7f375d75e700 1 -- 172.21.15.98:0/1629005833 >> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] conn(0x7f37400a11a0 msgr2=0x7f37400a3650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:28:58.369 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:58.368+0000 7f375d75e700 1 --2- 172.21.15.98:0/1629005833 >> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] conn(0x7f37400a11a0 0x7f37400a3650 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f3748006010 tx=0x7f374800b260 comp rx=0 tx=0).stop 2023-07-14T19:28:58.369 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:58.368+0000 7f375d75e700 1 -- 172.21.15.98:0/1629005833 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f37581006d0 msgr2=0x7f375819bd90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:28:58.369 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:58.368+0000 7f375d75e700 1 --2- 172.21.15.98:0/1629005833 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f37581006d0 0x7f375819bd90 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f3750007f10 tx=0x7f375000f5c0 comp rx=0 tx=0).stop 2023-07-14T19:28:58.369 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:58.368+0000 7f375d75e700 1 -- 172.21.15.98:0/1629005833 shutdown_connections 2023-07-14T19:28:58.370 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:58.368+0000 7f375d75e700 1 --2- 172.21.15.98:0/1629005833 >> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] conn(0x7f37400a11a0 0x7f37400a3650 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:28:58.370 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:58.368+0000 7f375d75e700 1 --2- 172.21.15.98:0/1629005833 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f37580feb50 0x7f37581947e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:28:58.370 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:58.368+0000 7f375d75e700 1 --2- 172.21.15.98:0/1629005833 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f37581006d0 0x7f375819bd90 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:28:58.370 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:58.368+0000 7f375d75e700 1 --2- 172.21.15.98:0/1629005833 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f37580ffd40 0x7f3758194d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:28:58.370 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:58.368+0000 7f375d75e700 1 -- 172.21.15.98:0/1629005833 >> 172.21.15.98:0/1629005833 conn(0x7f37580fa150 msgr2=0x7f37580fc310 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:28:58.370 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:58.369+0000 7f375d75e700 1 -- 172.21.15.98:0/1629005833 shutdown_connections 2023-07-14T19:28:58.370 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:58.369+0000 7f375d75e700 1 -- 172.21.15.98:0/1629005833 wait complete. 2023-07-14T19:28:58.382 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2023-07-14T19:28:58.383 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2023-07-14T19:28:58.511 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:58.508+0000 7ff6e27fc700 1 -- 172.21.15.98:0/923660836 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff6dc1006f0 msgr2=0x7ff6dc106fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:28:58.511 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:58.508+0000 7ff6e27fc700 1 --2- 172.21.15.98:0/923660836 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff6dc1006f0 0x7ff6dc106fb0 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7ff6d400b0a0 tx=0x7ff6d400b3b0 comp rx=0 tx=0).stop 2023-07-14T19:28:58.512 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:58.508+0000 7ff6e27fc700 1 -- 172.21.15.98:0/923660836 shutdown_connections 2023-07-14T19:28:58.512 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:58.508+0000 7ff6e27fc700 1 --2- 172.21.15.98:0/923660836 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff6dc1006f0 0x7ff6dc106fb0 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:28:58.512 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:58.508+0000 7ff6e27fc700 1 --2- 172.21.15.98:0/923660836 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7ff6dc0feb70 0x7ff6dc0fef80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:28:58.512 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:58.508+0000 7ff6e27fc700 1 --2- 172.21.15.98:0/923660836 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7ff6dc0ffd60 0x7ff6dc1001b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:28:58.512 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:58.508+0000 7ff6e27fc700 1 -- 172.21.15.98:0/923660836 >> 172.21.15.98:0/923660836 conn(0x7ff6dc0fa190 msgr2=0x7ff6dc0fc5c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:28:58.512 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:58.508+0000 7ff6e27fc700 1 -- 172.21.15.98:0/923660836 shutdown_connections 2023-07-14T19:28:58.512 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:58.508+0000 7ff6e27fc700 1 -- 172.21.15.98:0/923660836 wait complete. 2023-07-14T19:28:58.513 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:58.509+0000 7ff6e27fc700 1 Processor -- start 2023-07-14T19:28:58.513 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:58.509+0000 7ff6e27fc700 1 -- start start 2023-07-14T19:28:58.513 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:58.509+0000 7ff6e27fc700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff6dc0feb70 0x7ff6dc1949e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:28:58.513 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:58.509+0000 7ff6e27fc700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7ff6dc0ffd60 0x7ff6dc194f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:28:58.513 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:58.509+0000 7ff6e27fc700 1 --2- >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7ff6dc1006f0 0x7ff6dc19bf90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:28:58.513 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:58.509+0000 7ff6e27fc700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff6dc19c560 con 0x7ff6dc0feb70 2023-07-14T19:28:58.513 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:58.509+0000 7ff6e27fc700 1 -- --> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff6dc19c6a0 con 0x7ff6dc1006f0 2023-07-14T19:28:58.514 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:58.509+0000 7ff6e27fc700 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff6dc19c830 con 0x7ff6dc0ffd60 2023-07-14T19:28:58.514 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:58.509+0000 7ff6dbfff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff6dc0feb70 0x7ff6dc1949e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:28:58.514 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:58.509+0000 7ff6dbfff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff6dc0feb70 0x7ff6dc1949e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:45614/0 (socket says 172.21.15.98:45614) 2023-07-14T19:28:58.514 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:58.509+0000 7ff6dbfff700 1 -- 172.21.15.98:0/586381591 learned_addr learned my addr 172.21.15.98:0/586381591 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-07-14T19:28:58.515 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:58.509+0000 7ff6db7fe700 1 --2- 172.21.15.98:0/586381591 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7ff6dc0ffd60 0x7ff6dc194f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:28:58.515 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:58.510+0000 7ff6e0d99700 1 --2- 172.21.15.98:0/586381591 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7ff6dc1006f0 0x7ff6dc19bf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:28:58.515 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:58.510+0000 7ff6dbfff700 1 -- 172.21.15.98:0/586381591 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7ff6dc0ffd60 msgr2=0x7ff6dc194f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:28:58.515 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:58.510+0000 7ff6dbfff700 1 --2- 172.21.15.98:0/586381591 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7ff6dc0ffd60 0x7ff6dc194f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:28:58.516 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:58.510+0000 7ff6dbfff700 1 -- 172.21.15.98:0/586381591 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7ff6dc1006f0 msgr2=0x7ff6dc19bf90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:28:58.516 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:58.510+0000 7ff6dbfff700 1 --2- 172.21.15.98:0/586381591 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7ff6dc1006f0 0x7ff6dc19bf90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:28:58.517 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:58.510+0000 7ff6dbfff700 1 -- 172.21.15.98:0/586381591 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff6d4009d00 con 0x7ff6dc0feb70 2023-07-14T19:28:58.517 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:58.510+0000 7ff6dbfff700 1 --2- 172.21.15.98:0/586381591 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff6dc0feb70 0x7ff6dc1949e0 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7ff6cc0098f0 tx=0x7ff6cc009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:28:58.517 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:58.510+0000 7ff6d97fa700 1 -- 172.21.15.98:0/586381591 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff6cc012070 con 0x7ff6dc0feb70 2023-07-14T19:28:58.518 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:58.510+0000 7ff6d97fa700 1 -- 172.21.15.98:0/586381591 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7ff6cc0044e0 con 0x7ff6dc0feb70 2023-07-14T19:28:58.518 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:58.510+0000 7ff6d97fa700 1 -- 172.21.15.98:0/586381591 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff6cc003df0 con 0x7ff6dc0feb70 2023-07-14T19:28:58.518 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:58.510+0000 7ff6e27fc700 1 -- 172.21.15.98:0/586381591 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff6dc0689d0 con 0x7ff6dc0feb70 2023-07-14T19:28:58.519 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:58.510+0000 7ff6e27fc700 1 -- 172.21.15.98:0/586381591 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff6dc068e90 con 0x7ff6dc0feb70 2023-07-14T19:28:58.520 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:58.511+0000 7ff6e27fc700 1 -- 172.21.15.98:0/586381591 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff6dc10a330 con 0x7ff6dc0feb70 2023-07-14T19:28:58.521 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:58.513+0000 7ff6d97fa700 1 -- 172.21.15.98:0/586381591 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 16) v1 ==== 135715+0+0 (secure 0 0 0) 0x7ff6cc016030 con 0x7ff6dc0feb70 2023-07-14T19:28:58.521 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:58.514+0000 7ff6d97fa700 1 --2- 172.21.15.98:0/586381591 >> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] conn(0x7ff6c40a11d0 0x7ff6c40a3680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:28:58.521 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:58.514+0000 7ff6db7fe700 1 --2- 172.21.15.98:0/586381591 >> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] conn(0x7ff6c40a11d0 0x7ff6c40a3680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:28:58.522 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:58.514+0000 7ff6d97fa700 1 -- 172.21.15.98:0/586381591 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(79..79 src has 1..79) v4 ==== 7127+0+0 (secure 0 0 0) 0x7ff6cc0cd860 con 0x7ff6dc0feb70 2023-07-14T19:28:58.522 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:58.514+0000 7ff6db7fe700 1 --2- 172.21.15.98:0/586381591 >> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] conn(0x7ff6c40a11d0 0x7ff6c40a3680 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7ff6d0009c60 tx=0x7ff6d0009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:28:58.522 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:58.517+0000 7ff6d97fa700 1 -- 172.21.15.98:0/586381591 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7ff6cc097450 con 0x7ff6dc0feb70 2023-07-14T19:28:58.719 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:58.718+0000 7ff6e27fc700 1 -- 172.21.15.98:0/586381591 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) v1 -- 0x7ff6dc0697b0 con 0x7ff6dc0feb70 2023-07-14T19:28:58.781 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:28:58 smithi098 ceph-mon[127486]: pgmap v247: 1 pgs: 1 active+clean; 577 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-14T19:28:58.781 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:28:58 smithi098 ceph-mon[127486]: from='client.? 172.21.15.98:0/1198522029' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-07-14T19:28:58.781 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:28:58 smithi098 ceph-mon[127486]: from='client.? 172.21.15.98:0/3064899268' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2023-07-14T19:28:58.782 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:28:58 smithi098 ceph-mon[127486]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' 2023-07-14T19:28:58.782 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:28:58 smithi098 ceph-mon[127486]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' 2023-07-14T19:28:58.782 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:28:58 smithi098 ceph-mon[127486]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' 2023-07-14T19:28:58.782 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:28:58 smithi098 ceph-mon[127486]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' 2023-07-14T19:28:58.782 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:28:58 smithi098 ceph-mon[127486]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' 2023-07-14T19:28:58.782 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:28:58 smithi098 ceph-mon[127486]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' 2023-07-14T19:28:58.782 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:28:58 smithi098 ceph-mon[127486]: from='client.? 172.21.15.98:0/1629005833' entity='client.admin' cmd=[{"prefix": "osd lspools", "format": "json-pretty"}]: dispatch 2023-07-14T19:28:58.941 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:28:58 smithi101 ceph-mon[128807]: pgmap v247: 1 pgs: 1 active+clean; 577 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-14T19:28:58.942 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:28:58 smithi101 ceph-mon[128807]: from='client.? 172.21.15.98:0/1198522029' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-07-14T19:28:58.942 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:28:58 smithi101 ceph-mon[128807]: from='client.? 172.21.15.98:0/3064899268' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2023-07-14T19:28:58.942 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:28:58 smithi101 ceph-mon[128807]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' 2023-07-14T19:28:58.942 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:28:58 smithi101 ceph-mon[128807]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' 2023-07-14T19:28:58.942 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:28:58 smithi101 ceph-mon[128807]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' 2023-07-14T19:28:58.942 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:28:58 smithi101 ceph-mon[128807]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' 2023-07-14T19:28:58.943 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:28:58 smithi101 ceph-mon[128807]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' 2023-07-14T19:28:58.943 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:28:58 smithi101 ceph-mon[128807]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' 2023-07-14T19:28:58.943 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:28:58 smithi101 ceph-mon[128807]: from='client.? 172.21.15.98:0/1629005833' entity='client.admin' cmd=[{"prefix": "osd lspools", "format": "json-pretty"}]: dispatch 2023-07-14T19:28:58.970 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:28:58 smithi139 ceph-mon[129293]: pgmap v247: 1 pgs: 1 active+clean; 577 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-14T19:28:58.970 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:28:58 smithi139 ceph-mon[129293]: from='client.? 172.21.15.98:0/1198522029' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-07-14T19:28:58.971 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:28:58 smithi139 ceph-mon[129293]: from='client.? 172.21.15.98:0/3064899268' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2023-07-14T19:28:58.971 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:28:58 smithi139 ceph-mon[129293]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' 2023-07-14T19:28:58.971 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:28:58 smithi139 ceph-mon[129293]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' 2023-07-14T19:28:58.971 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:28:58 smithi139 ceph-mon[129293]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' 2023-07-14T19:28:58.971 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:28:58 smithi139 ceph-mon[129293]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' 2023-07-14T19:28:58.971 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:28:58 smithi139 ceph-mon[129293]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' 2023-07-14T19:28:58.971 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:28:58 smithi139 ceph-mon[129293]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' 2023-07-14T19:28:58.972 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:28:58 smithi139 ceph-mon[129293]: from='client.? 172.21.15.98:0/1629005833' entity='client.admin' cmd=[{"prefix": "osd lspools", "format": "json-pretty"}]: dispatch 2023-07-14T19:28:59.607 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:59.606+0000 7ff6d97fa700 1 -- 172.21.15.98:0/586381591 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v80) v1 ==== 138+0+0 (secure 0 0 0) 0x7ff6cc09aa70 con 0x7ff6dc0feb70 2023-07-14T19:28:59.610 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:59.609+0000 7ff6e27fc700 1 -- 172.21.15.98:0/586381591 >> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] conn(0x7ff6c40a11d0 msgr2=0x7ff6c40a3680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:28:59.611 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:59.609+0000 7ff6e27fc700 1 --2- 172.21.15.98:0/586381591 >> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] conn(0x7ff6c40a11d0 0x7ff6c40a3680 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7ff6d0009c60 tx=0x7ff6d0009260 comp rx=0 tx=0).stop 2023-07-14T19:28:59.611 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:59.610+0000 7ff6e27fc700 1 -- 172.21.15.98:0/586381591 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff6dc0feb70 msgr2=0x7ff6dc1949e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:28:59.612 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:59.610+0000 7ff6e27fc700 1 --2- 172.21.15.98:0/586381591 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff6dc0feb70 0x7ff6dc1949e0 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7ff6cc0098f0 tx=0x7ff6cc009c00 comp rx=0 tx=0).stop 2023-07-14T19:28:59.612 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:59.610+0000 7ff6e27fc700 1 -- 172.21.15.98:0/586381591 shutdown_connections 2023-07-14T19:28:59.612 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:59.610+0000 7ff6e27fc700 1 --2- 172.21.15.98:0/586381591 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7ff6dc1006f0 0x7ff6dc19bf90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:28:59.613 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:59.610+0000 7ff6e27fc700 1 --2- 172.21.15.98:0/586381591 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff6dc0feb70 0x7ff6dc1949e0 secure :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7ff6cc0098f0 tx=0x7ff6cc009c00 comp rx=0 tx=0).stop 2023-07-14T19:28:59.613 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:59.610+0000 7ff6e27fc700 1 --2- 172.21.15.98:0/586381591 >> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] conn(0x7ff6c40a11d0 0x7ff6c40a3680 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:28:59.613 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:59.610+0000 7ff6e27fc700 1 --2- 172.21.15.98:0/586381591 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7ff6dc0ffd60 0x7ff6dc194f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:28:59.614 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:59.610+0000 7ff6e27fc700 1 -- 172.21.15.98:0/586381591 >> 172.21.15.98:0/586381591 conn(0x7ff6dc0fa190 msgr2=0x7ff6dc0fc3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:28:59.615 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:59.611+0000 7ff6e27fc700 1 -- 172.21.15.98:0/586381591 shutdown_connections 2023-07-14T19:28:59.615 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:59.611+0000 7ff6e27fc700 1 -- 172.21.15.98:0/586381591 wait complete. 2023-07-14T19:28:59.615 INFO:teuthology.orchestra.run.smithi098.stderr:pool 'cephfs_metadata' created 2023-07-14T19:28:59.625 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2023-07-14T19:28:59.761 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:59.760+0000 7faf62d2e700 1 -- 172.21.15.98:0/504101826 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7faf5c0feb50 msgr2=0x7faf5c0fef60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:28:59.761 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:59.760+0000 7faf62d2e700 1 --2- 172.21.15.98:0/504101826 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7faf5c0feb50 0x7faf5c0fef60 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7faf50009990 tx=0x7faf50009ca0 comp rx=0 tx=0).stop 2023-07-14T19:28:59.762 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:59.760+0000 7faf62d2e700 1 -- 172.21.15.98:0/504101826 shutdown_connections 2023-07-14T19:28:59.762 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:59.760+0000 7faf62d2e700 1 --2- 172.21.15.98:0/504101826 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7faf5c1006d0 0x7faf5c106f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:28:59.762 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:59.760+0000 7faf62d2e700 1 --2- 172.21.15.98:0/504101826 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7faf5c0feb50 0x7faf5c0fef60 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:28:59.762 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:59.760+0000 7faf62d2e700 1 --2- 172.21.15.98:0/504101826 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7faf5c0ffd40 0x7faf5c100190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:28:59.762 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:59.760+0000 7faf62d2e700 1 -- 172.21.15.98:0/504101826 >> 172.21.15.98:0/504101826 conn(0x7faf5c0fa150 msgr2=0x7faf5c0fc5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:28:59.762 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:59.760+0000 7faf62d2e700 1 -- 172.21.15.98:0/504101826 shutdown_connections 2023-07-14T19:28:59.762 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:59.761+0000 7faf62d2e700 1 -- 172.21.15.98:0/504101826 wait complete. 2023-07-14T19:28:59.764 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:59.761+0000 7faf62d2e700 1 Processor -- start 2023-07-14T19:28:59.765 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:59.761+0000 7faf62d2e700 1 -- start start 2023-07-14T19:28:59.765 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:59.761+0000 7faf62d2e700 1 --2- >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7faf5c0feb50 0x7faf5c195a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:28:59.765 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:59.761+0000 7faf62d2e700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7faf5c0ffd40 0x7faf5c195f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:28:59.765 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:59.762+0000 7faf62d2e700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7faf5c1006d0 0x7faf5c19cfb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:28:59.765 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:59.762+0000 7faf62d2e700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faf5c19d580 con 0x7faf5c1006d0 2023-07-14T19:28:59.765 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:59.762+0000 7faf62d2e700 1 -- --> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faf5c19d6c0 con 0x7faf5c0feb50 2023-07-14T19:28:59.765 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:59.762+0000 7faf62d2e700 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faf5c19d850 con 0x7faf5c0ffd40 2023-07-14T19:28:59.766 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:59.762+0000 7faf5bfff700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7faf5c0ffd40 0x7faf5c195f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:28:59.766 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:59.762+0000 7faf612cb700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7faf5c1006d0 0x7faf5c19cfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:28:59.766 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:59.762+0000 7faf5bfff700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7faf5c0ffd40 0x7faf5c195f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.139:3300/0 says I am v2:172.21.15.98:52584/0 (socket says 172.21.15.98:52584) 2023-07-14T19:28:59.766 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:59.762+0000 7faf5bfff700 1 -- 172.21.15.98:0/3980117586 learned_addr learned my addr 172.21.15.98:0/3980117586 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-07-14T19:28:59.766 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:59.762+0000 7faf60aca700 1 --2- 172.21.15.98:0/3980117586 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7faf5c0feb50 0x7faf5c195a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:28:59.766 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:59.762+0000 7faf612cb700 1 -- 172.21.15.98:0/3980117586 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7faf5c0ffd40 msgr2=0x7faf5c195f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:28:59.766 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:59.762+0000 7faf612cb700 1 --2- 172.21.15.98:0/3980117586 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7faf5c0ffd40 0x7faf5c195f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:28:59.767 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:59.762+0000 7faf612cb700 1 -- 172.21.15.98:0/3980117586 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7faf5c0feb50 msgr2=0x7faf5c195a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:28:59.767 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:59.762+0000 7faf612cb700 1 --2- 172.21.15.98:0/3980117586 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7faf5c0feb50 0x7faf5c195a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:28:59.767 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:59.762+0000 7faf612cb700 1 -- 172.21.15.98:0/3980117586 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faf500095f0 con 0x7faf5c1006d0 2023-07-14T19:28:59.767 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:59.762+0000 7faf60aca700 1 --2- 172.21.15.98:0/3980117586 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7faf5c0feb50 0x7faf5c195a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-07-14T19:28:59.767 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:59.762+0000 7faf612cb700 1 --2- 172.21.15.98:0/3980117586 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7faf5c1006d0 0x7faf5c19cfb0 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7faf5400be20 tx=0x7faf54009ed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:28:59.767 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:59.762+0000 7faf59ffb700 1 -- 172.21.15.98:0/3980117586 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faf5400cc10 con 0x7faf5c1006d0 2023-07-14T19:28:59.767 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:59.762+0000 7faf59ffb700 1 -- 172.21.15.98:0/3980117586 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7faf5400cd70 con 0x7faf5c1006d0 2023-07-14T19:28:59.767 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:59.762+0000 7faf62d2e700 1 -- 172.21.15.98:0/3980117586 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faf5c19db30 con 0x7faf5c1006d0 2023-07-14T19:28:59.768 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:59.762+0000 7faf59ffb700 1 -- 172.21.15.98:0/3980117586 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faf540126c0 con 0x7faf5c1006d0 2023-07-14T19:28:59.768 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:59.763+0000 7faf62d2e700 1 -- 172.21.15.98:0/3980117586 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faf5c19e040 con 0x7faf5c1006d0 2023-07-14T19:28:59.768 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:59.763+0000 7faf62d2e700 1 -- 172.21.15.98:0/3980117586 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faf5c105a80 con 0x7faf5c1006d0 2023-07-14T19:28:59.769 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:59.764+0000 7faf59ffb700 1 -- 172.21.15.98:0/3980117586 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 16) v1 ==== 135715+0+0 (secure 0 0 0) 0x7faf54012820 con 0x7faf5c1006d0 2023-07-14T19:28:59.769 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:59.765+0000 7faf59ffb700 1 --2- 172.21.15.98:0/3980117586 >> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] conn(0x7faf4c0a11d0 0x7faf4c0a3680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:28:59.769 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:59.765+0000 7faf59ffb700 1 -- 172.21.15.98:0/3980117586 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(80..80 src has 1..80) v4 ==== 7497+0+0 (secure 0 0 0) 0x7faf540cbf20 con 0x7faf5c1006d0 2023-07-14T19:28:59.769 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:59.765+0000 7faf60aca700 1 --2- 172.21.15.98:0/3980117586 >> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] conn(0x7faf4c0a11d0 0x7faf4c0a3680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:28:59.769 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:59.765+0000 7faf60aca700 1 --2- 172.21.15.98:0/3980117586 >> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] conn(0x7faf4c0a11d0 0x7faf4c0a3680 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7faf5000b5c0 tx=0x7faf50005fb0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:28:59.770 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:59.769+0000 7faf59ffb700 1 -- 172.21.15.98:0/3980117586 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7faf5404e780 con 0x7faf5c1006d0 2023-07-14T19:28:59.941 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:28:59 smithi101 ceph-mon[128807]: from='client.? 172.21.15.98:0/586381591' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]: dispatch 2023-07-14T19:28:59.941 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:28:59 smithi101 ceph-mon[128807]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T19:28:59.941 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:28:59 smithi101 ceph-mon[128807]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-14T19:28:59.942 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:28:59 smithi101 ceph-mon[128807]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' 2023-07-14T19:28:59.942 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:28:59 smithi101 ceph-mon[128807]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-14T19:28:59.970 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:28:59 smithi139 ceph-mon[129293]: from='client.? 172.21.15.98:0/586381591' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]: dispatch 2023-07-14T19:28:59.970 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:28:59 smithi139 ceph-mon[129293]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T19:28:59.970 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:28:59 smithi139 ceph-mon[129293]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-14T19:28:59.971 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:28:59 smithi139 ceph-mon[129293]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' 2023-07-14T19:28:59.971 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:28:59 smithi139 ceph-mon[129293]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-14T19:28:59.982 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:28:59.982+0000 7faf62d2e700 1 -- 172.21.15.98:0/3980117586 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) v1 -- 0x7faf5c02c8e0 con 0x7faf5c1006d0 2023-07-14T19:29:00.031 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:28:59 smithi098 ceph-mon[127486]: from='client.? 172.21.15.98:0/586381591' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]: dispatch 2023-07-14T19:29:00.031 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:28:59 smithi098 ceph-mon[127486]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T19:29:00.031 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:28:59 smithi098 ceph-mon[127486]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-14T19:29:00.031 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:28:59 smithi098 ceph-mon[127486]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' 2023-07-14T19:29:00.032 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:28:59 smithi098 ceph-mon[127486]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-14T19:29:00.612 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:00.612+0000 7faf59ffb700 1 -- 172.21.15.98:0/3980117586 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v81) v1 ==== 170+0+0 (secure 0 0 0) 0x7faf54095900 con 0x7faf5c1006d0 2023-07-14T19:29:00.624 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:00.623+0000 7faf62d2e700 1 -- 172.21.15.98:0/3980117586 >> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] conn(0x7faf4c0a11d0 msgr2=0x7faf4c0a3680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:29:00.624 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:00.623+0000 7faf62d2e700 1 --2- 172.21.15.98:0/3980117586 >> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] conn(0x7faf4c0a11d0 0x7faf4c0a3680 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7faf5000b5c0 tx=0x7faf50005fb0 comp rx=0 tx=0).stop 2023-07-14T19:29:00.624 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:00.623+0000 7faf62d2e700 1 -- 172.21.15.98:0/3980117586 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7faf5c1006d0 msgr2=0x7faf5c19cfb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:29:00.624 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:00.623+0000 7faf62d2e700 1 --2- 172.21.15.98:0/3980117586 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7faf5c1006d0 0x7faf5c19cfb0 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7faf5400be20 tx=0x7faf54009ed0 comp rx=0 tx=0).stop 2023-07-14T19:29:00.625 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:00.624+0000 7faf62d2e700 1 -- 172.21.15.98:0/3980117586 shutdown_connections 2023-07-14T19:29:00.625 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:00.624+0000 7faf62d2e700 1 --2- 172.21.15.98:0/3980117586 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7faf5c1006d0 0x7faf5c19cfb0 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:29:00.625 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:00.624+0000 7faf62d2e700 1 --2- 172.21.15.98:0/3980117586 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7faf5c0feb50 0x7faf5c195a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:29:00.626 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:00.624+0000 7faf62d2e700 1 --2- 172.21.15.98:0/3980117586 >> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] conn(0x7faf4c0a11d0 0x7faf4c0a3680 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:29:00.626 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:00.624+0000 7faf62d2e700 1 --2- 172.21.15.98:0/3980117586 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7faf5c0ffd40 0x7faf5c195f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:29:00.626 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:00.624+0000 7faf62d2e700 1 -- 172.21.15.98:0/3980117586 >> 172.21.15.98:0/3980117586 conn(0x7faf5c0fa150 msgr2=0x7faf5c0fc3b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:29:00.627 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:00.626+0000 7faf62d2e700 1 -- 172.21.15.98:0/3980117586 shutdown_connections 2023-07-14T19:29:00.627 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:00.626+0000 7faf62d2e700 1 -- 172.21.15.98:0/3980117586 wait complete. 2023-07-14T19:29:00.627 INFO:teuthology.orchestra.run.smithi098.stderr:pool 'cephfs_data' created 2023-07-14T19:29:00.640 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2023-07-14T19:29:00.773 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:00.771+0000 7f2676eb6700 1 -- 172.21.15.98:0/743526384 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f26701006c0 msgr2=0x7f2670106f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:29:00.773 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:00.771+0000 7f2676eb6700 1 --2- 172.21.15.98:0/743526384 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f26701006c0 0x7f2670106f80 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f266800b0a0 tx=0x7f266800b3b0 comp rx=0 tx=0).stop 2023-07-14T19:29:00.774 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:00.771+0000 7f2676eb6700 1 -- 172.21.15.98:0/743526384 shutdown_connections 2023-07-14T19:29:00.774 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:00.771+0000 7f2676eb6700 1 --2- 172.21.15.98:0/743526384 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f26700feb40 0x7f26700fef50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:29:00.774 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:00.771+0000 7f2676eb6700 1 --2- 172.21.15.98:0/743526384 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f26701006c0 0x7f2670106f80 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:29:00.774 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:00.771+0000 7f2676eb6700 1 --2- 172.21.15.98:0/743526384 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f26700ffd30 0x7f2670100180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:29:00.774 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:00.771+0000 7f2676eb6700 1 -- 172.21.15.98:0/743526384 >> 172.21.15.98:0/743526384 conn(0x7f26700fa160 msgr2=0x7f26700fc590 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:29:00.774 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:00.771+0000 7f2676eb6700 1 -- 172.21.15.98:0/743526384 shutdown_connections 2023-07-14T19:29:00.775 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:00.771+0000 7f2676eb6700 1 -- 172.21.15.98:0/743526384 wait complete. 2023-07-14T19:29:00.775 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:00.772+0000 7f2676eb6700 1 Processor -- start 2023-07-14T19:29:00.775 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:00.772+0000 7f2676eb6700 1 -- start start 2023-07-14T19:29:00.775 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:00.772+0000 7f2676eb6700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f26700feb40 0x7f2670195a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:29:00.775 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:00.772+0000 7f2676eb6700 1 --2- >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f26700ffd30 0x7f2670195f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:29:00.775 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:00.772+0000 7f2676eb6700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f26701006c0 0x7f267019cff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:29:00.775 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:00.772+0000 7f2676eb6700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f267019d5c0 con 0x7f26701006c0 2023-07-14T19:29:00.776 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:00.772+0000 7f2676eb6700 1 -- --> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f267019d700 con 0x7f26700ffd30 2023-07-14T19:29:00.776 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:00.772+0000 7f2676eb6700 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f267019d890 con 0x7f26700feb40 2023-07-14T19:29:00.776 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:00.772+0000 7f2675453700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f26701006c0 0x7f267019cff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:29:00.776 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:00.772+0000 7f266ffff700 1 --2- >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f26700ffd30 0x7f2670195f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:29:00.776 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:00.772+0000 7f2675453700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f26701006c0 0x7f267019cff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:45646/0 (socket says 172.21.15.98:45646) 2023-07-14T19:29:00.777 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:00.772+0000 7f2675453700 1 -- 172.21.15.98:0/30005641 learned_addr learned my addr 172.21.15.98:0/30005641 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-07-14T19:29:00.777 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:00.772+0000 7f2674c52700 1 --2- 172.21.15.98:0/30005641 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f26700feb40 0x7f2670195a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:29:00.777 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:00.772+0000 7f2675453700 1 -- 172.21.15.98:0/30005641 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f26700feb40 msgr2=0x7f2670195a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:29:00.777 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:00.772+0000 7f2675453700 1 --2- 172.21.15.98:0/30005641 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f26700feb40 0x7f2670195a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:29:00.777 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:00.772+0000 7f2675453700 1 -- 172.21.15.98:0/30005641 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f26700ffd30 msgr2=0x7f2670195f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:29:00.777 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:00.772+0000 7f2675453700 1 --2- 172.21.15.98:0/30005641 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f26700ffd30 0x7f2670195f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:29:00.777 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:00.772+0000 7f2675453700 1 -- 172.21.15.98:0/30005641 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2668009d00 con 0x7f26701006c0 2023-07-14T19:29:00.778 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:00.772+0000 7f2674c52700 1 --2- 172.21.15.98:0/30005641 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f26700feb40 0x7f2670195a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-07-14T19:29:00.778 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:00.772+0000 7f2675453700 1 --2- 172.21.15.98:0/30005641 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f26701006c0 0x7f267019cff0 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f266800b830 tx=0x7f2668004c80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:29:00.778 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:00.772+0000 7f266dffb700 1 -- 172.21.15.98:0/30005641 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f266800e040 con 0x7f26701006c0 2023-07-14T19:29:00.778 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:00.772+0000 7f266dffb700 1 -- 172.21.15.98:0/30005641 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f266801db50 con 0x7f26701006c0 2023-07-14T19:29:00.778 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:00.772+0000 7f2676eb6700 1 -- 172.21.15.98:0/30005641 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f267019db10 con 0x7f26701006c0 2023-07-14T19:29:00.778 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:00.773+0000 7f266dffb700 1 -- 172.21.15.98:0/30005641 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f266801bbd0 con 0x7f26701006c0 2023-07-14T19:29:00.778 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:00.773+0000 7f2676eb6700 1 -- 172.21.15.98:0/30005641 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f267019df90 con 0x7f26701006c0 2023-07-14T19:29:00.779 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:00.774+0000 7f2676eb6700 1 -- 172.21.15.98:0/30005641 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2670106b40 con 0x7f26701006c0 2023-07-14T19:29:00.779 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:00.775+0000 7f266dffb700 1 -- 172.21.15.98:0/30005641 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 16) v1 ==== 135715+0+0 (secure 0 0 0) 0x7f266801dcc0 con 0x7f26701006c0 2023-07-14T19:29:00.780 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:00.775+0000 7f266dffb700 1 --2- 172.21.15.98:0/30005641 >> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] conn(0x7f26600a1150 0x7f26600a3600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:29:00.780 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:00.776+0000 7f2674c52700 1 --2- 172.21.15.98:0/30005641 >> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] conn(0x7f26600a1150 0x7f26600a3600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:29:00.780 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:00.776+0000 7f266dffb700 1 -- 172.21.15.98:0/30005641 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 7879+0+0 (secure 0 0 0) 0x7f26680ce9c0 con 0x7f26701006c0 2023-07-14T19:29:00.780 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:00.776+0000 7f2674c52700 1 --2- 172.21.15.98:0/30005641 >> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] conn(0x7f26600a1150 0x7f26600a3600 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f2664005fd0 tx=0x7f2664005f00 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:29:00.780 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:00.779+0000 7f266dffb700 1 -- 172.21.15.98:0/30005641 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f2668098350 con 0x7f26701006c0 2023-07-14T19:29:00.941 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:29:00 smithi101 ceph-mon[128807]: pgmap v248: 1 pgs: 1 active+clean; 577 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-14T19:29:00.941 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:29:00 smithi101 ceph-mon[128807]: from='client.? 172.21.15.98:0/586381591' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]': finished 2023-07-14T19:29:00.941 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:29:00 smithi101 ceph-mon[128807]: osdmap e80: 12 total, 12 up, 12 in 2023-07-14T19:29:00.942 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:29:00 smithi101 ceph-mon[128807]: from='client.? 172.21.15.98:0/3980117586' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]: dispatch 2023-07-14T19:29:00.970 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:29:00 smithi139 ceph-mon[129293]: pgmap v248: 1 pgs: 1 active+clean; 577 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-14T19:29:00.970 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:29:00 smithi139 ceph-mon[129293]: from='client.? 172.21.15.98:0/586381591' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]': finished 2023-07-14T19:29:00.971 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:29:00 smithi139 ceph-mon[129293]: osdmap e80: 12 total, 12 up, 12 in 2023-07-14T19:29:00.971 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:29:00 smithi139 ceph-mon[129293]: from='client.? 172.21.15.98:0/3980117586' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]: dispatch 2023-07-14T19:29:01.031 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:29:00 smithi098 ceph-mon[127486]: pgmap v248: 1 pgs: 1 active+clean; 577 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-14T19:29:01.031 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:29:00 smithi098 ceph-mon[127486]: from='client.? 172.21.15.98:0/586381591' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]': finished 2023-07-14T19:29:01.032 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:29:00 smithi098 ceph-mon[127486]: osdmap e80: 12 total, 12 up, 12 in 2023-07-14T19:29:01.032 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:29:00 smithi098 ceph-mon[127486]: from='client.? 172.21.15.98:0/3980117586' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]: dispatch 2023-07-14T19:29:01.033 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:01.031+0000 7f2676eb6700 1 -- 172.21.15.98:0/30005641 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) v1 -- 0x7f267002c8e0 con 0x7f26701006c0 2023-07-14T19:29:01.632 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:01.631+0000 7f266dffb700 1 -- 172.21.15.98:0/30005641 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 2 and data pool 3 v14) v1 ==== 172+0+0 (secure 0 0 0) 0x7f266809b970 con 0x7f26701006c0 2023-07-14T19:29:01.634 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:01.633+0000 7f2676eb6700 1 -- 172.21.15.98:0/30005641 >> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] conn(0x7f26600a1150 msgr2=0x7f26600a3600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:29:01.634 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:01.633+0000 7f2676eb6700 1 --2- 172.21.15.98:0/30005641 >> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] conn(0x7f26600a1150 0x7f26600a3600 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f2664005fd0 tx=0x7f2664005f00 comp rx=0 tx=0).stop 2023-07-14T19:29:01.634 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:01.633+0000 7f2676eb6700 1 -- 172.21.15.98:0/30005641 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f26701006c0 msgr2=0x7f267019cff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:29:01.635 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:01.633+0000 7f2676eb6700 1 --2- 172.21.15.98:0/30005641 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f26701006c0 0x7f267019cff0 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f266800b830 tx=0x7f2668004c80 comp rx=0 tx=0).stop 2023-07-14T19:29:01.635 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:01.633+0000 7f2676eb6700 1 -- 172.21.15.98:0/30005641 shutdown_connections 2023-07-14T19:29:01.635 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:01.633+0000 7f2676eb6700 1 --2- 172.21.15.98:0/30005641 >> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] conn(0x7f26600a1150 0x7f26600a3600 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:29:01.635 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:01.633+0000 7f2676eb6700 1 --2- 172.21.15.98:0/30005641 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f26700feb40 0x7f2670195a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:29:01.635 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:01.633+0000 7f2676eb6700 1 --2- 172.21.15.98:0/30005641 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f26701006c0 0x7f267019cff0 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:29:01.636 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:01.633+0000 7f2676eb6700 1 --2- 172.21.15.98:0/30005641 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f26700ffd30 0x7f2670195f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:29:01.636 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:01.634+0000 7f2676eb6700 1 -- 172.21.15.98:0/30005641 >> 172.21.15.98:0/30005641 conn(0x7f26700fa160 msgr2=0x7f26700fc2d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:29:01.637 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:01.634+0000 7f2676eb6700 1 -- 172.21.15.98:0/30005641 shutdown_connections 2023-07-14T19:29:01.637 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:01.634+0000 7f2676eb6700 1 -- 172.21.15.98:0/30005641 wait complete. 2023-07-14T19:29:01.637 INFO:teuthology.orchestra.run.smithi098.stderr:new fs with metadata pool 2 and data pool 3 2023-07-14T19:29:01.650 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-07-14T19:29:01.774 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:01.773+0000 7f36b680f700 1 -- 172.21.15.98:0/2576796584 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f36b01017b0 msgr2=0x7f36b0108070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:29:01.774 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:01.773+0000 7f36b680f700 1 --2- 172.21.15.98:0/2576796584 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f36b01017b0 0x7f36b0108070 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f36a800b0a0 tx=0x7f36a800b3b0 comp rx=0 tx=0).stop 2023-07-14T19:29:01.774 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:01.773+0000 7f36b680f700 1 -- 172.21.15.98:0/2576796584 shutdown_connections 2023-07-14T19:29:01.774 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:01.773+0000 7f36b680f700 1 --2- 172.21.15.98:0/2576796584 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f36b00ffc30 0x7f36b0100040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:29:01.775 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:01.773+0000 7f36b680f700 1 --2- 172.21.15.98:0/2576796584 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f36b01017b0 0x7f36b0108070 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:29:01.775 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:01.773+0000 7f36b680f700 1 --2- 172.21.15.98:0/2576796584 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f36b0100e20 0x7f36b0101270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:29:01.775 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:01.773+0000 7f36b680f700 1 -- 172.21.15.98:0/2576796584 >> 172.21.15.98:0/2576796584 conn(0x7f36b00fb230 msgr2=0x7f36b00fd680 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:29:01.776 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:01.773+0000 7f36b680f700 1 -- 172.21.15.98:0/2576796584 shutdown_connections 2023-07-14T19:29:01.776 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:01.773+0000 7f36b680f700 1 -- 172.21.15.98:0/2576796584 wait complete. 2023-07-14T19:29:01.776 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:01.773+0000 7f36b680f700 1 Processor -- start 2023-07-14T19:29:01.776 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:01.773+0000 7f36b680f700 1 -- start start 2023-07-14T19:29:01.780 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:01.774+0000 7f36b680f700 1 --2- >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f36b00ffc30 0x7f36b00743f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:29:01.780 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:01.774+0000 7f36b680f700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f36b0100e20 0x7f36b0072a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:29:01.780 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:01.774+0000 7f36b680f700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f36b01017b0 0x7f36b0072f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:29:01.781 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:01.774+0000 7f36b680f700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f36b0073710 con 0x7f36b0100e20 2023-07-14T19:29:01.781 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:01.774+0000 7f36b680f700 1 -- --> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f36b0073880 con 0x7f36b00ffc30 2023-07-14T19:29:01.781 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:01.774+0000 7f36b680f700 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f36b019fa90 con 0x7f36b01017b0 2023-07-14T19:29:01.781 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:01.774+0000 7f36af7fe700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f36b0100e20 0x7f36b0072a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:29:01.781 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:01.774+0000 7f36affff700 1 --2- >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f36b00ffc30 0x7f36b00743f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:29:01.782 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:01.774+0000 7f36af7fe700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f36b0100e20 0x7f36b0072a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:45666/0 (socket says 172.21.15.98:45666) 2023-07-14T19:29:01.782 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:01.774+0000 7f36af7fe700 1 -- 172.21.15.98:0/1659408846 learned_addr learned my addr 172.21.15.98:0/1659408846 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-07-14T19:29:01.782 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:01.774+0000 7f36b4dac700 1 --2- 172.21.15.98:0/1659408846 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f36b01017b0 0x7f36b0072f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:29:01.782 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:01.774+0000 7f36affff700 1 -- 172.21.15.98:0/1659408846 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f36b01017b0 msgr2=0x7f36b0072f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:29:01.783 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:01.774+0000 7f36affff700 1 --2- 172.21.15.98:0/1659408846 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f36b01017b0 0x7f36b0072f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:29:01.783 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:01.774+0000 7f36affff700 1 -- 172.21.15.98:0/1659408846 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f36b0100e20 msgr2=0x7f36b0072a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:29:01.783 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:01.774+0000 7f36affff700 1 --2- 172.21.15.98:0/1659408846 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f36b0100e20 0x7f36b0072a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:29:01.783 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:01.774+0000 7f36affff700 1 -- 172.21.15.98:0/1659408846 --> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f36a8009d00 con 0x7f36b00ffc30 2023-07-14T19:29:01.784 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:01.774+0000 7f36af7fe700 1 --2- 172.21.15.98:0/1659408846 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f36b0100e20 0x7f36b0072a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-07-14T19:29:01.784 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:01.774+0000 7f36affff700 1 --2- 172.21.15.98:0/1659408846 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f36b00ffc30 0x7f36b00743f0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f36a000e9c0 tx=0x7f36a000ecd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:29:01.784 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:01.774+0000 7f36ad7fa700 1 -- 172.21.15.98:0/1659408846 <== mon.2 v2:172.21.15.101:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f36a000cbd0 con 0x7f36b00ffc30 2023-07-14T19:29:01.784 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:01.774+0000 7f36b680f700 1 -- 172.21.15.98:0/1659408846 --> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f36b019fd70 con 0x7f36b00ffc30 2023-07-14T19:29:01.784 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:01.774+0000 7f36ad7fa700 1 -- 172.21.15.98:0/1659408846 <== mon.2 v2:172.21.15.101:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f36a000eeb0 con 0x7f36b00ffc30 2023-07-14T19:29:01.784 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:01.774+0000 7f36ad7fa700 1 -- 172.21.15.98:0/1659408846 <== mon.2 v2:172.21.15.101:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f36a0003c50 con 0x7f36b00ffc30 2023-07-14T19:29:01.784 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:01.775+0000 7f36b680f700 1 -- 172.21.15.98:0/1659408846 --> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f36b01a01c0 con 0x7f36b00ffc30 2023-07-14T19:29:01.785 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:01.775+0000 7f36b680f700 1 -- 172.21.15.98:0/1659408846 --> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f36b01070f0 con 0x7f36b00ffc30 2023-07-14T19:29:01.785 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:01.776+0000 7f36ad7fa700 1 -- 172.21.15.98:0/1659408846 <== mon.2 v2:172.21.15.101:3300/0 4 ==== mgrmap(e 16) v1 ==== 135715+0+0 (secure 0 0 0) 0x7f36a0004870 con 0x7f36b00ffc30 2023-07-14T19:29:01.785 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:01.777+0000 7f36ad7fa700 1 --2- 172.21.15.98:0/1659408846 >> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] conn(0x7f36980a11a0 0x7f36980a3650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:29:01.785 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:01.777+0000 7f36af7fe700 1 --2- 172.21.15.98:0/1659408846 >> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] conn(0x7f36980a11a0 0x7f36980a3650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:29:01.785 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:01.777+0000 7f36ad7fa700 1 -- 172.21.15.98:0/1659408846 <== mon.2 v2:172.21.15.101:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 7979+0+0 (secure 0 0 0) 0x7f36a0014070 con 0x7f36b00ffc30 2023-07-14T19:29:01.785 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:01.777+0000 7f36af7fe700 1 --2- 172.21.15.98:0/1659408846 >> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] conn(0x7f36980a11a0 0x7f36980a3650 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f36a4011550 tx=0x7f36a4009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:29:01.786 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:01.780+0000 7f36ad7fa700 1 -- 172.21.15.98:0/1659408846 <== mon.2 v2:172.21.15.101:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f36a00f9780 con 0x7f36b00ffc30 2023-07-14T19:29:01.941 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:29:01 smithi101 ceph-mon[128807]: from='client.? 172.21.15.98:0/3980117586' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]': finished 2023-07-14T19:29:01.941 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:29:01 smithi101 ceph-mon[128807]: osdmap e81: 12 total, 12 up, 12 in 2023-07-14T19:29:01.942 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:29:01 smithi101 ceph-mon[128807]: from='client.? 172.21.15.98:0/30005641' entity='client.admin' cmd=[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]: dispatch 2023-07-14T19:29:01.942 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:29:01 smithi101 ceph-mon[128807]: daemon mds.b assigned to filesystem cephfs as rank 0 2023-07-14T19:29:01.942 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:29:01 smithi101 ceph-mon[128807]: osdmap e82: 12 total, 12 up, 12 in 2023-07-14T19:29:01.970 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:29:01 smithi139 ceph-mon[129293]: from='client.? 172.21.15.98:0/3980117586' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]': finished 2023-07-14T19:29:01.970 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:29:01 smithi139 ceph-mon[129293]: osdmap e81: 12 total, 12 up, 12 in 2023-07-14T19:29:01.970 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:29:01 smithi139 ceph-mon[129293]: from='client.? 172.21.15.98:0/30005641' entity='client.admin' cmd=[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]: dispatch 2023-07-14T19:29:01.971 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:29:01 smithi139 ceph-mon[129293]: daemon mds.b assigned to filesystem cephfs as rank 0 2023-07-14T19:29:01.971 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:29:01 smithi139 ceph-mon[129293]: osdmap e82: 12 total, 12 up, 12 in 2023-07-14T19:29:01.984 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:01.984+0000 7f36b680f700 1 -- 172.21.15.98:0/1659408846 --> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f36b002cb90 con 0x7f36b00ffc30 2023-07-14T19:29:01.985 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:01.985+0000 7f36ad7fa700 1 -- 172.21.15.98:0/1659408846 <== mon.2 v2:172.21.15.101:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v82) v1 ==== 74+0+22130 (secure 0 0 0) 0x7f36a00968f0 con 0x7f36b00ffc30 2023-07-14T19:29:01.986 INFO:teuthology.orchestra.run.smithi098.stdout: 2023-07-14T19:29:01.986 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":82,"fsid":"9adea648-227b-11ee-9b33-001a4aab830c","created":"2023-07-14T19:22:07.178854+0000","modified":"2023-07-14T19:29:01.032629+0000","last_up_change":"2023-07-14T19:28:19.646827+0000","last_in_change":"2023-07-14T19:28:02.018325+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-07-14T19:25:09.845827+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2023-07-14T19:28:58.952921+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"82","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2023-07-14T19:29:00.228272+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"82","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"a3797e89-1754-41cf-9517-227e06ffdd72","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6802","nonce":120507228},{"type":"v1","addr":"172.21.15.98:6803","nonce":120507228}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6804","nonce":120507228},{"type":"v1","addr":"172.21.15.98:6805","nonce":120507228}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6808","nonce":120507228},{"type":"v1","addr":"172.21.15.98:6809","nonce":120507228}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6806","nonce":120507228},{"type":"v1","addr":"172.21.15.98:6807","nonce":120507228}]},"public_addr":"172.21.15.98:6803/120507228","cluster_addr":"172.21.15.98:6805/120507228","heartbeat_back_addr":"172.21.15.98:6809/120507228","heartbeat_front_addr":"172.21.15.98:6807/120507228","state":["exists","up"]},{"osd":1,"uuid":"2b001add-1aaa-4496-a403-7ef9725b80a0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.101:6800","nonce":2450584584},{"type":"v1","addr":"172.21.15.101:6801","nonce":2450584584}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.101:6802","nonce":2450584584},{"type":"v1","addr":"172.21.15.101:6803","nonce":2450584584}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.101:6806","nonce":2450584584},{"type":"v1","addr":"172.21.15.101:6807","nonce":2450584584}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.101:6804","nonce":2450584584},{"type":"v1","addr":"172.21.15.101:6805","nonce":2450584584}]},"public_addr":"172.21.15.101:6801/2450584584","cluster_addr":"172.21.15.101:6803/2450584584","heartbeat_back_addr":"172.21.15.101:6807/2450584584","heartbeat_front_addr":"172.21.15.101:6805/2450584584","state":["exists","up"]},{"osd":2,"uuid":"48e6d499-3ca9-424b-b0fb-3ba5f33b9fc9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":945038995},{"type":"v1","addr":"172.21.15.139:6801","nonce":945038995}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":945038995},{"type":"v1","addr":"172.21.15.139:6803","nonce":945038995}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":945038995},{"type":"v1","addr":"172.21.15.139:6807","nonce":945038995}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":945038995},{"type":"v1","addr":"172.21.15.139:6805","nonce":945038995}]},"public_addr":"172.21.15.139:6801/945038995","cluster_addr":"172.21.15.139:6803/945038995","heartbeat_back_addr":"172.21.15.139:6807/945038995","heartbeat_front_addr":"172.21.15.139:6805/945038995","state":["exists","up"]},{"osd":3,"uuid":"5530b265-31a5-447b-a1ea-17ebae9cfddf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6810","nonce":3095249665},{"type":"v1","addr":"172.21.15.98:6811","nonce":3095249665}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6812","nonce":3095249665},{"type":"v1","addr":"172.21.15.98:6813","nonce":3095249665}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6816","nonce":3095249665},{"type":"v1","addr":"172.21.15.98:6817","nonce":3095249665}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6814","nonce":3095249665},{"type":"v1","addr":"172.21.15.98:6815","nonce":3095249665}]},"public_addr":"172.21.15.98:6811/3095249665","cluster_addr":"172.21.15.98:6813/3095249665","heartbeat_back_addr":"172.21.15.98:6817/3095249665","heartbeat_front_addr":"172.21.15.98:6815/3095249665","state":["exists","up"]},{"osd":4,"uuid":"615d4642-a997-4f95-9e38-9fbdaa831400","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":32,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.101:6808","nonce":2844761406},{"type":"v1","addr":"172.21.15.101:6809","nonce":2844761406}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.101:6810","nonce":2844761406},{"type":"v1","addr":"172.21.15.101:6811","nonce":2844761406}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.101:6814","nonce":2844761406},{"type":"v1","addr":"172.21.15.101:6815","nonce":2844761406}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.101:6812","nonce":2844761406},{"type":"v1","addr":"172.21.15.101:6813","nonce":2844761406}]},"public_addr":"172.21.15.101:6809/2844761406","cluster_addr":"172.21.15.101:6811/2844761406","heartbeat_back_addr":"172.21.15.101:6815/2844761406","heartbeat_front_addr":"172.21.15.101:6813/2844761406","state":["exists","up"]},{"osd":5,"uuid":"415de5fa-607a-45a8-b732-28daf3be6dd3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":37,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":3532263042},{"type":"v1","addr":"172.21.15.139:6809","nonce":3532263042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6810","nonce":3532263042},{"type":"v1","addr":"172.21.15.139:6811","nonce":3532263042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6814","nonce":3532263042},{"type":"v1","addr":"172.21.15.139:6815","nonce":3532263042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":3532263042},{"type":"v1","addr":"172.21.15.139:6813","nonce":3532263042}]},"public_addr":"172.21.15.139:6809/3532263042","cluster_addr":"172.21.15.139:6811/3532263042","heartbeat_back_addr":"172.21.15.139:6815/3532263042","heartbeat_front_addr":"172.21.15.139:6813/3532263042","state":["exists","up"]},{"osd":6,"uuid":"d7f9d4ba-c309-4b75-a836-5a36384b528e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":45,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6818","nonce":658948743},{"type":"v1","addr":"172.21.15.98:6819","nonce":658948743}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6820","nonce":658948743},{"type":"v1","addr":"172.21.15.98:6821","nonce":658948743}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6824","nonce":658948743},{"type":"v1","addr":"172.21.15.98:6825","nonce":658948743}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6822","nonce":658948743},{"type":"v1","addr":"172.21.15.98:6823","nonce":658948743}]},"public_addr":"172.21.15.98:6819/658948743","cluster_addr":"172.21.15.98:6821/658948743","heartbeat_back_addr":"172.21.15.98:6825/658948743","heartbeat_front_addr":"172.21.15.98:6823/658948743","state":["exists","up"]},{"osd":7,"uuid":"eedb94d9-c5cd-4a93-8a8c-2648804b7195","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":50,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.101:6816","nonce":2881214066},{"type":"v1","addr":"172.21.15.101:6817","nonce":2881214066}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.101:6818","nonce":2881214066},{"type":"v1","addr":"172.21.15.101:6819","nonce":2881214066}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.101:6822","nonce":2881214066},{"type":"v1","addr":"172.21.15.101:6823","nonce":2881214066}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.101:6820","nonce":2881214066},{"type":"v1","addr":"172.21.15.101:6821","nonce":2881214066}]},"public_addr":"172.21.15.101:6817/2881214066","cluster_addr":"172.21.15.101:6819/2881214066","heartbeat_back_addr":"172.21.15.101:6823/2881214066","heartbeat_front_addr":"172.21.15.101:6821/2881214066","state":["exists","up"]},{"osd":8,"uuid":"6bfd77c2-5770-466e-aa6f-84b50c74ffb6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":57,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":3669514524},{"type":"v1","addr":"172.21.15.139:6817","nonce":3669514524}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":3669514524},{"type":"v1","addr":"172.21.15.139:6819","nonce":3669514524}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":3669514524},{"type":"v1","addr":"172.21.15.139:6823","nonce":3669514524}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6820","nonce":3669514524},{"type":"v1","addr":"172.21.15.139:6821","nonce":3669514524}]},"public_addr":"172.21.15.139:6817/3669514524","cluster_addr":"172.21.15.139:6819/3669514524","heartbeat_back_addr":"172.21.15.139:6823/3669514524","heartbeat_front_addr":"172.21.15.139:6821/3669514524","state":["exists","up"]},{"osd":9,"uuid":"19c5b54a-2242-4b23-b721-63be9559653d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":65,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6826","nonce":2893994080},{"type":"v1","addr":"172.21.15.98:6827","nonce":2893994080}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6828","nonce":2893994080},{"type":"v1","addr":"172.21.15.98:6829","nonce":2893994080}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6832","nonce":2893994080},{"type":"v1","addr":"172.21.15.98:6833","nonce":2893994080}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6830","nonce":2893994080},{"type":"v1","addr":"172.21.15.98:6831","nonce":2893994080}]},"public_addr":"172.21.15.98:6827/2893994080","cluster_addr":"172.21.15.98:6829/2893994080","heartbeat_back_addr":"172.21.15.98:6833/2893994080","heartbeat_front_addr":"172.21.15.98:6831/2893994080","state":["exists","up"]},{"osd":10,"uuid":"326f0209-ba9d-4b59-97cc-d61a758a8f29","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":70,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.101:6824","nonce":3545244225},{"type":"v1","addr":"172.21.15.101:6825","nonce":3545244225}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.101:6826","nonce":3545244225},{"type":"v1","addr":"172.21.15.101:6827","nonce":3545244225}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.101:6830","nonce":3545244225},{"type":"v1","addr":"172.21.15.101:6831","nonce":3545244225}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.101:6828","nonce":3545244225},{"type":"v1","addr":"172.21.15.101:6829","nonce":3545244225}]},"public_addr":"172.21.15.101:6825/3545244225","cluster_addr":"172.21.15.101:6827/3545244225","heartbeat_back_addr":"172.21.15.101:6831/3545244225","heartbeat_front_addr":"172.21.15.101:6829/3545244225","state":["exists","up"]},{"osd":11,"uuid":"c61c0320-1057-4f17-9bf1-7ad306ed7ed3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":78,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":2630023454},{"type":"v1","addr":"172.21.15.139:6825","nonce":2630023454}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6826","nonce":2630023454},{"type":"v1","addr":"172.21.15.139:6827","nonce":2630023454}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":2630023454},{"type":"v1","addr":"172.21.15.139:6831","nonce":2630023454}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":2630023454},{"type":"v1","addr":"172.21.15.139:6829","nonce":2630023454}]},"public_addr":"172.21.15.139:6825/2630023454","cluster_addr":"172.21.15.139:6827/2630023454","heartbeat_back_addr":"172.21.15.139:6831/2630023454","heartbeat_front_addr":"172.21.15.139:6829/2630023454","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T19:24:23.620028+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T19:24:45.779846+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T19:25:06.877949+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T19:25:27.994199+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T19:25:49.461139+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T19:26:10.409282+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T19:26:32.116390+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T19:26:53.536474+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T19:27:14.384839+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T19:27:35.635921+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T19:27:56.858744+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T19:28:18.342519+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.98:0/2164590253":"2023-07-15T19:23:01.520302+0000","172.21.15.98:0/2094734896":"2023-07-15T19:23:01.520302+0000","172.21.15.98:0/784745009":"2023-07-15T19:22:39.481502+0000","172.21.15.98:0/3657190818":"2023-07-15T19:22:39.481502+0000","172.21.15.98:6800/1732769087":"2023-07-15T19:23:01.520302+0000","172.21.15.98:0/3393542419":"2023-07-15T19:22:39.481502+0000","172.21.15.98:6800/749234665":"2023-07-15T19:22:39.481502+0000","172.21.15.98:6801/1732769087":"2023-07-15T19:23:01.520302+0000","172.21.15.98:6801/749234665":"2023-07-15T19:22:39.481502+0000","172.21.15.98:0/187822873":"2023-07-15T19:23:01.520302+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-07-14T19:29:01.988 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:01.988+0000 7f36b680f700 1 -- 172.21.15.98:0/1659408846 >> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] conn(0x7f36980a11a0 msgr2=0x7f36980a3650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:29:01.988 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:01.988+0000 7f36b680f700 1 --2- 172.21.15.98:0/1659408846 >> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] conn(0x7f36980a11a0 0x7f36980a3650 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f36a4011550 tx=0x7f36a4009260 comp rx=0 tx=0).stop 2023-07-14T19:29:01.989 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:01.988+0000 7f36b680f700 1 -- 172.21.15.98:0/1659408846 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f36b00ffc30 msgr2=0x7f36b00743f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:29:01.989 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:01.988+0000 7f36b680f700 1 --2- 172.21.15.98:0/1659408846 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f36b00ffc30 0x7f36b00743f0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f36a000e9c0 tx=0x7f36a000ecd0 comp rx=0 tx=0).stop 2023-07-14T19:29:01.989 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:01.988+0000 7f36b680f700 1 -- 172.21.15.98:0/1659408846 shutdown_connections 2023-07-14T19:29:01.989 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:01.988+0000 7f36b680f700 1 --2- 172.21.15.98:0/1659408846 >> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] conn(0x7f36980a11a0 0x7f36980a3650 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:29:01.989 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:01.988+0000 7f36b680f700 1 --2- 172.21.15.98:0/1659408846 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f36b01017b0 0x7f36b0072f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:29:01.989 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:01.988+0000 7f36b680f700 1 --2- 172.21.15.98:0/1659408846 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f36b00ffc30 0x7f36b00743f0 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:29:01.990 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:01.988+0000 7f36b680f700 1 --2- 172.21.15.98:0/1659408846 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f36b0100e20 0x7f36b0072a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:29:01.990 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:01.988+0000 7f36b680f700 1 -- 172.21.15.98:0/1659408846 >> 172.21.15.98:0/1659408846 conn(0x7f36b00fb230 msgr2=0x7f36b0104230 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:29:01.990 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:01.988+0000 7f36b680f700 1 -- 172.21.15.98:0/1659408846 shutdown_connections 2023-07-14T19:29:01.990 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:01.988+0000 7f36b680f700 1 -- 172.21.15.98:0/1659408846 wait complete. 2023-07-14T19:29:01.999 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:29:01 smithi098 ceph-mon[127486]: from='client.? 172.21.15.98:0/3980117586' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]': finished 2023-07-14T19:29:01.999 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:29:01 smithi098 ceph-mon[127486]: osdmap e81: 12 total, 12 up, 12 in 2023-07-14T19:29:02.000 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:29:01 smithi098 ceph-mon[127486]: from='client.? 172.21.15.98:0/30005641' entity='client.admin' cmd=[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]: dispatch 2023-07-14T19:29:02.000 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:29:01 smithi098 ceph-mon[127486]: daemon mds.b assigned to filesystem cephfs as rank 0 2023-07-14T19:29:02.000 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:29:01 smithi098 ceph-mon[127486]: osdmap e82: 12 total, 12 up, 12 in 2023-07-14T19:29:02.001 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-07-14T19:29:02.125 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:02.122+0000 7fac47f77700 1 -- 172.21.15.98:0/3170899197 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fac400ffe00 msgr2=0x7fac401066c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:29:02.126 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:02.122+0000 7fac47f77700 1 --2- 172.21.15.98:0/3170899197 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fac400ffe00 0x7fac401066c0 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7fac3800b0a0 tx=0x7fac3800b3b0 comp rx=0 tx=0).stop 2023-07-14T19:29:02.126 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:02.122+0000 7fac47f77700 1 -- 172.21.15.98:0/3170899197 shutdown_connections 2023-07-14T19:29:02.126 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:02.122+0000 7fac47f77700 1 --2- 172.21.15.98:0/3170899197 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fac400ffe00 0x7fac401066c0 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:29:02.126 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:02.122+0000 7fac47f77700 1 --2- 172.21.15.98:0/3170899197 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7fac400fe280 0x7fac400fe690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:29:02.127 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:02.122+0000 7fac47f77700 1 --2- 172.21.15.98:0/3170899197 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fac400ff470 0x7fac400ff8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:29:02.127 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:02.122+0000 7fac47f77700 1 -- 172.21.15.98:0/3170899197 >> 172.21.15.98:0/3170899197 conn(0x7fac400f98a0 msgr2=0x7fac400fbcd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:29:02.127 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:02.122+0000 7fac47f77700 1 -- 172.21.15.98:0/3170899197 shutdown_connections 2023-07-14T19:29:02.127 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:02.122+0000 7fac47f77700 1 -- 172.21.15.98:0/3170899197 wait complete. 2023-07-14T19:29:02.128 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:02.122+0000 7fac47f77700 1 Processor -- start 2023-07-14T19:29:02.128 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:02.122+0000 7fac47f77700 1 -- start start 2023-07-14T19:29:02.128 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:02.123+0000 7fac47f77700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fac400fe280 0x7fac40107b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:29:02.129 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:02.123+0000 7fac47f77700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fac400ff470 0x7fac4019a6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:29:02.129 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:02.123+0000 7fac47f77700 1 --2- >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7fac400ffe00 0x7fac4019cd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:29:02.129 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:02.123+0000 7fac47f77700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fac4019d4f0 con 0x7fac400fe280 2023-07-14T19:29:02.129 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:02.123+0000 7fac47f77700 1 -- --> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fac4019d660 con 0x7fac400ffe00 2023-07-14T19:29:02.129 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:02.123+0000 7fac47f77700 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fac4019d820 con 0x7fac400ff470 2023-07-14T19:29:02.130 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:02.123+0000 7fac45d13700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fac400fe280 0x7fac40107b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:29:02.130 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:02.123+0000 7fac45d13700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fac400fe280 0x7fac40107b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:45678/0 (socket says 172.21.15.98:45678) 2023-07-14T19:29:02.130 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:02.123+0000 7fac45d13700 1 -- 172.21.15.98:0/231756188 learned_addr learned my addr 172.21.15.98:0/231756188 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-07-14T19:29:02.130 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:02.123+0000 7fac46514700 1 --2- 172.21.15.98:0/231756188 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7fac400ffe00 0x7fac4019cd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:29:02.131 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:02.123+0000 7fac45512700 1 --2- 172.21.15.98:0/231756188 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fac400ff470 0x7fac4019a6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:29:02.131 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:02.123+0000 7fac45d13700 1 -- 172.21.15.98:0/231756188 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fac400ff470 msgr2=0x7fac4019a6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:29:02.131 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:02.123+0000 7fac45d13700 1 --2- 172.21.15.98:0/231756188 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fac400ff470 0x7fac4019a6d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:29:02.131 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:02.123+0000 7fac45d13700 1 -- 172.21.15.98:0/231756188 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7fac400ffe00 msgr2=0x7fac4019cd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:29:02.132 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:02.123+0000 7fac45d13700 1 --2- 172.21.15.98:0/231756188 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7fac400ffe00 0x7fac4019cd20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:29:02.132 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:02.123+0000 7fac45d13700 1 -- 172.21.15.98:0/231756188 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fac38009d00 con 0x7fac400fe280 2023-07-14T19:29:02.132 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:02.123+0000 7fac45512700 1 --2- 172.21.15.98:0/231756188 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fac400ff470 0x7fac4019a6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-07-14T19:29:02.132 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:02.123+0000 7fac45d13700 1 --2- 172.21.15.98:0/231756188 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fac400fe280 0x7fac40107b80 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7fac3000d980 tx=0x7fac3000dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:29:02.133 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:02.123+0000 7fac36ffd700 1 -- 172.21.15.98:0/231756188 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fac30004020 con 0x7fac400fe280 2023-07-14T19:29:02.133 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:02.123+0000 7fac47f77700 1 -- 172.21.15.98:0/231756188 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fac4019db00 con 0x7fac400fe280 2023-07-14T19:29:02.133 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:02.123+0000 7fac36ffd700 1 -- 172.21.15.98:0/231756188 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fac30009730 con 0x7fac400fe280 2023-07-14T19:29:02.133 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:02.123+0000 7fac36ffd700 1 -- 172.21.15.98:0/231756188 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fac30003a80 con 0x7fac400fe280 2023-07-14T19:29:02.134 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:02.124+0000 7fac47f77700 1 -- 172.21.15.98:0/231756188 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fac4019e060 con 0x7fac400fe280 2023-07-14T19:29:02.134 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:02.124+0000 7fac47f77700 1 -- 172.21.15.98:0/231756188 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fac401056d0 con 0x7fac400fe280 2023-07-14T19:29:02.135 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:02.126+0000 7fac36ffd700 1 -- 172.21.15.98:0/231756188 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 16) v1 ==== 135715+0+0 (secure 0 0 0) 0x7fac300098a0 con 0x7fac400fe280 2023-07-14T19:29:02.136 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:02.126+0000 7fac36ffd700 1 --2- 172.21.15.98:0/231756188 >> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] conn(0x7fac2c0a1220 0x7fac2c0a36d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:29:02.136 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:02.126+0000 7fac45512700 1 --2- 172.21.15.98:0/231756188 >> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] conn(0x7fac2c0a1220 0x7fac2c0a36d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:29:02.136 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:02.126+0000 7fac36ffd700 1 -- 172.21.15.98:0/231756188 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 7979+0+0 (secure 0 0 0) 0x7fac300cc780 con 0x7fac400fe280 2023-07-14T19:29:02.136 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:02.127+0000 7fac45512700 1 --2- 172.21.15.98:0/231756188 >> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] conn(0x7fac2c0a1220 0x7fac2c0a36d0 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7fac3c005fd0 tx=0x7fac3c005ee0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:29:02.137 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:02.130+0000 7fac36ffd700 1 -- 172.21.15.98:0/231756188 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7fac300960a0 con 0x7fac400fe280 2023-07-14T19:29:02.323 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:02.323+0000 7fac47f77700 1 -- 172.21.15.98:0/231756188 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fac4002cb30 con 0x7fac400fe280 2023-07-14T19:29:02.324 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:02.324+0000 7fac36ffd700 1 -- 172.21.15.98:0/231756188 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v83) v1 ==== 74+0+22117 (secure 0 0 0) 0x7fac300996c0 con 0x7fac400fe280 2023-07-14T19:29:02.325 INFO:teuthology.orchestra.run.smithi098.stdout: 2023-07-14T19:29:02.325 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":83,"fsid":"9adea648-227b-11ee-9b33-001a4aab830c","created":"2023-07-14T19:22:07.178854+0000","modified":"2023-07-14T19:29:02.293099+0000","last_up_change":"2023-07-14T19:28:19.646827+0000","last_in_change":"2023-07-14T19:28:02.018325+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-07-14T19:25:09.845827+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2023-07-14T19:28:58.952921+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"82","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2023-07-14T19:29:00.228272+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"83","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"a3797e89-1754-41cf-9517-227e06ffdd72","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6802","nonce":120507228},{"type":"v1","addr":"172.21.15.98:6803","nonce":120507228}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6804","nonce":120507228},{"type":"v1","addr":"172.21.15.98:6805","nonce":120507228}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6808","nonce":120507228},{"type":"v1","addr":"172.21.15.98:6809","nonce":120507228}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6806","nonce":120507228},{"type":"v1","addr":"172.21.15.98:6807","nonce":120507228}]},"public_addr":"172.21.15.98:6803/120507228","cluster_addr":"172.21.15.98:6805/120507228","heartbeat_back_addr":"172.21.15.98:6809/120507228","heartbeat_front_addr":"172.21.15.98:6807/120507228","state":["exists","up"]},{"osd":1,"uuid":"2b001add-1aaa-4496-a403-7ef9725b80a0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.101:6800","nonce":2450584584},{"type":"v1","addr":"172.21.15.101:6801","nonce":2450584584}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.101:6802","nonce":2450584584},{"type":"v1","addr":"172.21.15.101:6803","nonce":2450584584}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.101:6806","nonce":2450584584},{"type":"v1","addr":"172.21.15.101:6807","nonce":2450584584}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.101:6804","nonce":2450584584},{"type":"v1","addr":"172.21.15.101:6805","nonce":2450584584}]},"public_addr":"172.21.15.101:6801/2450584584","cluster_addr":"172.21.15.101:6803/2450584584","heartbeat_back_addr":"172.21.15.101:6807/2450584584","heartbeat_front_addr":"172.21.15.101:6805/2450584584","state":["exists","up"]},{"osd":2,"uuid":"48e6d499-3ca9-424b-b0fb-3ba5f33b9fc9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":945038995},{"type":"v1","addr":"172.21.15.139:6801","nonce":945038995}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":945038995},{"type":"v1","addr":"172.21.15.139:6803","nonce":945038995}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":945038995},{"type":"v1","addr":"172.21.15.139:6807","nonce":945038995}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":945038995},{"type":"v1","addr":"172.21.15.139:6805","nonce":945038995}]},"public_addr":"172.21.15.139:6801/945038995","cluster_addr":"172.21.15.139:6803/945038995","heartbeat_back_addr":"172.21.15.139:6807/945038995","heartbeat_front_addr":"172.21.15.139:6805/945038995","state":["exists","up"]},{"osd":3,"uuid":"5530b265-31a5-447b-a1ea-17ebae9cfddf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6810","nonce":3095249665},{"type":"v1","addr":"172.21.15.98:6811","nonce":3095249665}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6812","nonce":3095249665},{"type":"v1","addr":"172.21.15.98:6813","nonce":3095249665}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6816","nonce":3095249665},{"type":"v1","addr":"172.21.15.98:6817","nonce":3095249665}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6814","nonce":3095249665},{"type":"v1","addr":"172.21.15.98:6815","nonce":3095249665}]},"public_addr":"172.21.15.98:6811/3095249665","cluster_addr":"172.21.15.98:6813/3095249665","heartbeat_back_addr":"172.21.15.98:6817/3095249665","heartbeat_front_addr":"172.21.15.98:6815/3095249665","state":["exists","up"]},{"osd":4,"uuid":"615d4642-a997-4f95-9e38-9fbdaa831400","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":32,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.101:6808","nonce":2844761406},{"type":"v1","addr":"172.21.15.101:6809","nonce":2844761406}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.101:6810","nonce":2844761406},{"type":"v1","addr":"172.21.15.101:6811","nonce":2844761406}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.101:6814","nonce":2844761406},{"type":"v1","addr":"172.21.15.101:6815","nonce":2844761406}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.101:6812","nonce":2844761406},{"type":"v1","addr":"172.21.15.101:6813","nonce":2844761406}]},"public_addr":"172.21.15.101:6809/2844761406","cluster_addr":"172.21.15.101:6811/2844761406","heartbeat_back_addr":"172.21.15.101:6815/2844761406","heartbeat_front_addr":"172.21.15.101:6813/2844761406","state":["exists","up"]},{"osd":5,"uuid":"415de5fa-607a-45a8-b732-28daf3be6dd3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":37,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":3532263042},{"type":"v1","addr":"172.21.15.139:6809","nonce":3532263042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6810","nonce":3532263042},{"type":"v1","addr":"172.21.15.139:6811","nonce":3532263042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6814","nonce":3532263042},{"type":"v1","addr":"172.21.15.139:6815","nonce":3532263042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":3532263042},{"type":"v1","addr":"172.21.15.139:6813","nonce":3532263042}]},"public_addr":"172.21.15.139:6809/3532263042","cluster_addr":"172.21.15.139:6811/3532263042","heartbeat_back_addr":"172.21.15.139:6815/3532263042","heartbeat_front_addr":"172.21.15.139:6813/3532263042","state":["exists","up"]},{"osd":6,"uuid":"d7f9d4ba-c309-4b75-a836-5a36384b528e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":45,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6818","nonce":658948743},{"type":"v1","addr":"172.21.15.98:6819","nonce":658948743}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6820","nonce":658948743},{"type":"v1","addr":"172.21.15.98:6821","nonce":658948743}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6824","nonce":658948743},{"type":"v1","addr":"172.21.15.98:6825","nonce":658948743}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6822","nonce":658948743},{"type":"v1","addr":"172.21.15.98:6823","nonce":658948743}]},"public_addr":"172.21.15.98:6819/658948743","cluster_addr":"172.21.15.98:6821/658948743","heartbeat_back_addr":"172.21.15.98:6825/658948743","heartbeat_front_addr":"172.21.15.98:6823/658948743","state":["exists","up"]},{"osd":7,"uuid":"eedb94d9-c5cd-4a93-8a8c-2648804b7195","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":50,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.101:6816","nonce":2881214066},{"type":"v1","addr":"172.21.15.101:6817","nonce":2881214066}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.101:6818","nonce":2881214066},{"type":"v1","addr":"172.21.15.101:6819","nonce":2881214066}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.101:6822","nonce":2881214066},{"type":"v1","addr":"172.21.15.101:6823","nonce":2881214066}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.101:6820","nonce":2881214066},{"type":"v1","addr":"172.21.15.101:6821","nonce":2881214066}]},"public_addr":"172.21.15.101:6817/2881214066","cluster_addr":"172.21.15.101:6819/2881214066","heartbeat_back_addr":"172.21.15.101:6823/2881214066","heartbeat_front_addr":"172.21.15.101:6821/2881214066","state":["exists","up"]},{"osd":8,"uuid":"6bfd77c2-5770-466e-aa6f-84b50c74ffb6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":57,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":3669514524},{"type":"v1","addr":"172.21.15.139:6817","nonce":3669514524}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":3669514524},{"type":"v1","addr":"172.21.15.139:6819","nonce":3669514524}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":3669514524},{"type":"v1","addr":"172.21.15.139:6823","nonce":3669514524}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6820","nonce":3669514524},{"type":"v1","addr":"172.21.15.139:6821","nonce":3669514524}]},"public_addr":"172.21.15.139:6817/3669514524","cluster_addr":"172.21.15.139:6819/3669514524","heartbeat_back_addr":"172.21.15.139:6823/3669514524","heartbeat_front_addr":"172.21.15.139:6821/3669514524","state":["exists","up"]},{"osd":9,"uuid":"19c5b54a-2242-4b23-b721-63be9559653d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":65,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6826","nonce":2893994080},{"type":"v1","addr":"172.21.15.98:6827","nonce":2893994080}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6828","nonce":2893994080},{"type":"v1","addr":"172.21.15.98:6829","nonce":2893994080}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6832","nonce":2893994080},{"type":"v1","addr":"172.21.15.98:6833","nonce":2893994080}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6830","nonce":2893994080},{"type":"v1","addr":"172.21.15.98:6831","nonce":2893994080}]},"public_addr":"172.21.15.98:6827/2893994080","cluster_addr":"172.21.15.98:6829/2893994080","heartbeat_back_addr":"172.21.15.98:6833/2893994080","heartbeat_front_addr":"172.21.15.98:6831/2893994080","state":["exists","up"]},{"osd":10,"uuid":"326f0209-ba9d-4b59-97cc-d61a758a8f29","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":70,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.101:6824","nonce":3545244225},{"type":"v1","addr":"172.21.15.101:6825","nonce":3545244225}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.101:6826","nonce":3545244225},{"type":"v1","addr":"172.21.15.101:6827","nonce":3545244225}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.101:6830","nonce":3545244225},{"type":"v1","addr":"172.21.15.101:6831","nonce":3545244225}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.101:6828","nonce":3545244225},{"type":"v1","addr":"172.21.15.101:6829","nonce":3545244225}]},"public_addr":"172.21.15.101:6825/3545244225","cluster_addr":"172.21.15.101:6827/3545244225","heartbeat_back_addr":"172.21.15.101:6831/3545244225","heartbeat_front_addr":"172.21.15.101:6829/3545244225","state":["exists","up"]},{"osd":11,"uuid":"c61c0320-1057-4f17-9bf1-7ad306ed7ed3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":78,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":2630023454},{"type":"v1","addr":"172.21.15.139:6825","nonce":2630023454}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6826","nonce":2630023454},{"type":"v1","addr":"172.21.15.139:6827","nonce":2630023454}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":2630023454},{"type":"v1","addr":"172.21.15.139:6831","nonce":2630023454}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":2630023454},{"type":"v1","addr":"172.21.15.139:6829","nonce":2630023454}]},"public_addr":"172.21.15.139:6825/2630023454","cluster_addr":"172.21.15.139:6827/2630023454","heartbeat_back_addr":"172.21.15.139:6831/2630023454","heartbeat_front_addr":"172.21.15.139:6829/2630023454","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T19:24:23.620028+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T19:24:45.779846+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T19:25:06.877949+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T19:25:27.994199+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T19:25:49.461139+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T19:26:10.409282+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T19:26:32.116390+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T19:26:53.536474+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T19:27:14.384839+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T19:27:35.635921+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T19:27:56.858744+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T19:28:18.342519+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.98:0/2164590253":"2023-07-15T19:23:01.520302+0000","172.21.15.98:0/2094734896":"2023-07-15T19:23:01.520302+0000","172.21.15.98:0/784745009":"2023-07-15T19:22:39.481502+0000","172.21.15.98:0/3657190818":"2023-07-15T19:22:39.481502+0000","172.21.15.98:6800/1732769087":"2023-07-15T19:23:01.520302+0000","172.21.15.98:0/3393542419":"2023-07-15T19:22:39.481502+0000","172.21.15.98:6800/749234665":"2023-07-15T19:22:39.481502+0000","172.21.15.98:6801/1732769087":"2023-07-15T19:23:01.520302+0000","172.21.15.98:6801/749234665":"2023-07-15T19:22:39.481502+0000","172.21.15.98:0/187822873":"2023-07-15T19:23:01.520302+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-07-14T19:29:02.327 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:02.326+0000 7fac47f77700 1 -- 172.21.15.98:0/231756188 >> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] conn(0x7fac2c0a1220 msgr2=0x7fac2c0a36d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:29:02.327 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:02.326+0000 7fac47f77700 1 --2- 172.21.15.98:0/231756188 >> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] conn(0x7fac2c0a1220 0x7fac2c0a36d0 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7fac3c005fd0 tx=0x7fac3c005ee0 comp rx=0 tx=0).stop 2023-07-14T19:29:02.327 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:02.326+0000 7fac47f77700 1 -- 172.21.15.98:0/231756188 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fac400fe280 msgr2=0x7fac40107b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:29:02.327 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:02.326+0000 7fac47f77700 1 --2- 172.21.15.98:0/231756188 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fac400fe280 0x7fac40107b80 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7fac3000d980 tx=0x7fac3000dc90 comp rx=0 tx=0).stop 2023-07-14T19:29:02.327 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:02.326+0000 7fac47f77700 1 -- 172.21.15.98:0/231756188 shutdown_connections 2023-07-14T19:29:02.327 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:02.326+0000 7fac47f77700 1 --2- 172.21.15.98:0/231756188 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7fac400ffe00 0x7fac4019cd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:29:02.328 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:02.326+0000 7fac47f77700 1 --2- 172.21.15.98:0/231756188 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fac400fe280 0x7fac40107b80 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:29:02.328 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:02.326+0000 7fac47f77700 1 --2- 172.21.15.98:0/231756188 >> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] conn(0x7fac2c0a1220 0x7fac2c0a36d0 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:29:02.328 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:02.326+0000 7fac47f77700 1 --2- 172.21.15.98:0/231756188 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fac400ff470 0x7fac4019a6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:29:02.328 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:02.326+0000 7fac47f77700 1 -- 172.21.15.98:0/231756188 >> 172.21.15.98:0/231756188 conn(0x7fac400f98a0 msgr2=0x7fac40102810 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:29:02.328 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:02.326+0000 7fac47f77700 1 -- 172.21.15.98:0/231756188 shutdown_connections 2023-07-14T19:29:02.328 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:02.326+0000 7fac47f77700 1 -- 172.21.15.98:0/231756188 wait complete. 2023-07-14T19:29:02.339 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2023-07-14T19:29:02.464 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:02.464+0000 7fb81fe4d700 1 -- 172.21.15.98:0/2749739731 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb8180ffd30 msgr2=0x7fb8181001a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:29:02.465 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:02.464+0000 7fb81fe4d700 1 --2- 172.21.15.98:0/2749739731 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb8180ffd30 0x7fb8181001a0 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7fb808009990 tx=0x7fb808009ca0 comp rx=0 tx=0).stop 2023-07-14T19:29:02.465 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:02.464+0000 7fb81fe4d700 1 -- 172.21.15.98:0/2749739731 shutdown_connections 2023-07-14T19:29:02.465 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:02.464+0000 7fb81fe4d700 1 --2- 172.21.15.98:0/2749739731 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fb8181006e0 0x7fb818106f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:29:02.465 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:02.464+0000 7fb81fe4d700 1 --2- 172.21.15.98:0/2749739731 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7fb8180ff020 0x7fb8180ff430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:29:02.465 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:02.464+0000 7fb81fe4d700 1 --2- 172.21.15.98:0/2749739731 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb8180ffd30 0x7fb8181001a0 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:29:02.465 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:02.464+0000 7fb81fe4d700 1 -- 172.21.15.98:0/2749739731 >> 172.21.15.98:0/2749739731 conn(0x7fb8180f98a0 msgr2=0x7fb8180fbcd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:29:02.466 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:02.464+0000 7fb81fe4d700 1 -- 172.21.15.98:0/2749739731 shutdown_connections 2023-07-14T19:29:02.466 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:02.464+0000 7fb81fe4d700 1 -- 172.21.15.98:0/2749739731 wait complete. 2023-07-14T19:29:02.467 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:02.464+0000 7fb81fe4d700 1 Processor -- start 2023-07-14T19:29:02.468 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:02.464+0000 7fb81fe4d700 1 -- start start 2023-07-14T19:29:02.468 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:02.464+0000 7fb81fe4d700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb8180ff020 0x7fb8181948d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:29:02.468 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:02.464+0000 7fb81fe4d700 1 --2- >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7fb8180ffd30 0x7fb818194e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:29:02.468 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:02.464+0000 7fb81fe4d700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fb8181006e0 0x7fb81819be80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:29:02.468 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:02.465+0000 7fb81fe4d700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb81819c450 con 0x7fb8180ff020 2023-07-14T19:29:02.469 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:02.465+0000 7fb81fe4d700 1 -- --> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb81819c590 con 0x7fb8180ffd30 2023-07-14T19:29:02.469 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:02.465+0000 7fb81fe4d700 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb81819c720 con 0x7fb8181006e0 2023-07-14T19:29:02.469 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:02.465+0000 7fb81dbe9700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb8180ff020 0x7fb8181948d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:29:02.469 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:02.465+0000 7fb81dbe9700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb8180ff020 0x7fb8181948d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:45692/0 (socket says 172.21.15.98:45692) 2023-07-14T19:29:02.469 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:02.465+0000 7fb81dbe9700 1 -- 172.21.15.98:0/4044846340 learned_addr learned my addr 172.21.15.98:0/4044846340 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-07-14T19:29:02.469 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:02.465+0000 7fb81d3e8700 1 --2- 172.21.15.98:0/4044846340 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7fb8180ffd30 0x7fb818194e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:29:02.469 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:02.465+0000 7fb81e3ea700 1 --2- 172.21.15.98:0/4044846340 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fb8181006e0 0x7fb81819be80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:29:02.469 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:02.465+0000 7fb81dbe9700 1 -- 172.21.15.98:0/4044846340 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fb8181006e0 msgr2=0x7fb81819be80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:29:02.470 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:02.465+0000 7fb81dbe9700 1 --2- 172.21.15.98:0/4044846340 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fb8181006e0 0x7fb81819be80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:29:02.470 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:02.465+0000 7fb81dbe9700 1 -- 172.21.15.98:0/4044846340 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7fb8180ffd30 msgr2=0x7fb818194e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:29:02.470 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:02.465+0000 7fb81dbe9700 1 --2- 172.21.15.98:0/4044846340 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7fb8180ffd30 0x7fb818194e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:29:02.470 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:02.465+0000 7fb81dbe9700 1 -- 172.21.15.98:0/4044846340 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb8080095f0 con 0x7fb8180ff020 2023-07-14T19:29:02.470 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:02.465+0000 7fb81e3ea700 1 --2- 172.21.15.98:0/4044846340 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fb8181006e0 0x7fb81819be80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-07-14T19:29:02.470 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:02.465+0000 7fb81dbe9700 1 --2- 172.21.15.98:0/4044846340 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb8180ff020 0x7fb8181948d0 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7fb81400e970 tx=0x7fb81400ec80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:29:02.470 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:02.465+0000 7fb80effd700 1 -- 172.21.15.98:0/4044846340 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb814009790 con 0x7fb8180ff020 2023-07-14T19:29:02.471 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:02.465+0000 7fb81fe4d700 1 -- 172.21.15.98:0/4044846340 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb81819ca00 con 0x7fb8180ff020 2023-07-14T19:29:02.471 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:02.465+0000 7fb80effd700 1 -- 172.21.15.98:0/4044846340 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fb81400eec0 con 0x7fb8180ff020 2023-07-14T19:29:02.471 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:02.465+0000 7fb80effd700 1 -- 172.21.15.98:0/4044846340 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb814003cf0 con 0x7fb8180ff020 2023-07-14T19:29:02.471 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:02.465+0000 7fb81fe4d700 1 -- 172.21.15.98:0/4044846340 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb81819cf10 con 0x7fb8180ff020 2023-07-14T19:29:02.471 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:02.466+0000 7fb81fe4d700 1 -- 172.21.15.98:0/4044846340 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb81810a2e0 con 0x7fb8180ff020 2023-07-14T19:29:02.471 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:02.467+0000 7fb80effd700 1 -- 172.21.15.98:0/4044846340 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 16) v1 ==== 135715+0+0 (secure 0 0 0) 0x7fb81400ccd0 con 0x7fb8180ff020 2023-07-14T19:29:02.472 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:02.468+0000 7fb80effd700 1 --2- 172.21.15.98:0/4044846340 >> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] conn(0x7fb8040a1150 0x7fb8040a3600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:29:02.472 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:02.468+0000 7fb81d3e8700 1 --2- 172.21.15.98:0/4044846340 >> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] conn(0x7fb8040a1150 0x7fb8040a3600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:29:02.472 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:02.468+0000 7fb80effd700 1 -- 172.21.15.98:0/4044846340 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 7979+0+0 (secure 0 0 0) 0x7fb814014070 con 0x7fb8180ff020 2023-07-14T19:29:02.472 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:02.468+0000 7fb81d3e8700 1 --2- 172.21.15.98:0/4044846340 >> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] conn(0x7fb8040a1150 0x7fb8040a3600 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7fb80800b2e0 tx=0x7fb808005fb0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:29:02.473 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:02.472+0000 7fb80effd700 1 -- 172.21.15.98:0/4044846340 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7fb8140962b0 con 0x7fb8180ff020 2023-07-14T19:29:02.711 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:02.711+0000 7fb81fe4d700 1 -- 172.21.15.98:0/4044846340 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) v1 -- 0x7fb818105af0 con 0x7fb8180ff020 2023-07-14T19:29:02.941 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:29:02 smithi101 ceph-mon[128807]: pgmap v252: 97 pgs: 24 creating+peering, 63 unknown, 10 active+clean; 577 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-14T19:29:02.941 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:29:02 smithi101 ceph-mon[128807]: from='client.? 172.21.15.98:0/30005641' entity='client.admin' cmd='[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]': finished 2023-07-14T19:29:02.942 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:29:02 smithi101 ceph-mon[128807]: fsmap cephfs:1 {0=b=up:creating} 11 up:standby 2023-07-14T19:29:02.942 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:29:02 smithi101 ceph-mon[128807]: daemon mds.b is now active in filesystem cephfs as rank 0 2023-07-14T19:29:02.942 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:29:02 smithi101 ceph-mon[128807]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' 2023-07-14T19:29:02.942 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:29:02 smithi101 ceph-mon[128807]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' 2023-07-14T19:29:02.942 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:29:02 smithi101 ceph-mon[128807]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' 2023-07-14T19:29:02.942 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:29:02 smithi101 ceph-mon[128807]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' 2023-07-14T19:29:02.942 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:29:02 smithi101 ceph-mon[128807]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' 2023-07-14T19:29:02.943 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:29:02 smithi101 ceph-mon[128807]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' 2023-07-14T19:29:02.943 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:29:02 smithi101 ceph-mon[128807]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T19:29:02.943 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:29:02 smithi101 ceph-mon[128807]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-14T19:29:02.943 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:29:02 smithi101 ceph-mon[128807]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' 2023-07-14T19:29:02.943 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:29:02 smithi101 ceph-mon[128807]: from='client.? 172.21.15.98:0/1659408846' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-07-14T19:29:02.943 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:29:02 smithi101 ceph-mon[128807]: osdmap e83: 12 total, 12 up, 12 in 2023-07-14T19:29:02.943 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:29:02 smithi101 ceph-mon[128807]: from='client.? 172.21.15.98:0/231756188' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-07-14T19:29:02.970 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:29:02 smithi139 ceph-mon[129293]: pgmap v252: 97 pgs: 24 creating+peering, 63 unknown, 10 active+clean; 577 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-14T19:29:02.971 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:29:02 smithi139 ceph-mon[129293]: from='client.? 172.21.15.98:0/30005641' entity='client.admin' cmd='[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]': finished 2023-07-14T19:29:02.971 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:29:02 smithi139 ceph-mon[129293]: fsmap cephfs:1 {0=b=up:creating} 11 up:standby 2023-07-14T19:29:02.971 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:29:02 smithi139 ceph-mon[129293]: daemon mds.b is now active in filesystem cephfs as rank 0 2023-07-14T19:29:02.971 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:29:02 smithi139 ceph-mon[129293]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' 2023-07-14T19:29:02.972 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:29:02 smithi139 ceph-mon[129293]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' 2023-07-14T19:29:02.972 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:29:02 smithi139 ceph-mon[129293]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' 2023-07-14T19:29:02.972 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:29:02 smithi139 ceph-mon[129293]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' 2023-07-14T19:29:02.973 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:29:02 smithi139 ceph-mon[129293]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' 2023-07-14T19:29:02.973 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:29:02 smithi139 ceph-mon[129293]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' 2023-07-14T19:29:02.973 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:29:02 smithi139 ceph-mon[129293]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T19:29:02.973 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:29:02 smithi139 ceph-mon[129293]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-14T19:29:02.974 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:29:02 smithi139 ceph-mon[129293]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' 2023-07-14T19:29:02.974 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:29:02 smithi139 ceph-mon[129293]: from='client.? 172.21.15.98:0/1659408846' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-07-14T19:29:02.974 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:29:02 smithi139 ceph-mon[129293]: osdmap e83: 12 total, 12 up, 12 in 2023-07-14T19:29:02.974 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:29:02 smithi139 ceph-mon[129293]: from='client.? 172.21.15.98:0/231756188' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-07-14T19:29:03.031 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:29:02 smithi098 ceph-mon[127486]: pgmap v252: 97 pgs: 24 creating+peering, 63 unknown, 10 active+clean; 577 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-14T19:29:03.031 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:29:02 smithi098 ceph-mon[127486]: from='client.? 172.21.15.98:0/30005641' entity='client.admin' cmd='[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]': finished 2023-07-14T19:29:03.032 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:29:02 smithi098 ceph-mon[127486]: fsmap cephfs:1 {0=b=up:creating} 11 up:standby 2023-07-14T19:29:03.032 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:29:02 smithi098 ceph-mon[127486]: daemon mds.b is now active in filesystem cephfs as rank 0 2023-07-14T19:29:03.032 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:29:02 smithi098 ceph-mon[127486]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' 2023-07-14T19:29:03.032 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:29:02 smithi098 ceph-mon[127486]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' 2023-07-14T19:29:03.033 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:29:02 smithi098 ceph-mon[127486]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' 2023-07-14T19:29:03.033 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:29:02 smithi098 ceph-mon[127486]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' 2023-07-14T19:29:03.033 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:29:02 smithi098 ceph-mon[127486]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' 2023-07-14T19:29:03.033 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:29:02 smithi098 ceph-mon[127486]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' 2023-07-14T19:29:03.034 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:29:02 smithi098 ceph-mon[127486]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T19:29:03.034 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:29:02 smithi098 ceph-mon[127486]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-14T19:29:03.034 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:29:02 smithi098 ceph-mon[127486]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' 2023-07-14T19:29:03.034 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:29:02 smithi098 ceph-mon[127486]: from='client.? 172.21.15.98:0/1659408846' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-07-14T19:29:03.034 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:29:02 smithi098 ceph-mon[127486]: osdmap e83: 12 total, 12 up, 12 in 2023-07-14T19:29:03.034 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:29:02 smithi098 ceph-mon[127486]: from='client.? 172.21.15.98:0/231756188' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-07-14T19:29:03.653 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:03.652+0000 7fb80effd700 1 -- 172.21.15.98:0/4044846340 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v17) v1 ==== 118+0+0 (secure 0 0 0) 0x7fb8140998d0 con 0x7fb8180ff020 2023-07-14T19:29:03.657 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:03.655+0000 7fb81fe4d700 1 -- 172.21.15.98:0/4044846340 >> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] conn(0x7fb8040a1150 msgr2=0x7fb8040a3600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:29:03.657 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:03.655+0000 7fb81fe4d700 1 --2- 172.21.15.98:0/4044846340 >> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] conn(0x7fb8040a1150 0x7fb8040a3600 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7fb80800b2e0 tx=0x7fb808005fb0 comp rx=0 tx=0).stop 2023-07-14T19:29:03.657 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:03.655+0000 7fb81fe4d700 1 -- 172.21.15.98:0/4044846340 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb8180ff020 msgr2=0x7fb8181948d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:29:03.658 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:03.655+0000 7fb81fe4d700 1 --2- 172.21.15.98:0/4044846340 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb8180ff020 0x7fb8181948d0 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7fb81400e970 tx=0x7fb81400ec80 comp rx=0 tx=0).stop 2023-07-14T19:29:03.658 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:03.655+0000 7fb81fe4d700 1 -- 172.21.15.98:0/4044846340 shutdown_connections 2023-07-14T19:29:03.658 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:03.655+0000 7fb81fe4d700 1 --2- 172.21.15.98:0/4044846340 >> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] conn(0x7fb8040a1150 0x7fb8040a3600 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:29:03.658 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:03.655+0000 7fb81fe4d700 1 --2- 172.21.15.98:0/4044846340 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fb8181006e0 0x7fb81819be80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:29:03.658 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:03.655+0000 7fb81fe4d700 1 --2- 172.21.15.98:0/4044846340 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb8180ff020 0x7fb8181948d0 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:29:03.659 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:03.655+0000 7fb81fe4d700 1 --2- 172.21.15.98:0/4044846340 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7fb8180ffd30 0x7fb818194e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:29:03.659 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:03.655+0000 7fb81fe4d700 1 -- 172.21.15.98:0/4044846340 >> 172.21.15.98:0/4044846340 conn(0x7fb8180f98a0 msgr2=0x7fb8180fbb90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:29:03.659 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:03.656+0000 7fb81fe4d700 1 -- 172.21.15.98:0/4044846340 shutdown_connections 2023-07-14T19:29:03.659 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:03.656+0000 7fb81fe4d700 1 -- 172.21.15.98:0/4044846340 wait complete. 2023-07-14T19:29:03.671 DEBUG:tasks.cephfs.filesystem:fs_config: {'max_mds': 3, 'session_timeout': 300, 'standby_replay': True, 'subvols': {'create': 2}} 2023-07-14T19:29:03.672 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs max_mds 3 2023-07-14T19:29:03.806 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:03.801+0000 7fcda6015700 1 -- 172.21.15.98:0/2848770875 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcda00fed40 msgr2=0x7fcda00ff1b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:29:03.807 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:03.801+0000 7fcda6015700 1 --2- 172.21.15.98:0/2848770875 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcda00fed40 0x7fcda00ff1b0 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7fcd90009990 tx=0x7fcd90009ca0 comp rx=0 tx=0).stop 2023-07-14T19:29:03.807 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:03.801+0000 7fcda6015700 1 -- 172.21.15.98:0/2848770875 shutdown_connections 2023-07-14T19:29:03.808 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:03.801+0000 7fcda6015700 1 --2- 172.21.15.98:0/2848770875 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fcda00ff6f0 0x7fcda0106fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:29:03.808 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:03.801+0000 7fcda6015700 1 --2- 172.21.15.98:0/2848770875 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7fcda0100180 0x7fcda0100590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:29:03.808 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:03.801+0000 7fcda6015700 1 --2- 172.21.15.98:0/2848770875 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcda00fed40 0x7fcda00ff1b0 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:29:03.808 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:03.801+0000 7fcda6015700 1 -- 172.21.15.98:0/2848770875 >> 172.21.15.98:0/2848770875 conn(0x7fcda00f98a0 msgr2=0x7fcda00fbcd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:29:03.809 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:03.801+0000 7fcda6015700 1 -- 172.21.15.98:0/2848770875 shutdown_connections 2023-07-14T19:29:03.809 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:03.801+0000 7fcda6015700 1 -- 172.21.15.98:0/2848770875 wait complete. 2023-07-14T19:29:03.809 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:03.802+0000 7fcda6015700 1 Processor -- start 2023-07-14T19:29:03.810 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:03.802+0000 7fcda6015700 1 -- start start 2023-07-14T19:29:03.810 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:03.802+0000 7fcda6015700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fcda00fed40 0x7fcda0102af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:29:03.810 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:03.802+0000 7fcda6015700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcda00ff6f0 0x7fcda0101130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:29:03.810 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:03.802+0000 7fcda6015700 1 --2- >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7fcda0100180 0x7fcda0101670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:29:03.810 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:03.802+0000 7fcda6015700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcda0101e10 con 0x7fcda00ff6f0 2023-07-14T19:29:03.810 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:03.802+0000 7fcda6015700 1 -- --> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcda0101f80 con 0x7fcda0100180 2023-07-14T19:29:03.811 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:03.802+0000 7fcda6015700 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcda019f980 con 0x7fcda00fed40 2023-07-14T19:29:03.811 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:03.802+0000 7fcd9effd700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcda00ff6f0 0x7fcda0101130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:29:03.811 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:03.802+0000 7fcd9effd700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcda00ff6f0 0x7fcda0101130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:45708/0 (socket says 172.21.15.98:45708) 2023-07-14T19:29:03.811 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:03.802+0000 7fcd9effd700 1 -- 172.21.15.98:0/1234726812 learned_addr learned my addr 172.21.15.98:0/1234726812 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-07-14T19:29:03.811 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:03.802+0000 7fcd9ffff700 1 --2- 172.21.15.98:0/1234726812 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7fcda0100180 0x7fcda0101670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:29:03.811 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:03.802+0000 7fcd9f7fe700 1 --2- 172.21.15.98:0/1234726812 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fcda00fed40 0x7fcda0102af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:29:03.811 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:03.802+0000 7fcd9effd700 1 -- 172.21.15.98:0/1234726812 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fcda00fed40 msgr2=0x7fcda0102af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:29:03.812 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:03.802+0000 7fcd9effd700 1 --2- 172.21.15.98:0/1234726812 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fcda00fed40 0x7fcda0102af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:29:03.812 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:03.802+0000 7fcd9effd700 1 -- 172.21.15.98:0/1234726812 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7fcda0100180 msgr2=0x7fcda0101670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:29:03.812 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:03.802+0000 7fcd9effd700 1 --2- 172.21.15.98:0/1234726812 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7fcda0100180 0x7fcda0101670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:29:03.813 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:03.802+0000 7fcd9effd700 1 -- 172.21.15.98:0/1234726812 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcd900095f0 con 0x7fcda00ff6f0 2023-07-14T19:29:03.813 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:03.802+0000 7fcd9ffff700 1 --2- 172.21.15.98:0/1234726812 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7fcda0100180 0x7fcda0101670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-07-14T19:29:03.813 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:03.803+0000 7fcd9effd700 1 --2- 172.21.15.98:0/1234726812 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcda00ff6f0 0x7fcda0101130 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7fcd90000c00 tx=0x7fcd9000f710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:29:03.813 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:03.803+0000 7fcd9cff9700 1 -- 172.21.15.98:0/1234726812 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcd9001d070 con 0x7fcda00ff6f0 2023-07-14T19:29:03.813 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:03.803+0000 7fcd9cff9700 1 -- 172.21.15.98:0/1234726812 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fcd9000fc00 con 0x7fcda00ff6f0 2023-07-14T19:29:03.814 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:03.803+0000 7fcda6015700 1 -- 172.21.15.98:0/1234726812 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcda019fc00 con 0x7fcda00ff6f0 2023-07-14T19:29:03.814 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:03.803+0000 7fcd9cff9700 1 -- 172.21.15.98:0/1234726812 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcd90017650 con 0x7fcda00ff6f0 2023-07-14T19:29:03.814 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:03.803+0000 7fcda6015700 1 -- 172.21.15.98:0/1234726812 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcda01a00b0 con 0x7fcda00ff6f0 2023-07-14T19:29:03.814 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:03.804+0000 7fcda6015700 1 -- 172.21.15.98:0/1234726812 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcda010a2d0 con 0x7fcda00ff6f0 2023-07-14T19:29:03.814 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:03.805+0000 7fcd9cff9700 1 -- 172.21.15.98:0/1234726812 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 16) v1 ==== 135715+0+0 (secure 0 0 0) 0x7fcd90022470 con 0x7fcda00ff6f0 2023-07-14T19:29:03.814 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:03.805+0000 7fcd9cff9700 1 --2- 172.21.15.98:0/1234726812 >> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] conn(0x7fcd880a1150 0x7fcd880a3600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:29:03.815 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:03.805+0000 7fcd9f7fe700 1 --2- 172.21.15.98:0/1234726812 >> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] conn(0x7fcd880a1150 0x7fcd880a3600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:29:03.815 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:03.806+0000 7fcd9cff9700 1 -- 172.21.15.98:0/1234726812 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 7979+0+0 (secure 0 0 0) 0x7fcd900cda00 con 0x7fcda00ff6f0 2023-07-14T19:29:03.815 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:03.806+0000 7fcd9f7fe700 1 --2- 172.21.15.98:0/1234726812 >> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] conn(0x7fcd880a1150 0x7fcd880a3600 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7fcd94005950 tx=0x7fcd940058e0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:29:03.816 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:03.810+0000 7fcd9cff9700 1 -- 172.21.15.98:0/1234726812 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7fcd90097330 con 0x7fcda00ff6f0 2023-07-14T19:29:03.941 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:29:03 smithi101 ceph-mon[128807]: mds.? [v2:172.21.15.101:6838/48393381,v1:172.21.15.101:6839/48393381] up:active 2023-07-14T19:29:03.942 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:29:03 smithi101 ceph-mon[128807]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2023-07-14T19:29:03.942 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:29:03 smithi101 ceph-mon[128807]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2023-07-14T19:29:03.942 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:29:03 smithi101 ceph-mon[128807]: from='client.? 172.21.15.98:0/4044846340' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]: dispatch 2023-07-14T19:29:03.970 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:29:03 smithi139 ceph-mon[129293]: mds.? [v2:172.21.15.101:6838/48393381,v1:172.21.15.101:6839/48393381] up:active 2023-07-14T19:29:03.970 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:29:03 smithi139 ceph-mon[129293]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2023-07-14T19:29:03.970 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:29:03 smithi139 ceph-mon[129293]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2023-07-14T19:29:03.971 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:29:03 smithi139 ceph-mon[129293]: from='client.? 172.21.15.98:0/4044846340' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]: dispatch 2023-07-14T19:29:04.031 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:29:03 smithi098 ceph-mon[127486]: mds.? [v2:172.21.15.101:6838/48393381,v1:172.21.15.101:6839/48393381] up:active 2023-07-14T19:29:04.031 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:29:03 smithi098 ceph-mon[127486]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2023-07-14T19:29:04.031 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:29:03 smithi098 ceph-mon[127486]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2023-07-14T19:29:04.031 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:29:03 smithi098 ceph-mon[127486]: from='client.? 172.21.15.98:0/4044846340' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]: dispatch 2023-07-14T19:29:04.045 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:04.044+0000 7fcda6015700 1 -- 172.21.15.98:0/1234726812 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "3"} v 0) v1 -- 0x7fcda01a0660 con 0x7fcda00ff6f0 2023-07-14T19:29:04.656 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:04.655+0000 7fcd9cff9700 1 -- 172.21.15.98:0/1234726812 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "3"}]=0 v18) v1 ==== 105+0+0 (secure 0 0 0) 0x7fcd9009a950 con 0x7fcda00ff6f0 2023-07-14T19:29:04.660 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:04.658+0000 7fcda6015700 1 -- 172.21.15.98:0/1234726812 >> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] conn(0x7fcd880a1150 msgr2=0x7fcd880a3600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:29:04.660 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:04.658+0000 7fcda6015700 1 --2- 172.21.15.98:0/1234726812 >> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] conn(0x7fcd880a1150 0x7fcd880a3600 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7fcd94005950 tx=0x7fcd940058e0 comp rx=0 tx=0).stop 2023-07-14T19:29:04.661 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:04.658+0000 7fcda6015700 1 -- 172.21.15.98:0/1234726812 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcda00ff6f0 msgr2=0x7fcda0101130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:29:04.661 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:04.658+0000 7fcda6015700 1 --2- 172.21.15.98:0/1234726812 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcda00ff6f0 0x7fcda0101130 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7fcd90000c00 tx=0x7fcd9000f710 comp rx=0 tx=0).stop 2023-07-14T19:29:04.661 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:04.658+0000 7fcda6015700 1 -- 172.21.15.98:0/1234726812 shutdown_connections 2023-07-14T19:29:04.661 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:04.659+0000 7fcda6015700 1 --2- 172.21.15.98:0/1234726812 >> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] conn(0x7fcd880a1150 0x7fcd880a3600 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:29:04.661 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:04.659+0000 7fcda6015700 1 --2- 172.21.15.98:0/1234726812 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fcda00fed40 0x7fcda0102af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:29:04.661 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:04.659+0000 7fcda6015700 1 --2- 172.21.15.98:0/1234726812 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7fcda0100180 0x7fcda0101670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:29:04.661 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:04.659+0000 7fcda6015700 1 --2- 172.21.15.98:0/1234726812 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcda00ff6f0 0x7fcda0101130 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:29:04.662 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:04.659+0000 7fcda6015700 1 -- 172.21.15.98:0/1234726812 >> 172.21.15.98:0/1234726812 conn(0x7fcda00f98a0 msgr2=0x7fcda00fdc30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:29:04.668 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:04.659+0000 7fcda6015700 1 -- 172.21.15.98:0/1234726812 shutdown_connections 2023-07-14T19:29:04.668 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:04.659+0000 7fcda6015700 1 -- 172.21.15.98:0/1234726812 wait complete. 2023-07-14T19:29:04.677 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay true 2023-07-14T19:29:04.811 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:04.807+0000 7f08eaa16700 1 -- 172.21.15.98:0/937170048 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f08e40ffee0 msgr2=0x7f08e41067a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:29:04.811 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:04.807+0000 7f08eaa16700 1 --2- 172.21.15.98:0/937170048 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f08e40ffee0 0x7f08e41067a0 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f08dc00b0a0 tx=0x7f08dc00b3b0 comp rx=0 tx=0).stop 2023-07-14T19:29:04.812 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:04.807+0000 7f08eaa16700 1 -- 172.21.15.98:0/937170048 shutdown_connections 2023-07-14T19:29:04.812 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:04.807+0000 7f08eaa16700 1 --2- 172.21.15.98:0/937170048 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f08e40fe360 0x7f08e40fe770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:29:04.812 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:04.807+0000 7f08eaa16700 1 --2- 172.21.15.98:0/937170048 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f08e40ffee0 0x7f08e41067a0 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:29:04.813 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:04.807+0000 7f08eaa16700 1 --2- 172.21.15.98:0/937170048 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f08e40ff550 0x7f08e40ff9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:29:04.813 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:04.807+0000 7f08eaa16700 1 -- 172.21.15.98:0/937170048 >> 172.21.15.98:0/937170048 conn(0x7f08e40f9980 msgr2=0x7f08e40fbdb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:29:04.814 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:04.807+0000 7f08eaa16700 1 -- 172.21.15.98:0/937170048 shutdown_connections 2023-07-14T19:29:04.814 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:04.807+0000 7f08eaa16700 1 -- 172.21.15.98:0/937170048 wait complete. 2023-07-14T19:29:04.814 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:04.807+0000 7f08eaa16700 1 Processor -- start 2023-07-14T19:29:04.815 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:04.807+0000 7f08eaa16700 1 -- start start 2023-07-14T19:29:04.815 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:04.808+0000 7f08eaa16700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f08e40fe360 0x7f08e40fbc90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:29:04.815 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:04.808+0000 7f08eaa16700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f08e40ff550 0x7f08e419a820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:29:04.815 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:04.808+0000 7f08eaa16700 1 --2- >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f08e40ffee0 0x7f08e419ce60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:29:04.815 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:04.808+0000 7f08eaa16700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f08e419d600 con 0x7f08e40fe360 2023-07-14T19:29:04.815 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:04.808+0000 7f08eaa16700 1 -- --> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f08e419d770 con 0x7f08e40ffee0 2023-07-14T19:29:04.815 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:04.808+0000 7f08eaa16700 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f08e419d930 con 0x7f08e40ff550 2023-07-14T19:29:04.816 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:04.808+0000 7f08e3fff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f08e40fe360 0x7f08e40fbc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:29:04.816 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:04.808+0000 7f08e3fff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f08e40fe360 0x7f08e40fbc90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:45720/0 (socket says 172.21.15.98:45720) 2023-07-14T19:29:04.816 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:04.808+0000 7f08e3fff700 1 -- 172.21.15.98:0/1201225294 learned_addr learned my addr 172.21.15.98:0/1201225294 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-07-14T19:29:04.816 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:04.808+0000 7f08e8fb3700 1 --2- 172.21.15.98:0/1201225294 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f08e40ffee0 0x7f08e419ce60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:29:04.816 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:04.808+0000 7f08e3fff700 1 -- 172.21.15.98:0/1201225294 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f08e40ff550 msgr2=0x7f08e419a820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:29:04.816 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:04.808+0000 7f08e3fff700 1 --2- 172.21.15.98:0/1201225294 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f08e40ff550 0x7f08e419a820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:29:04.816 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:04.808+0000 7f08e3fff700 1 -- 172.21.15.98:0/1201225294 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f08e40ffee0 msgr2=0x7f08e419ce60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:29:04.817 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:04.808+0000 7f08e3fff700 1 --2- 172.21.15.98:0/1201225294 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f08e40ffee0 0x7f08e419ce60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:29:04.817 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:04.808+0000 7f08e3fff700 1 -- 172.21.15.98:0/1201225294 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f08dc009d00 con 0x7f08e40fe360 2023-07-14T19:29:04.817 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:04.808+0000 7f08e8fb3700 1 --2- 172.21.15.98:0/1201225294 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f08e40ffee0 0x7f08e419ce60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-07-14T19:29:04.817 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:04.808+0000 7f08e3fff700 1 --2- 172.21.15.98:0/1201225294 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f08e40fe360 0x7f08e40fbc90 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f08d400d980 tx=0x7f08d400dd40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:29:04.817 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:04.808+0000 7f08e17fa700 1 -- 172.21.15.98:0/1201225294 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f08d400de60 con 0x7f08e40fe360 2023-07-14T19:29:04.817 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:04.808+0000 7f08e17fa700 1 -- 172.21.15.98:0/1201225294 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f08d40044e0 con 0x7f08e40fe360 2023-07-14T19:29:04.817 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:04.808+0000 7f08eaa16700 1 -- 172.21.15.98:0/1201225294 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f08e419dc10 con 0x7f08e40fe360 2023-07-14T19:29:04.818 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:04.808+0000 7f08e17fa700 1 -- 172.21.15.98:0/1201225294 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f08d4003be0 con 0x7f08e40fe360 2023-07-14T19:29:04.818 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:04.808+0000 7f08eaa16700 1 -- 172.21.15.98:0/1201225294 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f08e419e120 con 0x7f08e40fe360 2023-07-14T19:29:04.818 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:04.809+0000 7f08eaa16700 1 -- 172.21.15.98:0/1201225294 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f08e4105ed0 con 0x7f08e40fe360 2023-07-14T19:29:04.818 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:04.810+0000 7f08e17fa700 1 -- 172.21.15.98:0/1201225294 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 16) v1 ==== 135715+0+0 (secure 0 0 0) 0x7f08d4009bb0 con 0x7f08e40fe360 2023-07-14T19:29:04.819 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:04.811+0000 7f08e17fa700 1 --2- 172.21.15.98:0/1201225294 >> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] conn(0x7f08d00a1150 0x7f08d00a3600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:29:04.819 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:04.811+0000 7f08e17fa700 1 -- 172.21.15.98:0/1201225294 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 7979+0+0 (secure 0 0 0) 0x7f08d40cc910 con 0x7f08e40fe360 2023-07-14T19:29:04.819 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:04.811+0000 7f08e37fe700 1 --2- 172.21.15.98:0/1201225294 >> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] conn(0x7f08d00a1150 0x7f08d00a3600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:29:04.819 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:04.811+0000 7f08e37fe700 1 --2- 172.21.15.98:0/1201225294 >> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] conn(0x7f08d00a1150 0x7f08d00a3600 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f08cc0097b0 tx=0x7f08cc006d20 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:29:04.819 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:04.814+0000 7f08e17fa700 1 -- 172.21.15.98:0/1201225294 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f08d40961b0 con 0x7f08e40fe360 2023-07-14T19:29:04.941 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:29:04 smithi101 ceph-mon[128807]: pgmap v254: 97 pgs: 24 creating+peering, 41 unknown, 32 active+clean; 577 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail; 767 B/s wr, 2 op/s 2023-07-14T19:29:04.941 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:29:04 smithi101 ceph-mon[128807]: from='client.? 172.21.15.98:0/4044846340' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]': finished 2023-07-14T19:29:04.942 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:29:04 smithi101 ceph-mon[128807]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2023-07-14T19:29:04.942 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:29:04 smithi101 ceph-mon[128807]: from='client.? 172.21.15.98:0/1234726812' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "3"}]: dispatch 2023-07-14T19:29:04.970 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:29:04 smithi139 ceph-mon[129293]: pgmap v254: 97 pgs: 24 creating+peering, 41 unknown, 32 active+clean; 577 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail; 767 B/s wr, 2 op/s 2023-07-14T19:29:04.970 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:29:04 smithi139 ceph-mon[129293]: from='client.? 172.21.15.98:0/4044846340' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]': finished 2023-07-14T19:29:04.971 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:29:04 smithi139 ceph-mon[129293]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2023-07-14T19:29:04.971 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:29:04 smithi139 ceph-mon[129293]: from='client.? 172.21.15.98:0/1234726812' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "3"}]: dispatch 2023-07-14T19:29:05.031 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:29:04 smithi098 ceph-mon[127486]: pgmap v254: 97 pgs: 24 creating+peering, 41 unknown, 32 active+clean; 577 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail; 767 B/s wr, 2 op/s 2023-07-14T19:29:05.032 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:29:04 smithi098 ceph-mon[127486]: from='client.? 172.21.15.98:0/4044846340' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]': finished 2023-07-14T19:29:05.032 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:29:04 smithi098 ceph-mon[127486]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2023-07-14T19:29:05.032 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:29:04 smithi098 ceph-mon[127486]: from='client.? 172.21.15.98:0/1234726812' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "3"}]: dispatch 2023-07-14T19:29:05.047 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:05.047+0000 7f08eaa16700 1 -- 172.21.15.98:0/1201225294 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"} v 0) v1 -- 0x7f08e402cb30 con 0x7f08e40fe360 2023-07-14T19:29:05.666 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:05.665+0000 7f08e17fa700 1 -- 172.21.15.98:0/1201225294 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]=0 v20) v1 ==== 121+0+0 (secure 0 0 0) 0x7f08d4009940 con 0x7f08e40fe360 2023-07-14T19:29:05.668 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:05.667+0000 7f08eaa16700 1 -- 172.21.15.98:0/1201225294 >> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] conn(0x7f08d00a1150 msgr2=0x7f08d00a3600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:29:05.668 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:05.667+0000 7f08eaa16700 1 --2- 172.21.15.98:0/1201225294 >> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] conn(0x7f08d00a1150 0x7f08d00a3600 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f08cc0097b0 tx=0x7f08cc006d20 comp rx=0 tx=0).stop 2023-07-14T19:29:05.668 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:05.667+0000 7f08eaa16700 1 -- 172.21.15.98:0/1201225294 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f08e40fe360 msgr2=0x7f08e40fbc90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:29:05.668 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:05.667+0000 7f08eaa16700 1 --2- 172.21.15.98:0/1201225294 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f08e40fe360 0x7f08e40fbc90 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f08d400d980 tx=0x7f08d400dd40 comp rx=0 tx=0).stop 2023-07-14T19:29:05.669 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:05.667+0000 7f08eaa16700 1 -- 172.21.15.98:0/1201225294 shutdown_connections 2023-07-14T19:29:05.669 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:05.667+0000 7f08eaa16700 1 --2- 172.21.15.98:0/1201225294 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f08e40ffee0 0x7f08e419ce60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:29:05.669 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:05.667+0000 7f08eaa16700 1 --2- 172.21.15.98:0/1201225294 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f08e40fe360 0x7f08e40fbc90 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:29:05.669 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:05.667+0000 7f08eaa16700 1 --2- 172.21.15.98:0/1201225294 >> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] conn(0x7f08d00a1150 0x7f08d00a3600 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:29:05.669 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:05.667+0000 7f08eaa16700 1 --2- 172.21.15.98:0/1201225294 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f08e40ff550 0x7f08e419a820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:29:05.669 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:05.667+0000 7f08eaa16700 1 -- 172.21.15.98:0/1201225294 >> 172.21.15.98:0/1201225294 conn(0x7f08e40f9980 msgr2=0x7f08e4103010 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:29:05.669 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:05.668+0000 7f08eaa16700 1 -- 172.21.15.98:0/1201225294 shutdown_connections 2023-07-14T19:29:05.670 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:05.668+0000 7f08eaa16700 1 -- 172.21.15.98:0/1201225294 wait complete. 2023-07-14T19:29:05.683 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs session_timeout 300 2023-07-14T19:29:05.806 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:05.805+0000 7f4909a7f700 1 -- 172.21.15.98:0/3434395576 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f49040fed40 msgr2=0x7f49040ff1b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:29:05.807 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:05.806+0000 7f4909a7f700 1 --2- 172.21.15.98:0/3434395576 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f49040fed40 0x7f49040ff1b0 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f48f4009990 tx=0x7f48f4009ca0 comp rx=0 tx=0).stop 2023-07-14T19:29:05.807 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:05.806+0000 7f4909a7f700 1 -- 172.21.15.98:0/3434395576 shutdown_connections 2023-07-14T19:29:05.807 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:05.806+0000 7f4909a7f700 1 --2- 172.21.15.98:0/3434395576 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f4904100180 0x7f4904100590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:29:05.807 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:05.806+0000 7f4909a7f700 1 --2- 172.21.15.98:0/3434395576 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f49040ff6f0 0x7f4904106fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:29:05.807 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:05.806+0000 7f4909a7f700 1 --2- 172.21.15.98:0/3434395576 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f49040fed40 0x7f49040ff1b0 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:29:05.808 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:05.806+0000 7f4909a7f700 1 -- 172.21.15.98:0/3434395576 >> 172.21.15.98:0/3434395576 conn(0x7f49040f98a0 msgr2=0x7f49040fbcd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:29:05.808 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:05.806+0000 7f4909a7f700 1 -- 172.21.15.98:0/3434395576 shutdown_connections 2023-07-14T19:29:05.808 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:05.806+0000 7f4909a7f700 1 -- 172.21.15.98:0/3434395576 wait complete. 2023-07-14T19:29:05.808 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:05.806+0000 7f4909a7f700 1 Processor -- start 2023-07-14T19:29:05.810 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:05.806+0000 7f4909a7f700 1 -- start start 2023-07-14T19:29:05.810 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:05.807+0000 7f4909a7f700 1 --2- >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f49040fed40 0x7f4904194830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:29:05.810 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:05.807+0000 7f4909a7f700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f49040ff6f0 0x7f4904194d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:29:05.810 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:05.807+0000 7f4909a7f700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f4904100180 0x7f490419bde0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:29:05.810 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:05.807+0000 7f4909a7f700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f490419c3b0 con 0x7f49040ff6f0 2023-07-14T19:29:05.810 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:05.807+0000 7f4909a7f700 1 -- --> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f490419c4f0 con 0x7f49040fed40 2023-07-14T19:29:05.811 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:05.807+0000 7f4909a7f700 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f490419c680 con 0x7f4904100180 2023-07-14T19:29:05.811 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:05.807+0000 7f4902ffd700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f49040ff6f0 0x7f4904194d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:29:05.811 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:05.807+0000 7f4902ffd700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f49040ff6f0 0x7f4904194d70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:45736/0 (socket says 172.21.15.98:45736) 2023-07-14T19:29:05.811 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:05.807+0000 7f4902ffd700 1 -- 172.21.15.98:0/938489406 learned_addr learned my addr 172.21.15.98:0/938489406 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-07-14T19:29:05.811 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:05.807+0000 7f49037fe700 1 --2- 172.21.15.98:0/938489406 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f49040fed40 0x7f4904194830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:29:05.811 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:05.807+0000 7f4903fff700 1 --2- 172.21.15.98:0/938489406 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f4904100180 0x7f490419bde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:29:05.811 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:05.807+0000 7f4902ffd700 1 -- 172.21.15.98:0/938489406 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f4904100180 msgr2=0x7f490419bde0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:29:05.812 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:05.807+0000 7f4902ffd700 1 --2- 172.21.15.98:0/938489406 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f4904100180 0x7f490419bde0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:29:05.812 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:05.807+0000 7f4902ffd700 1 -- 172.21.15.98:0/938489406 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f49040fed40 msgr2=0x7f4904194830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:29:05.812 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:05.807+0000 7f4902ffd700 1 --2- 172.21.15.98:0/938489406 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f49040fed40 0x7f4904194830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:29:05.812 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:05.807+0000 7f4902ffd700 1 -- 172.21.15.98:0/938489406 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f48f40095f0 con 0x7f49040ff6f0 2023-07-14T19:29:05.812 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:05.807+0000 7f4902ffd700 1 --2- 172.21.15.98:0/938489406 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f49040ff6f0 0x7f4904194d70 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f48f4000c00 tx=0x7f48f400f710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:29:05.812 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:05.807+0000 7f4900ff9700 1 -- 172.21.15.98:0/938489406 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f48f401d070 con 0x7f49040ff6f0 2023-07-14T19:29:05.812 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:05.807+0000 7f4909a7f700 1 -- 172.21.15.98:0/938489406 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f490419c900 con 0x7f49040ff6f0 2023-07-14T19:29:05.813 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:05.807+0000 7f4900ff9700 1 -- 172.21.15.98:0/938489406 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f48f400fb50 con 0x7f49040ff6f0 2023-07-14T19:29:05.813 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:05.807+0000 7f4900ff9700 1 -- 172.21.15.98:0/938489406 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f48f4017650 con 0x7f49040ff6f0 2023-07-14T19:29:05.813 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:05.807+0000 7f4909a7f700 1 -- 172.21.15.98:0/938489406 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f490419ce00 con 0x7f49040ff6f0 2023-07-14T19:29:05.813 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:05.808+0000 7f4909a7f700 1 -- 172.21.15.98:0/938489406 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f49041060f0 con 0x7f49040ff6f0 2023-07-14T19:29:05.813 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:05.809+0000 7f4900ff9700 1 -- 172.21.15.98:0/938489406 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 16) v1 ==== 135715+0+0 (secure 0 0 0) 0x7f48f400fcc0 con 0x7f49040ff6f0 2023-07-14T19:29:05.814 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:05.810+0000 7f4900ff9700 1 --2- 172.21.15.98:0/938489406 >> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] conn(0x7f48ec0a1150 0x7f48ec0a3600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:29:05.814 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:05.810+0000 7f49037fe700 1 --2- 172.21.15.98:0/938489406 >> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] conn(0x7f48ec0a1150 0x7f48ec0a3600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:29:05.814 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:05.810+0000 7f4900ff9700 1 -- 172.21.15.98:0/938489406 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 7979+0+0 (secure 0 0 0) 0x7f48f40cdf50 con 0x7f49040ff6f0 2023-07-14T19:29:05.814 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:05.810+0000 7f49037fe700 1 --2- 172.21.15.98:0/938489406 >> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] conn(0x7f48ec0a1150 0x7f48ec0a3600 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f48f8005950 tx=0x7f48f800b410 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:29:05.814 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:05.813+0000 7f4900ff9700 1 -- 172.21.15.98:0/938489406 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f48f40977e0 con 0x7f49040ff6f0 2023-07-14T19:29:05.941 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:29:05 smithi101 ceph-mon[128807]: Health check failed: 1 filesystem is online with fewer MDS than max_mds (MDS_UP_LESS_THAN_MAX) 2023-07-14T19:29:05.941 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:29:05 smithi101 ceph-mon[128807]: from='client.? 172.21.15.98:0/1234726812' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "3"}]': finished 2023-07-14T19:29:05.942 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:29:05 smithi101 ceph-mon[128807]: daemon mds.i assigned to filesystem cephfs as rank 1 (now has 2 ranks) 2023-07-14T19:29:05.942 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:29:05 smithi101 ceph-mon[128807]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2023-07-14T19:29:05.942 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:29:05 smithi101 ceph-mon[128807]: fsmap cephfs:2 {0=b=up:active,1=i=up:creating} 10 up:standby 2023-07-14T19:29:05.942 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:29:05 smithi101 ceph-mon[128807]: daemon mds.i is now active in filesystem cephfs as rank 1 2023-07-14T19:29:05.942 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:29:05 smithi101 ceph-mon[128807]: from='client.? 172.21.15.98:0/1201225294' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]: dispatch 2023-07-14T19:29:05.970 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:29:05 smithi139 ceph-mon[129293]: Health check failed: 1 filesystem is online with fewer MDS than max_mds (MDS_UP_LESS_THAN_MAX) 2023-07-14T19:29:05.970 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:29:05 smithi139 ceph-mon[129293]: from='client.? 172.21.15.98:0/1234726812' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "3"}]': finished 2023-07-14T19:29:05.971 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:29:05 smithi139 ceph-mon[129293]: daemon mds.i assigned to filesystem cephfs as rank 1 (now has 2 ranks) 2023-07-14T19:29:05.971 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:29:05 smithi139 ceph-mon[129293]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2023-07-14T19:29:05.971 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:29:05 smithi139 ceph-mon[129293]: fsmap cephfs:2 {0=b=up:active,1=i=up:creating} 10 up:standby 2023-07-14T19:29:05.971 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:29:05 smithi139 ceph-mon[129293]: daemon mds.i is now active in filesystem cephfs as rank 1 2023-07-14T19:29:05.971 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:29:05 smithi139 ceph-mon[129293]: from='client.? 172.21.15.98:0/1201225294' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]: dispatch 2023-07-14T19:29:06.031 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:29:05 smithi098 ceph-mon[127486]: Health check failed: 1 filesystem is online with fewer MDS than max_mds (MDS_UP_LESS_THAN_MAX) 2023-07-14T19:29:06.031 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:29:05 smithi098 ceph-mon[127486]: from='client.? 172.21.15.98:0/1234726812' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "3"}]': finished 2023-07-14T19:29:06.031 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:29:05 smithi098 ceph-mon[127486]: daemon mds.i assigned to filesystem cephfs as rank 1 (now has 2 ranks) 2023-07-14T19:29:06.031 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:29:05 smithi098 ceph-mon[127486]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2023-07-14T19:29:06.031 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:29:05 smithi098 ceph-mon[127486]: fsmap cephfs:2 {0=b=up:active,1=i=up:creating} 10 up:standby 2023-07-14T19:29:06.032 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:29:05 smithi098 ceph-mon[127486]: daemon mds.i is now active in filesystem cephfs as rank 1 2023-07-14T19:29:06.032 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:29:05 smithi098 ceph-mon[127486]: from='client.? 172.21.15.98:0/1201225294' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]: dispatch 2023-07-14T19:29:06.050 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:06.050+0000 7f4909a7f700 1 -- 172.21.15.98:0/938489406 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"} v 0) v1 -- 0x7f490402c8e0 con 0x7f49040ff6f0 2023-07-14T19:29:06.295 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:06.295+0000 7f4900ff9700 1 -- 172.21.15.98:0/938489406 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]=0 v22) v1 ==== 115+0+0 (secure 0 0 0) 0x7f48f409ae00 con 0x7f49040ff6f0 2023-07-14T19:29:06.298 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:06.297+0000 7f4909a7f700 1 -- 172.21.15.98:0/938489406 >> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] conn(0x7f48ec0a1150 msgr2=0x7f48ec0a3600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:29:06.298 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:06.297+0000 7f4909a7f700 1 --2- 172.21.15.98:0/938489406 >> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] conn(0x7f48ec0a1150 0x7f48ec0a3600 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f48f8005950 tx=0x7f48f800b410 comp rx=0 tx=0).stop 2023-07-14T19:29:06.298 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:06.297+0000 7f4909a7f700 1 -- 172.21.15.98:0/938489406 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f49040ff6f0 msgr2=0x7f4904194d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:29:06.298 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:06.297+0000 7f4909a7f700 1 --2- 172.21.15.98:0/938489406 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f49040ff6f0 0x7f4904194d70 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f48f4000c00 tx=0x7f48f400f710 comp rx=0 tx=0).stop 2023-07-14T19:29:06.298 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:06.297+0000 7f4909a7f700 1 -- 172.21.15.98:0/938489406 shutdown_connections 2023-07-14T19:29:06.298 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:06.297+0000 7f4909a7f700 1 --2- 172.21.15.98:0/938489406 >> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] conn(0x7f48ec0a1150 0x7f48ec0a3600 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:29:06.299 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:06.297+0000 7f4909a7f700 1 --2- 172.21.15.98:0/938489406 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f4904100180 0x7f490419bde0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:29:06.299 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:06.297+0000 7f4909a7f700 1 --2- 172.21.15.98:0/938489406 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f49040fed40 0x7f4904194830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:29:06.299 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:06.297+0000 7f4909a7f700 1 --2- 172.21.15.98:0/938489406 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f49040ff6f0 0x7f4904194d70 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:29:06.299 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:06.297+0000 7f4909a7f700 1 -- 172.21.15.98:0/938489406 >> 172.21.15.98:0/938489406 conn(0x7f49040f98a0 msgr2=0x7f4904103230 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:29:06.299 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:06.297+0000 7f4909a7f700 1 -- 172.21.15.98:0/938489406 shutdown_connections 2023-07-14T19:29:06.299 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:06.297+0000 7f4909a7f700 1 -- 172.21.15.98:0/938489406 wait complete. 2023-07-14T19:29:06.310 DEBUG:tasks.cephfs.filesystem:Creating {'create': 2} subvols for filesystem 'cephfs' 2023-07-14T19:29:06.311 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume create cephfs sv_0 '' 2023-07-14T19:29:06.436 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:06.435+0000 7f096618e700 1 -- 172.21.15.98:0/3924233987 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f09600ffd30 msgr2=0x7f0960100180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:29:06.436 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:06.435+0000 7f096618e700 1 --2- 172.21.15.98:0/3924233987 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f09600ffd30 0x7f0960100180 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f0950009990 tx=0x7f0950009ca0 comp rx=0 tx=0).stop 2023-07-14T19:29:06.436 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:06.435+0000 7f096618e700 1 -- 172.21.15.98:0/3924233987 shutdown_connections 2023-07-14T19:29:06.436 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:06.435+0000 7f096618e700 1 --2- 172.21.15.98:0/3924233987 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f09600feb40 0x7f09600fef50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:29:06.436 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:06.435+0000 7f096618e700 1 --2- 172.21.15.98:0/3924233987 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f09601006c0 0x7f0960106f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:29:06.437 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:06.435+0000 7f096618e700 1 --2- 172.21.15.98:0/3924233987 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f09600ffd30 0x7f0960100180 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:29:06.437 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:06.435+0000 7f096618e700 1 -- 172.21.15.98:0/3924233987 >> 172.21.15.98:0/3924233987 conn(0x7f09600fa160 msgr2=0x7f09600fc590 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:29:06.437 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:06.435+0000 7f096618e700 1 -- 172.21.15.98:0/3924233987 shutdown_connections 2023-07-14T19:29:06.437 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:06.435+0000 7f096618e700 1 -- 172.21.15.98:0/3924233987 wait complete. 2023-07-14T19:29:06.437 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:06.435+0000 7f096618e700 1 Processor -- start 2023-07-14T19:29:06.440 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:06.436+0000 7f096618e700 1 -- start start 2023-07-14T19:29:06.440 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:06.436+0000 7f096618e700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f09600feb40 0x7f09600692e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:29:06.440 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:06.436+0000 7f096618e700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f09600ffd30 0x7f096019a690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:29:06.441 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:06.436+0000 7f096618e700 1 --2- >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f09601006c0 0x7f096019ccd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:29:06.441 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:06.436+0000 7f096618e700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f096019d470 con 0x7f09600ffd30 2023-07-14T19:29:06.441 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:06.436+0000 7f096618e700 1 -- --> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f096019d5e0 con 0x7f09601006c0 2023-07-14T19:29:06.441 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:06.436+0000 7f096618e700 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f096019d7a0 con 0x7f09600feb40 2023-07-14T19:29:06.441 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:06.436+0000 7f095effd700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f09600ffd30 0x7f096019a690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:29:06.441 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:06.436+0000 7f095effd700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f09600ffd30 0x7f096019a690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:45750/0 (socket says 172.21.15.98:45750) 2023-07-14T19:29:06.441 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:06.436+0000 7f095effd700 1 -- 172.21.15.98:0/56156018 learned_addr learned my addr 172.21.15.98:0/56156018 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-07-14T19:29:06.442 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:06.436+0000 7f095f7fe700 1 --2- 172.21.15.98:0/56156018 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f09600feb40 0x7f09600692e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:29:06.442 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:06.436+0000 7f095ffff700 1 --2- 172.21.15.98:0/56156018 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f09601006c0 0x7f096019ccd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:29:06.442 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:06.436+0000 7f095effd700 1 -- 172.21.15.98:0/56156018 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f09600feb40 msgr2=0x7f09600692e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:29:06.442 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:06.436+0000 7f095effd700 1 --2- 172.21.15.98:0/56156018 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f09600feb40 0x7f09600692e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:29:06.442 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:06.436+0000 7f095effd700 1 -- 172.21.15.98:0/56156018 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f09601006c0 msgr2=0x7f096019ccd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:29:06.442 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:06.436+0000 7f095effd700 1 --2- 172.21.15.98:0/56156018 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f09601006c0 0x7f096019ccd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:29:06.443 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:06.436+0000 7f095effd700 1 -- 172.21.15.98:0/56156018 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f09500095f0 con 0x7f09600ffd30 2023-07-14T19:29:06.443 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:06.436+0000 7f095f7fe700 1 --2- 172.21.15.98:0/56156018 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f09600feb40 0x7f09600692e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-07-14T19:29:06.443 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:06.436+0000 7f095effd700 1 --2- 172.21.15.98:0/56156018 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f09600ffd30 0x7f096019a690 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f095000c010 tx=0x7f0950005e80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:29:06.443 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:06.436+0000 7f095cff9700 1 -- 172.21.15.98:0/56156018 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f095001d070 con 0x7f09600ffd30 2023-07-14T19:29:06.443 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:06.436+0000 7f096618e700 1 -- 172.21.15.98:0/56156018 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f096019da20 con 0x7f09600ffd30 2023-07-14T19:29:06.443 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:06.436+0000 7f095cff9700 1 -- 172.21.15.98:0/56156018 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f0950004ca0 con 0x7f09600ffd30 2023-07-14T19:29:06.443 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:06.436+0000 7f095cff9700 1 -- 172.21.15.98:0/56156018 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f09500055e0 con 0x7f09600ffd30 2023-07-14T19:29:06.444 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:06.437+0000 7f096618e700 1 -- 172.21.15.98:0/56156018 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f096019dde0 con 0x7f09600ffd30 2023-07-14T19:29:06.444 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:06.437+0000 7f096618e700 1 -- 172.21.15.98:0/56156018 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f096010a3d0 con 0x7f09600ffd30 2023-07-14T19:29:06.444 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:06.438+0000 7f095cff9700 1 -- 172.21.15.98:0/56156018 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 16) v1 ==== 135715+0+0 (secure 0 0 0) 0x7f095000fc70 con 0x7f09600ffd30 2023-07-14T19:29:06.444 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:06.439+0000 7f095cff9700 1 --2- 172.21.15.98:0/56156018 >> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] conn(0x7f09480a1150 0x7f09480a3600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:29:06.444 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:06.439+0000 7f095f7fe700 1 --2- 172.21.15.98:0/56156018 >> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] conn(0x7f09480a1150 0x7f09480a3600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:29:06.444 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:06.439+0000 7f095cff9700 1 -- 172.21.15.98:0/56156018 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 7979+0+0 (secure 0 0 0) 0x7f09500ce000 con 0x7f09600ffd30 2023-07-14T19:29:06.444 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:06.439+0000 7f095f7fe700 1 --2- 172.21.15.98:0/56156018 >> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] conn(0x7f09480a1150 0x7f09480a3600 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f09600ffb90 tx=0x7f0954009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:29:06.445 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:06.443+0000 7f095cff9700 1 -- 172.21.15.98:0/56156018 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f0950097930 con 0x7f09600ffd30 2023-07-14T19:29:06.664 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:06.663+0000 7f096618e700 1 -- 172.21.15.98:0/56156018 --> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] -- mgr_command(tid 0: {"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_0", "group_name": "", "target": ["mon-mgr", ""]}) v1 -- 0x7f0960060a80 con 0x7f09480a1150 2023-07-14T19:29:06.665 INFO:journalctl@ceph.mds.b.smithi101.stdout:Jul 14 19:29:06 smithi101 ceph-9adea648-227b-11ee-9b33-001a4aab830c-mds-b[154069]: 2023-07-14T19:29:06.219+0000 7f7e3d9ac700 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2023-07-14T19:29:06.665 INFO:journalctl@ceph.mds.b.smithi101.stdout:Jul 14 19:29:06 smithi101 ceph-9adea648-227b-11ee-9b33-001a4aab830c-mds-b[154069]: 2023-07-14T19:29:06.219+0000 7f7e3d9ac700 -1 mds.pinger is_rank_lagging: rank=1 was never sent ping request. 2023-07-14T19:29:06.706 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:06.706+0000 7f095cff9700 1 -- 172.21.15.98:0/56156018 <== mgr.14152 v2:172.21.15.98:6800/452042164 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f0960060a80 con 0x7f09480a1150 2023-07-14T19:29:06.709 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:06.708+0000 7f096618e700 1 -- 172.21.15.98:0/56156018 >> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] conn(0x7f09480a1150 msgr2=0x7f09480a3600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:29:06.710 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:06.708+0000 7f096618e700 1 --2- 172.21.15.98:0/56156018 >> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] conn(0x7f09480a1150 0x7f09480a3600 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f09600ffb90 tx=0x7f0954009260 comp rx=0 tx=0).stop 2023-07-14T19:29:06.711 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:06.708+0000 7f096618e700 1 -- 172.21.15.98:0/56156018 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f09600ffd30 msgr2=0x7f096019a690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:29:06.711 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:06.708+0000 7f096618e700 1 --2- 172.21.15.98:0/56156018 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f09600ffd30 0x7f096019a690 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f095000c010 tx=0x7f0950005e80 comp rx=0 tx=0).stop 2023-07-14T19:29:06.712 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:06.708+0000 7f096618e700 1 -- 172.21.15.98:0/56156018 shutdown_connections 2023-07-14T19:29:06.712 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:06.708+0000 7f096618e700 1 --2- 172.21.15.98:0/56156018 >> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] conn(0x7f09480a1150 0x7f09480a3600 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:29:06.712 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:06.708+0000 7f096618e700 1 --2- 172.21.15.98:0/56156018 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f09600feb40 0x7f09600692e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:29:06.713 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:06.708+0000 7f096618e700 1 --2- 172.21.15.98:0/56156018 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f09601006c0 0x7f096019ccd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:29:06.713 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:06.708+0000 7f096618e700 1 --2- 172.21.15.98:0/56156018 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f09600ffd30 0x7f096019a690 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:29:06.713 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:06.708+0000 7f096618e700 1 -- 172.21.15.98:0/56156018 >> 172.21.15.98:0/56156018 conn(0x7f09600fa160 msgr2=0x7f09600fc330 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:29:06.713 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:06.708+0000 7f096618e700 1 -- 172.21.15.98:0/56156018 shutdown_connections 2023-07-14T19:29:06.714 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:06.708+0000 7f096618e700 1 -- 172.21.15.98:0/56156018 wait complete. 2023-07-14T19:29:06.720 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume getpath cephfs sv_0 2023-07-14T19:29:06.845 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:06.843+0000 7faff42bc700 1 -- 172.21.15.98:0/2655230559 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fafec1006f0 msgr2=0x7fafec102ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:29:06.845 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:06.843+0000 7faff42bc700 1 --2- 172.21.15.98:0/2655230559 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fafec1006f0 0x7fafec102ad0 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7fafdc009990 tx=0x7fafdc009ca0 comp rx=0 tx=0).stop 2023-07-14T19:29:06.846 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:06.843+0000 7faff42bc700 1 -- 172.21.15.98:0/2655230559 shutdown_connections 2023-07-14T19:29:06.846 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:06.843+0000 7faff42bc700 1 --2- 172.21.15.98:0/2655230559 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7fafec105930 0x7fafec107d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:29:06.846 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:06.843+0000 7faff42bc700 1 --2- 172.21.15.98:0/2655230559 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fafec1006f0 0x7fafec102ad0 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:29:06.846 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:06.843+0000 7faff42bc700 1 --2- 172.21.15.98:0/2655230559 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fafec103010 0x7fafec1053f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:29:06.846 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:06.843+0000 7faff42bc700 1 -- 172.21.15.98:0/2655230559 >> 172.21.15.98:0/2655230559 conn(0x7fafec0fa130 msgr2=0x7fafec0fc560 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:29:06.846 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:06.843+0000 7faff42bc700 1 -- 172.21.15.98:0/2655230559 shutdown_connections 2023-07-14T19:29:06.846 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:06.843+0000 7faff42bc700 1 -- 172.21.15.98:0/2655230559 wait complete. 2023-07-14T19:29:06.847 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:06.844+0000 7faff42bc700 1 Processor -- start 2023-07-14T19:29:06.847 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:06.844+0000 7faff42bc700 1 -- start start 2023-07-14T19:29:06.847 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:06.844+0000 7faff42bc700 1 --2- >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7fafec1006f0 0x7fafec100060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:29:06.847 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:06.844+0000 7faff42bc700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fafec103010 0x7fafec0fe6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:29:06.847 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:06.844+0000 7faff42bc700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fafec105930 0x7fafec0febe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:29:06.847 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:06.844+0000 7faff42bc700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fafec0ff380 con 0x7fafec103010 2023-07-14T19:29:06.847 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:06.844+0000 7faff42bc700 1 -- --> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fafec0ff4f0 con 0x7fafec1006f0 2023-07-14T19:29:06.847 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:06.844+0000 7faff42bc700 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fafec19f980 con 0x7fafec105930 2023-07-14T19:29:06.848 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:06.844+0000 7faff1857700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fafec103010 0x7fafec0fe6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:29:06.848 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:06.844+0000 7faff1857700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fafec103010 0x7fafec0fe6a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:45784/0 (socket says 172.21.15.98:45784) 2023-07-14T19:29:06.848 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:06.844+0000 7faff1857700 1 -- 172.21.15.98:0/1505844203 learned_addr learned my addr 172.21.15.98:0/1505844203 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-07-14T19:29:06.848 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:06.844+0000 7faff2058700 1 --2- 172.21.15.98:0/1505844203 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7fafec1006f0 0x7fafec100060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:29:06.848 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:06.844+0000 7faff1857700 1 -- 172.21.15.98:0/1505844203 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fafec105930 msgr2=0x7fafec0febe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:29:06.848 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:06.844+0000 7faff1857700 1 --2- 172.21.15.98:0/1505844203 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fafec105930 0x7fafec0febe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:29:06.848 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:06.844+0000 7faff1857700 1 -- 172.21.15.98:0/1505844203 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7fafec1006f0 msgr2=0x7fafec100060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:29:06.849 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:06.844+0000 7faff1857700 1 --2- 172.21.15.98:0/1505844203 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7fafec1006f0 0x7fafec100060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:29:06.849 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:06.844+0000 7faff1857700 1 -- 172.21.15.98:0/1505844203 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fafdc0095f0 con 0x7fafec103010 2023-07-14T19:29:06.849 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:06.845+0000 7faff1857700 1 --2- 172.21.15.98:0/1505844203 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fafec103010 0x7fafec0fe6a0 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7fafe800c920 tx=0x7fafe800cc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:29:06.849 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:06.845+0000 7fafe37fe700 1 -- 172.21.15.98:0/1505844203 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fafe8007b00 con 0x7fafec103010 2023-07-14T19:29:06.849 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:06.845+0000 7faff42bc700 1 -- 172.21.15.98:0/1505844203 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fafec19fc60 con 0x7fafec103010 2023-07-14T19:29:06.849 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:06.845+0000 7fafe37fe700 1 -- 172.21.15.98:0/1505844203 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fafe800ce70 con 0x7fafec103010 2023-07-14T19:29:06.849 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:06.845+0000 7fafe37fe700 1 -- 172.21.15.98:0/1505844203 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fafe80054e0 con 0x7fafec103010 2023-07-14T19:29:06.850 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:06.845+0000 7faff42bc700 1 -- 172.21.15.98:0/1505844203 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fafec1a0170 con 0x7fafec103010 2023-07-14T19:29:06.850 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:06.846+0000 7faff42bc700 1 -- 172.21.15.98:0/1505844203 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fafec0fbca0 con 0x7fafec103010 2023-07-14T19:29:06.851 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:06.847+0000 7fafe37fe700 1 -- 172.21.15.98:0/1505844203 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 16) v1 ==== 135715+0+0 (secure 0 0 0) 0x7fafe8004750 con 0x7fafec103010 2023-07-14T19:29:06.851 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:06.847+0000 7fafe37fe700 1 --2- 172.21.15.98:0/1505844203 >> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] conn(0x7fafd80a1150 0x7fafd80a3600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:29:06.851 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:06.847+0000 7faff2058700 1 --2- 172.21.15.98:0/1505844203 >> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] conn(0x7fafd80a1150 0x7fafd80a3600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:29:06.851 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:06.847+0000 7fafe37fe700 1 -- 172.21.15.98:0/1505844203 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 7979+0+0 (secure 0 0 0) 0x7fafe80cc310 con 0x7fafec103010 2023-07-14T19:29:06.851 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:06.848+0000 7faff2058700 1 --2- 172.21.15.98:0/1505844203 >> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] conn(0x7fafd80a1150 0x7fafd80a3600 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7fafdc00b5c0 tx=0x7fafdc005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:29:06.852 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:06.851+0000 7fafe37fe700 1 -- 172.21.15.98:0/1505844203 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7fafe8095bb0 con 0x7fafec103010 2023-07-14T19:29:06.941 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:29:06 smithi101 ceph-mon[128807]: pgmap v255: 97 pgs: 21 creating+peering, 76 active+clean; 579 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s wr, 4 op/s 2023-07-14T19:29:06.942 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:29:06 smithi101 ceph-mon[128807]: mds.? [v2:172.21.15.139:6838/1258854143,v1:172.21.15.139:6839/1258854143] up:active 2023-07-14T19:29:06.942 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:29:06 smithi101 ceph-mon[128807]: from='client.? 172.21.15.98:0/1201225294' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]': finished 2023-07-14T19:29:06.942 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:29:06 smithi101 ceph-mon[128807]: daemon mds.h assigned to filesystem cephfs as rank 2 (now has 3 ranks) 2023-07-14T19:29:06.942 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:29:06 smithi101 ceph-mon[128807]: Health check cleared: MDS_UP_LESS_THAN_MAX (was: 1 filesystem is online with fewer MDS than max_mds) 2023-07-14T19:29:06.942 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:29:06 smithi101 ceph-mon[128807]: Cluster is now healthy 2023-07-14T19:29:06.942 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:29:06 smithi101 ceph-mon[128807]: fsmap cephfs:2 {0=b=up:active,1=i=up:active} 10 up:standby 2023-07-14T19:29:06.942 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:29:06 smithi101 ceph-mon[128807]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:creating} 9 up:standby 2023-07-14T19:29:06.943 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:29:06 smithi101 ceph-mon[128807]: daemon mds.h is now active in filesystem cephfs as rank 2 2023-07-14T19:29:06.943 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:29:06 smithi101 ceph-mon[128807]: from='client.? 172.21.15.98:0/938489406' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]: dispatch 2023-07-14T19:29:06.943 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:29:06 smithi101 ceph-mon[128807]: mds.? [v2:172.21.15.101:6836/485741528,v1:172.21.15.101:6837/485741528] up:active 2023-07-14T19:29:06.943 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:29:06 smithi101 ceph-mon[128807]: from='client.? 172.21.15.98:0/938489406' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]': finished 2023-07-14T19:29:06.943 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:29:06 smithi101 ceph-mon[128807]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2023-07-14T19:29:06.970 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:29:06 smithi139 ceph-mon[129293]: pgmap v255: 97 pgs: 21 creating+peering, 76 active+clean; 579 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s wr, 4 op/s 2023-07-14T19:29:06.970 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:29:06 smithi139 ceph-mon[129293]: mds.? [v2:172.21.15.139:6838/1258854143,v1:172.21.15.139:6839/1258854143] up:active 2023-07-14T19:29:06.971 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:29:06 smithi139 ceph-mon[129293]: from='client.? 172.21.15.98:0/1201225294' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]': finished 2023-07-14T19:29:06.971 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:29:06 smithi139 ceph-mon[129293]: daemon mds.h assigned to filesystem cephfs as rank 2 (now has 3 ranks) 2023-07-14T19:29:06.971 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:29:06 smithi139 ceph-mon[129293]: Health check cleared: MDS_UP_LESS_THAN_MAX (was: 1 filesystem is online with fewer MDS than max_mds) 2023-07-14T19:29:06.971 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:29:06 smithi139 ceph-mon[129293]: Cluster is now healthy 2023-07-14T19:29:06.971 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:29:06 smithi139 ceph-mon[129293]: fsmap cephfs:2 {0=b=up:active,1=i=up:active} 10 up:standby 2023-07-14T19:29:06.971 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:29:06 smithi139 ceph-mon[129293]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:creating} 9 up:standby 2023-07-14T19:29:06.971 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:29:06 smithi139 ceph-mon[129293]: daemon mds.h is now active in filesystem cephfs as rank 2 2023-07-14T19:29:06.972 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:29:06 smithi139 ceph-mon[129293]: from='client.? 172.21.15.98:0/938489406' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]: dispatch 2023-07-14T19:29:06.972 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:29:06 smithi139 ceph-mon[129293]: mds.? [v2:172.21.15.101:6836/485741528,v1:172.21.15.101:6837/485741528] up:active 2023-07-14T19:29:06.972 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:29:06 smithi139 ceph-mon[129293]: from='client.? 172.21.15.98:0/938489406' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]': finished 2023-07-14T19:29:06.972 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:29:06 smithi139 ceph-mon[129293]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2023-07-14T19:29:07.031 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Jul 14 19:29:06 smithi098 ceph-9adea648-227b-11ee-9b33-001a4aab830c-mgr-x[127734]: 2023-07-14T19:29:06.671+0000 7f245642d700 -1 client.0 error registering admin socket command: (17) File exists 2023-07-14T19:29:07.031 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Jul 14 19:29:06 smithi098 ceph-9adea648-227b-11ee-9b33-001a4aab830c-mgr-x[127734]: 2023-07-14T19:29:06.671+0000 7f245642d700 -1 client.0 error registering admin socket command: (17) File exists 2023-07-14T19:29:07.031 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Jul 14 19:29:06 smithi098 ceph-9adea648-227b-11ee-9b33-001a4aab830c-mgr-x[127734]: 2023-07-14T19:29:06.671+0000 7f245642d700 -1 client.0 error registering admin socket command: (17) File exists 2023-07-14T19:29:07.032 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Jul 14 19:29:06 smithi098 ceph-9adea648-227b-11ee-9b33-001a4aab830c-mgr-x[127734]: 2023-07-14T19:29:06.671+0000 7f245642d700 -1 client.0 error registering admin socket command: (17) File exists 2023-07-14T19:29:07.032 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Jul 14 19:29:06 smithi098 ceph-9adea648-227b-11ee-9b33-001a4aab830c-mgr-x[127734]: 2023-07-14T19:29:06.671+0000 7f245642d700 -1 client.0 error registering admin socket command: (17) File exists 2023-07-14T19:29:07.032 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:29:06 smithi098 ceph-mon[127486]: pgmap v255: 97 pgs: 21 creating+peering, 76 active+clean; 579 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s wr, 4 op/s 2023-07-14T19:29:07.032 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:29:06 smithi098 ceph-mon[127486]: mds.? [v2:172.21.15.139:6838/1258854143,v1:172.21.15.139:6839/1258854143] up:active 2023-07-14T19:29:07.032 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:29:06 smithi098 ceph-mon[127486]: from='client.? 172.21.15.98:0/1201225294' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]': finished 2023-07-14T19:29:07.033 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:29:06 smithi098 ceph-mon[127486]: daemon mds.h assigned to filesystem cephfs as rank 2 (now has 3 ranks) 2023-07-14T19:29:07.033 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:29:06 smithi098 ceph-mon[127486]: Health check cleared: MDS_UP_LESS_THAN_MAX (was: 1 filesystem is online with fewer MDS than max_mds) 2023-07-14T19:29:07.033 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:29:06 smithi098 ceph-mon[127486]: Cluster is now healthy 2023-07-14T19:29:07.033 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:29:06 smithi098 ceph-mon[127486]: fsmap cephfs:2 {0=b=up:active,1=i=up:active} 10 up:standby 2023-07-14T19:29:07.033 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:29:06 smithi098 ceph-mon[127486]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:creating} 9 up:standby 2023-07-14T19:29:07.033 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:29:06 smithi098 ceph-mon[127486]: daemon mds.h is now active in filesystem cephfs as rank 2 2023-07-14T19:29:07.033 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:29:06 smithi098 ceph-mon[127486]: from='client.? 172.21.15.98:0/938489406' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]: dispatch 2023-07-14T19:29:07.034 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:29:06 smithi098 ceph-mon[127486]: mds.? [v2:172.21.15.101:6836/485741528,v1:172.21.15.101:6837/485741528] up:active 2023-07-14T19:29:07.034 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:29:06 smithi098 ceph-mon[127486]: from='client.? 172.21.15.98:0/938489406' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]': finished 2023-07-14T19:29:07.034 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:29:06 smithi098 ceph-mon[127486]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2023-07-14T19:29:07.075 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:07.074+0000 7faff42bc700 1 -- 172.21.15.98:0/1505844203 --> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] -- mgr_command(tid 0: {"prefix": "fs subvolume getpath", "vol_name": "cephfs", "sub_name": "sv_0", "target": ["mon-mgr", ""]}) v1 -- 0x7fafec060a80 con 0x7fafd80a1150 2023-07-14T19:29:07.078 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:07.077+0000 7fafe37fe700 1 -- 172.21.15.98:0/1505844203 <== mgr.14152 v2:172.21.15.98:6800/452042164 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+60 (secure 0 0 0) 0x7fafec060a80 con 0x7fafd80a1150 2023-07-14T19:29:07.078 INFO:teuthology.orchestra.run.smithi098.stdout:/volumes/_nogroup/sv_0/202163a2-c0bf-49ef-8a50-5d0ccb9a4081 2023-07-14T19:29:07.080 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:07.080+0000 7faff42bc700 1 -- 172.21.15.98:0/1505844203 >> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] conn(0x7fafd80a1150 msgr2=0x7fafd80a3600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:29:07.081 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:07.080+0000 7faff42bc700 1 --2- 172.21.15.98:0/1505844203 >> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] conn(0x7fafd80a1150 0x7fafd80a3600 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7fafdc00b5c0 tx=0x7fafdc005fd0 comp rx=0 tx=0).stop 2023-07-14T19:29:07.081 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:07.080+0000 7faff42bc700 1 -- 172.21.15.98:0/1505844203 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fafec103010 msgr2=0x7fafec0fe6a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:29:07.081 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:07.080+0000 7faff42bc700 1 --2- 172.21.15.98:0/1505844203 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fafec103010 0x7fafec0fe6a0 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7fafe800c920 tx=0x7fafe800cc30 comp rx=0 tx=0).stop 2023-07-14T19:29:07.081 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:07.080+0000 7faff42bc700 1 -- 172.21.15.98:0/1505844203 shutdown_connections 2023-07-14T19:29:07.081 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:07.080+0000 7faff42bc700 1 --2- 172.21.15.98:0/1505844203 >> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] conn(0x7fafd80a1150 0x7fafd80a3600 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:29:07.081 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:07.080+0000 7faff42bc700 1 --2- 172.21.15.98:0/1505844203 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fafec105930 0x7fafec0febe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:29:07.082 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:07.080+0000 7faff42bc700 1 --2- 172.21.15.98:0/1505844203 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7fafec1006f0 0x7fafec100060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:29:07.082 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:07.080+0000 7faff42bc700 1 --2- 172.21.15.98:0/1505844203 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fafec103010 0x7fafec0fe6a0 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:29:07.082 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:07.080+0000 7faff42bc700 1 -- 172.21.15.98:0/1505844203 >> 172.21.15.98:0/1505844203 conn(0x7fafec0fa130 msgr2=0x7fafec101350 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:29:07.082 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:07.080+0000 7faff42bc700 1 -- 172.21.15.98:0/1505844203 shutdown_connections 2023-07-14T19:29:07.082 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:07.080+0000 7faff42bc700 1 -- 172.21.15.98:0/1505844203 wait complete. 2023-07-14T19:29:07.093 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume create cephfs sv_1 '' 2023-07-14T19:29:07.217 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:07.217+0000 7f8de81a9700 1 -- 172.21.15.98:0/2922661639 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8de01017c0 msgr2=0x7f8de0108080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:29:07.218 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:07.217+0000 7f8de81a9700 1 --2- 172.21.15.98:0/2922661639 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8de01017c0 0x7f8de0108080 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f8dd800b0a0 tx=0x7f8dd800b3b0 comp rx=0 tx=0).stop 2023-07-14T19:29:07.218 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:07.217+0000 7f8de81a9700 1 -- 172.21.15.98:0/2922661639 shutdown_connections 2023-07-14T19:29:07.218 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:07.217+0000 7f8de81a9700 1 --2- 172.21.15.98:0/2922661639 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8de01017c0 0x7f8de0108080 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:29:07.218 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:07.217+0000 7f8de81a9700 1 --2- 172.21.15.98:0/2922661639 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f8de00ffc40 0x7f8de0100050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:29:07.218 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:07.217+0000 7f8de81a9700 1 --2- 172.21.15.98:0/2922661639 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f8de0100e30 0x7f8de0101280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:29:07.218 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:07.217+0000 7f8de81a9700 1 -- 172.21.15.98:0/2922661639 >> 172.21.15.98:0/2922661639 conn(0x7f8de00fb260 msgr2=0x7f8de00fd690 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:29:07.219 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:07.217+0000 7f8de81a9700 1 -- 172.21.15.98:0/2922661639 shutdown_connections 2023-07-14T19:29:07.219 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:07.217+0000 7f8de81a9700 1 -- 172.21.15.98:0/2922661639 wait complete. 2023-07-14T19:29:07.221 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:07.217+0000 7f8de81a9700 1 Processor -- start 2023-07-14T19:29:07.221 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:07.217+0000 7f8de81a9700 1 -- start start 2023-07-14T19:29:07.221 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:07.218+0000 7f8de81a9700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f8de00ffc40 0x7f8de00743f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:29:07.221 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:07.218+0000 7f8de81a9700 1 --2- >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f8de0100e30 0x7f8de0072a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:29:07.221 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:07.218+0000 7f8de81a9700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8de01017c0 0x7f8de0072f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:29:07.222 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:07.218+0000 7f8de81a9700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8de0073710 con 0x7f8de01017c0 2023-07-14T19:29:07.222 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:07.218+0000 7f8de81a9700 1 -- --> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8de0073880 con 0x7f8de0100e30 2023-07-14T19:29:07.222 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:07.218+0000 7f8de81a9700 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8de019fb20 con 0x7f8de00ffc40 2023-07-14T19:29:07.222 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:07.218+0000 7f8de6746700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8de01017c0 0x7f8de0072f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:29:07.222 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:07.218+0000 7f8de6746700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8de01017c0 0x7f8de0072f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:45800/0 (socket says 172.21.15.98:45800) 2023-07-14T19:29:07.223 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:07.218+0000 7f8de6746700 1 -- 172.21.15.98:0/149419900 learned_addr learned my addr 172.21.15.98:0/149419900 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-07-14T19:29:07.223 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:07.218+0000 7f8de5f45700 1 --2- 172.21.15.98:0/149419900 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f8de00ffc40 0x7f8de00743f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:29:07.223 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:07.218+0000 7f8de5744700 1 --2- 172.21.15.98:0/149419900 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f8de0100e30 0x7f8de0072a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:29:07.223 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:07.218+0000 7f8de6746700 1 -- 172.21.15.98:0/149419900 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f8de00ffc40 msgr2=0x7f8de00743f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:29:07.223 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:07.218+0000 7f8de6746700 1 --2- 172.21.15.98:0/149419900 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f8de00ffc40 0x7f8de00743f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:29:07.223 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:07.218+0000 7f8de6746700 1 -- 172.21.15.98:0/149419900 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f8de0100e30 msgr2=0x7f8de0072a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:29:07.224 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:07.218+0000 7f8de6746700 1 --2- 172.21.15.98:0/149419900 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f8de0100e30 0x7f8de0072a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:29:07.224 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:07.218+0000 7f8de6746700 1 -- 172.21.15.98:0/149419900 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8dd8009d00 con 0x7f8de01017c0 2023-07-14T19:29:07.224 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:07.218+0000 7f8de6746700 1 --2- 172.21.15.98:0/149419900 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8de01017c0 0x7f8de0072f70 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f8dd800b830 tx=0x7f8dd8004c80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:29:07.224 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:07.218+0000 7f8dd6ffd700 1 -- 172.21.15.98:0/149419900 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8dd800e040 con 0x7f8de01017c0 2023-07-14T19:29:07.224 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:07.218+0000 7f8dd6ffd700 1 -- 172.21.15.98:0/149419900 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f8dd801db50 con 0x7f8de01017c0 2023-07-14T19:29:07.224 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:07.218+0000 7f8de81a9700 1 -- 172.21.15.98:0/149419900 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8de019fda0 con 0x7f8de01017c0 2023-07-14T19:29:07.224 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:07.218+0000 7f8dd6ffd700 1 -- 172.21.15.98:0/149419900 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8dd801bb20 con 0x7f8de01017c0 2023-07-14T19:29:07.225 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:07.218+0000 7f8de81a9700 1 -- 172.21.15.98:0/149419900 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8de01a01d0 con 0x7f8de01017c0 2023-07-14T19:29:07.225 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:07.219+0000 7f8de81a9700 1 -- 172.21.15.98:0/149419900 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8de0106a80 con 0x7f8de01017c0 2023-07-14T19:29:07.225 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:07.220+0000 7f8dd6ffd700 1 -- 172.21.15.98:0/149419900 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 16) v1 ==== 135715+0+0 (secure 0 0 0) 0x7f8dd801dcc0 con 0x7f8de01017c0 2023-07-14T19:29:07.226 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:07.221+0000 7f8dd6ffd700 1 --2- 172.21.15.98:0/149419900 >> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] conn(0x7f8dcc0a1150 0x7f8dcc0a3600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:29:07.226 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:07.221+0000 7f8de5f45700 1 --2- 172.21.15.98:0/149419900 >> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] conn(0x7f8dcc0a1150 0x7f8dcc0a3600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:29:07.226 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:07.221+0000 7f8dd6ffd700 1 -- 172.21.15.98:0/149419900 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 7979+0+0 (secure 0 0 0) 0x7f8dd80cea10 con 0x7f8de01017c0 2023-07-14T19:29:07.226 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:07.221+0000 7f8de5f45700 1 --2- 172.21.15.98:0/149419900 >> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] conn(0x7f8dcc0a1150 0x7f8dcc0a3600 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f8de0100c90 tx=0x7f8dd0009190 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:29:07.226 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:07.224+0000 7f8dd6ffd700 1 -- 172.21.15.98:0/149419900 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f8dd8098340 con 0x7f8de01017c0 2023-07-14T19:29:07.447 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:07.445+0000 7f8de81a9700 1 -- 172.21.15.98:0/149419900 --> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] -- mgr_command(tid 0: {"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_1", "group_name": "", "target": ["mon-mgr", ""]}) v1 -- 0x7f8de0060a80 con 0x7f8dcc0a1150 2023-07-14T19:29:07.470 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:07.468+0000 7f8dd6ffd700 1 -- 172.21.15.98:0/149419900 <== mgr.14152 v2:172.21.15.98:6800/452042164 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f8de0060a80 con 0x7f8dcc0a1150 2023-07-14T19:29:07.472 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:07.470+0000 7f8de81a9700 1 -- 172.21.15.98:0/149419900 >> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] conn(0x7f8dcc0a1150 msgr2=0x7f8dcc0a3600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:29:07.473 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:07.470+0000 7f8de81a9700 1 --2- 172.21.15.98:0/149419900 >> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] conn(0x7f8dcc0a1150 0x7f8dcc0a3600 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f8de0100c90 tx=0x7f8dd0009190 comp rx=0 tx=0).stop 2023-07-14T19:29:07.473 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:07.470+0000 7f8de81a9700 1 -- 172.21.15.98:0/149419900 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8de01017c0 msgr2=0x7f8de0072f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:29:07.474 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:07.470+0000 7f8de81a9700 1 --2- 172.21.15.98:0/149419900 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8de01017c0 0x7f8de0072f70 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f8dd800b830 tx=0x7f8dd8004c80 comp rx=0 tx=0).stop 2023-07-14T19:29:07.474 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:07.470+0000 7f8de81a9700 1 -- 172.21.15.98:0/149419900 shutdown_connections 2023-07-14T19:29:07.474 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:07.470+0000 7f8de81a9700 1 --2- 172.21.15.98:0/149419900 >> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] conn(0x7f8dcc0a1150 0x7f8dcc0a3600 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:29:07.475 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:07.470+0000 7f8de81a9700 1 --2- 172.21.15.98:0/149419900 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f8de00ffc40 0x7f8de00743f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:29:07.475 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:07.470+0000 7f8de81a9700 1 --2- 172.21.15.98:0/149419900 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8de01017c0 0x7f8de0072f70 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:29:07.475 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:07.470+0000 7f8de81a9700 1 --2- 172.21.15.98:0/149419900 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f8de0100e30 0x7f8de0072a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:29:07.475 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:07.470+0000 7f8de81a9700 1 -- 172.21.15.98:0/149419900 >> 172.21.15.98:0/149419900 conn(0x7f8de00fb260 msgr2=0x7f8de00fd430 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:29:07.475 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:07.470+0000 7f8de81a9700 1 -- 172.21.15.98:0/149419900 shutdown_connections 2023-07-14T19:29:07.476 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:07.470+0000 7f8de81a9700 1 -- 172.21.15.98:0/149419900 wait complete. 2023-07-14T19:29:07.484 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume getpath cephfs sv_1 2023-07-14T19:29:07.609 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:07.607+0000 7fc8fada2700 1 -- 172.21.15.98:0/2388842098 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc8f4100580 msgr2=0x7fc8f4102960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:29:07.609 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:07.607+0000 7fc8fada2700 1 --2- 172.21.15.98:0/2388842098 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc8f4100580 0x7fc8f4102960 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7fc8e8009990 tx=0x7fc8e8009ca0 comp rx=0 tx=0).stop 2023-07-14T19:29:07.609 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:07.607+0000 7fc8fada2700 1 -- 172.21.15.98:0/2388842098 shutdown_connections 2023-07-14T19:29:07.609 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:07.607+0000 7fc8fada2700 1 --2- 172.21.15.98:0/2388842098 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fc8f41057c0 0x7fc8f4107c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:29:07.610 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:07.607+0000 7fc8fada2700 1 --2- 172.21.15.98:0/2388842098 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc8f4100580 0x7fc8f4102960 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:29:07.610 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:07.607+0000 7fc8fada2700 1 --2- 172.21.15.98:0/2388842098 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7fc8f4102ea0 0x7fc8f4105280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:29:07.610 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:07.607+0000 7fc8fada2700 1 -- 172.21.15.98:0/2388842098 >> 172.21.15.98:0/2388842098 conn(0x7fc8f40fa000 msgr2=0x7fc8f40fc450 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:29:07.610 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:07.607+0000 7fc8fada2700 1 -- 172.21.15.98:0/2388842098 shutdown_connections 2023-07-14T19:29:07.610 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:07.607+0000 7fc8fada2700 1 -- 172.21.15.98:0/2388842098 wait complete. 2023-07-14T19:29:07.612 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:07.608+0000 7fc8fada2700 1 Processor -- start 2023-07-14T19:29:07.613 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:07.608+0000 7fc8fada2700 1 -- start start 2023-07-14T19:29:07.613 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:07.608+0000 7fc8fada2700 1 --2- >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7fc8f4100580 0x7fc8f4195670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:29:07.613 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:07.608+0000 7fc8fada2700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fc8f4102ea0 0x7fc8f4195bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:29:07.613 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:07.608+0000 7fc8fada2700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc8f41057c0 0x7fc8f419cc20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:29:07.613 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:07.608+0000 7fc8fada2700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc8f419d1f0 con 0x7fc8f41057c0 2023-07-14T19:29:07.614 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:07.608+0000 7fc8fada2700 1 -- --> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc8f419d330 con 0x7fc8f4100580 2023-07-14T19:29:07.614 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:07.608+0000 7fc8fada2700 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc8f419d4c0 con 0x7fc8f4102ea0 2023-07-14T19:29:07.614 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:07.608+0000 7fc8fa5a1700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc8f41057c0 0x7fc8f419cc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:29:07.615 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:07.608+0000 7fc8fa5a1700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc8f41057c0 0x7fc8f419cc20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:45816/0 (socket says 172.21.15.98:45816) 2023-07-14T19:29:07.615 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:07.608+0000 7fc8fa5a1700 1 -- 172.21.15.98:0/3909903296 learned_addr learned my addr 172.21.15.98:0/3909903296 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-07-14T19:29:07.615 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:07.608+0000 7fc8f959f700 1 --2- 172.21.15.98:0/3909903296 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fc8f4102ea0 0x7fc8f4195bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:29:07.615 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:07.609+0000 7fc8f9da0700 1 --2- 172.21.15.98:0/3909903296 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7fc8f4100580 0x7fc8f4195670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:29:07.616 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:07.609+0000 7fc8fa5a1700 1 -- 172.21.15.98:0/3909903296 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fc8f4102ea0 msgr2=0x7fc8f4195bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:29:07.616 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:07.609+0000 7fc8fa5a1700 1 --2- 172.21.15.98:0/3909903296 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fc8f4102ea0 0x7fc8f4195bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:29:07.616 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:07.609+0000 7fc8fa5a1700 1 -- 172.21.15.98:0/3909903296 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7fc8f4100580 msgr2=0x7fc8f4195670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:29:07.617 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:07.609+0000 7fc8fa5a1700 1 --2- 172.21.15.98:0/3909903296 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7fc8f4100580 0x7fc8f4195670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:29:07.617 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:07.609+0000 7fc8fa5a1700 1 -- 172.21.15.98:0/3909903296 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc8e80095f0 con 0x7fc8f41057c0 2023-07-14T19:29:07.617 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:07.609+0000 7fc8f9da0700 1 --2- 172.21.15.98:0/3909903296 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7fc8f4100580 0x7fc8f4195670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-07-14T19:29:07.617 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:07.609+0000 7fc8fa5a1700 1 --2- 172.21.15.98:0/3909903296 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc8f41057c0 0x7fc8f419cc20 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7fc8ec007ee0 tx=0x7fc8ec00d3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:29:07.618 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:07.609+0000 7fc8e6ffd700 1 -- 172.21.15.98:0/3909903296 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc8ec017070 con 0x7fc8f41057c0 2023-07-14T19:29:07.618 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:07.609+0000 7fc8fada2700 1 -- 172.21.15.98:0/3909903296 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc8f419d7a0 con 0x7fc8f41057c0 2023-07-14T19:29:07.618 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:07.609+0000 7fc8e6ffd700 1 -- 172.21.15.98:0/3909903296 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fc8ec00f040 con 0x7fc8f41057c0 2023-07-14T19:29:07.619 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:07.609+0000 7fc8e6ffd700 1 -- 172.21.15.98:0/3909903296 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc8ec005740 con 0x7fc8f41057c0 2023-07-14T19:29:07.619 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:07.609+0000 7fc8fada2700 1 -- 172.21.15.98:0/3909903296 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc8f419dc60 con 0x7fc8f41057c0 2023-07-14T19:29:07.619 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:07.610+0000 7fc8fada2700 1 -- 172.21.15.98:0/3909903296 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc8f40fbb70 con 0x7fc8f41057c0 2023-07-14T19:29:07.619 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:07.611+0000 7fc8e6ffd700 1 -- 172.21.15.98:0/3909903296 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 16) v1 ==== 135715+0+0 (secure 0 0 0) 0x7fc8ec00dc60 con 0x7fc8f41057c0 2023-07-14T19:29:07.620 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:07.611+0000 7fc8e6ffd700 1 --2- 172.21.15.98:0/3909903296 >> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] conn(0x7fc8e00a11f0 0x7fc8e00a36a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:29:07.621 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:07.611+0000 7fc8f9da0700 1 --2- 172.21.15.98:0/3909903296 >> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] conn(0x7fc8e00a11f0 0x7fc8e00a36a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:29:07.621 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:07.611+0000 7fc8e6ffd700 1 -- 172.21.15.98:0/3909903296 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 7979+0+0 (secure 0 0 0) 0x7fc8ec0cc000 con 0x7fc8f41057c0 2023-07-14T19:29:07.622 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:07.612+0000 7fc8f9da0700 1 --2- 172.21.15.98:0/3909903296 >> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] conn(0x7fc8e00a11f0 0x7fc8e00a36a0 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7fc8e800b2e0 tx=0x7fc8e8005e80 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:29:07.622 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:07.616+0000 7fc8e6ffd700 1 -- 172.21.15.98:0/3909903296 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7fc8ec0958a0 con 0x7fc8f41057c0 2023-07-14T19:29:07.681 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:07.680+0000 7fc8e6ffd700 1 -- 172.21.15.98:0/3909903296 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mgrmap(e 17) v1 ==== 135766+0+0 (secure 0 0 0) 0x7fc8ec095c70 con 0x7fc8f41057c0 2023-07-14T19:29:07.781 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:29:07 smithi098 ceph-mon[127486]: from='client.14742 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_0", "group_name": "", "target": ["mon-mgr", ""]}]: dispatch 2023-07-14T19:29:07.849 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:07.848+0000 7fc8fada2700 1 -- 172.21.15.98:0/3909903296 --> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] -- mgr_command(tid 0: {"prefix": "fs subvolume getpath", "vol_name": "cephfs", "sub_name": "sv_1", "target": ["mon-mgr", ""]}) v1 -- 0x7fc8f4060a80 con 0x7fc8e00a11f0 2023-07-14T19:29:07.853 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:07.852+0000 7fc8e6ffd700 1 -- 172.21.15.98:0/3909903296 <== mgr.14152 v2:172.21.15.98:6800/452042164 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+60 (secure 0 0 0) 0x7fc8f4060a80 con 0x7fc8e00a11f0 2023-07-14T19:29:07.854 INFO:teuthology.orchestra.run.smithi098.stdout:/volumes/_nogroup/sv_1/abb1cea8-1576-4a15-9246-a8da768b187f 2023-07-14T19:29:07.856 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:07.854+0000 7fc8fada2700 1 -- 172.21.15.98:0/3909903296 >> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] conn(0x7fc8e00a11f0 msgr2=0x7fc8e00a36a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:29:07.856 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:07.854+0000 7fc8fada2700 1 --2- 172.21.15.98:0/3909903296 >> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] conn(0x7fc8e00a11f0 0x7fc8e00a36a0 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7fc8e800b2e0 tx=0x7fc8e8005e80 comp rx=0 tx=0).stop 2023-07-14T19:29:07.857 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:07.854+0000 7fc8fada2700 1 -- 172.21.15.98:0/3909903296 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc8f41057c0 msgr2=0x7fc8f419cc20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:29:07.857 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:07.854+0000 7fc8fada2700 1 --2- 172.21.15.98:0/3909903296 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc8f41057c0 0x7fc8f419cc20 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7fc8ec007ee0 tx=0x7fc8ec00d3b0 comp rx=0 tx=0).stop 2023-07-14T19:29:07.857 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:07.854+0000 7fc8fada2700 1 -- 172.21.15.98:0/3909903296 shutdown_connections 2023-07-14T19:29:07.858 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:07.854+0000 7fc8fada2700 1 --2- 172.21.15.98:0/3909903296 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc8f41057c0 0x7fc8f419cc20 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:29:07.858 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:07.854+0000 7fc8fada2700 1 --2- 172.21.15.98:0/3909903296 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7fc8f4100580 0x7fc8f4195670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:29:07.858 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:07.854+0000 7fc8fada2700 1 --2- 172.21.15.98:0/3909903296 >> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] conn(0x7fc8e00a11f0 0x7fc8e00a36a0 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:29:07.859 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:07.854+0000 7fc8fada2700 1 --2- 172.21.15.98:0/3909903296 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fc8f4102ea0 0x7fc8f4195bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:29:07.859 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:07.854+0000 7fc8fada2700 1 -- 172.21.15.98:0/3909903296 >> 172.21.15.98:0/3909903296 conn(0x7fc8f40fa000 msgr2=0x7fc8f41011e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:29:07.859 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:07.854+0000 7fc8fada2700 1 -- 172.21.15.98:0/3909903296 shutdown_connections 2023-07-14T19:29:07.859 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:07.854+0000 7fc8fada2700 1 -- 172.21.15.98:0/3909903296 wait complete. 2023-07-14T19:29:07.868 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-07-14T19:29:07.941 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:29:07 smithi101 ceph-mon[128807]: from='client.14742 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_0", "group_name": "", "target": ["mon-mgr", ""]}]: dispatch 2023-07-14T19:29:07.970 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:29:07 smithi139 ceph-mon[129293]: from='client.14742 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_0", "group_name": "", "target": ["mon-mgr", ""]}]: dispatch 2023-07-14T19:29:07.991 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:07.989+0000 7fc93ac94700 1 -- 172.21.15.98:0/1633475040 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc9340ffc10 msgr2=0x7fc934100020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:29:07.992 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:07.989+0000 7fc93ac94700 1 --2- 172.21.15.98:0/1633475040 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc9340ffc10 0x7fc934100020 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7fc928009990 tx=0x7fc928009ca0 comp rx=0 tx=0).stop 2023-07-14T19:29:07.992 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:07.989+0000 7fc93ac94700 1 -- 172.21.15.98:0/1633475040 shutdown_connections 2023-07-14T19:29:07.992 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:07.989+0000 7fc93ac94700 1 --2- 172.21.15.98:0/1633475040 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fc934101790 0x7fc934108050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:29:07.992 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:07.989+0000 7fc93ac94700 1 --2- 172.21.15.98:0/1633475040 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc9340ffc10 0x7fc934100020 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:29:07.993 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:07.989+0000 7fc93ac94700 1 --2- 172.21.15.98:0/1633475040 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7fc934100e00 0x7fc934101250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:29:07.993 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:07.989+0000 7fc93ac94700 1 -- 172.21.15.98:0/1633475040 >> 172.21.15.98:0/1633475040 conn(0x7fc9340fb230 msgr2=0x7fc9340fd660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:29:07.993 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:07.989+0000 7fc93ac94700 1 -- 172.21.15.98:0/1633475040 shutdown_connections 2023-07-14T19:29:07.993 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:07.989+0000 7fc93ac94700 1 -- 172.21.15.98:0/1633475040 wait complete. 2023-07-14T19:29:07.993 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:07.990+0000 7fc93ac94700 1 Processor -- start 2023-07-14T19:29:07.993 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:07.990+0000 7fc93ac94700 1 -- start start 2023-07-14T19:29:07.993 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:07.990+0000 7fc93ac94700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc9340ffc10 0x7fc934195990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:29:07.994 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:07.990+0000 7fc93ac94700 1 --2- >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7fc934100e00 0x7fc934195ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:29:07.994 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:07.990+0000 7fc93ac94700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fc934101790 0x7fc93419cf40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:29:07.995 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:07.990+0000 7fc93ac94700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc93419d510 con 0x7fc9340ffc10 2023-07-14T19:29:07.995 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:07.990+0000 7fc93ac94700 1 -- --> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc93419d650 con 0x7fc934100e00 2023-07-14T19:29:07.995 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:07.990+0000 7fc93ac94700 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc93419d7e0 con 0x7fc934101790 2023-07-14T19:29:07.996 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:07.990+0000 7fc938a30700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc9340ffc10 0x7fc934195990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:29:07.996 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:07.990+0000 7fc938a30700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc9340ffc10 0x7fc934195990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:45826/0 (socket says 172.21.15.98:45826) 2023-07-14T19:29:07.996 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:07.990+0000 7fc938a30700 1 -- 172.21.15.98:0/4260159366 learned_addr learned my addr 172.21.15.98:0/4260159366 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-07-14T19:29:07.996 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:07.990+0000 7fc933fff700 1 --2- 172.21.15.98:0/4260159366 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7fc934100e00 0x7fc934195ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:29:07.996 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:07.990+0000 7fc939231700 1 --2- 172.21.15.98:0/4260159366 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fc934101790 0x7fc93419cf40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:29:07.997 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:07.990+0000 7fc938a30700 1 -- 172.21.15.98:0/4260159366 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fc934101790 msgr2=0x7fc93419cf40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:29:07.997 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:07.990+0000 7fc938a30700 1 --2- 172.21.15.98:0/4260159366 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fc934101790 0x7fc93419cf40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:29:07.997 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:07.990+0000 7fc938a30700 1 -- 172.21.15.98:0/4260159366 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7fc934100e00 msgr2=0x7fc934195ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:29:07.997 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:07.990+0000 7fc938a30700 1 --2- 172.21.15.98:0/4260159366 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7fc934100e00 0x7fc934195ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:29:07.997 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:07.990+0000 7fc938a30700 1 -- 172.21.15.98:0/4260159366 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc9280095f0 con 0x7fc9340ffc10 2023-07-14T19:29:07.997 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:07.990+0000 7fc933fff700 1 --2- 172.21.15.98:0/4260159366 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7fc934100e00 0x7fc934195ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-07-14T19:29:07.997 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:07.991+0000 7fc938a30700 1 --2- 172.21.15.98:0/4260159366 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc9340ffc10 0x7fc934195990 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7fc928000c00 tx=0x7fc92800b770 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:29:07.997 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:07.991+0000 7fc931ffb700 1 -- 172.21.15.98:0/4260159366 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc92801c660 con 0x7fc9340ffc10 2023-07-14T19:29:07.998 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:07.991+0000 7fc93ac94700 1 -- 172.21.15.98:0/4260159366 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc934072870 con 0x7fc9340ffc10 2023-07-14T19:29:07.998 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:07.991+0000 7fc931ffb700 1 -- 172.21.15.98:0/4260159366 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fc92800bed0 con 0x7fc9340ffc10 2023-07-14T19:29:07.998 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:07.991+0000 7fc931ffb700 1 -- 172.21.15.98:0/4260159366 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc928003af0 con 0x7fc9340ffc10 2023-07-14T19:29:07.998 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:07.991+0000 7fc93ac94700 1 -- 172.21.15.98:0/4260159366 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc934072d20 con 0x7fc9340ffc10 2023-07-14T19:29:07.998 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:07.992+0000 7fc93ac94700 1 -- 172.21.15.98:0/4260159366 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc934107150 con 0x7fc9340ffc10 2023-07-14T19:29:07.998 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:07.993+0000 7fc931ffb700 1 -- 172.21.15.98:0/4260159366 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 17) v1 ==== 135766+0+0 (secure 0 0 0) 0x7fc92800ba10 con 0x7fc9340ffc10 2023-07-14T19:29:07.998 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:07.993+0000 7fc931ffb700 1 --2- 172.21.15.98:0/4260159366 >> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] conn(0x7fc9240a1220 0x7fc9240a36d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:29:07.999 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:07.993+0000 7fc933fff700 1 --2- 172.21.15.98:0/4260159366 >> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] conn(0x7fc9240a1220 0x7fc9240a36d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:29:07.999 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:07.993+0000 7fc931ffb700 1 -- 172.21.15.98:0/4260159366 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 7979+0+0 (secure 0 0 0) 0x7fc9280ded90 con 0x7fc9340ffc10 2023-07-14T19:29:07.999 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:07.994+0000 7fc933fff700 1 --2- 172.21.15.98:0/4260159366 >> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] conn(0x7fc9240a1220 0x7fc9240a36d0 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7fc920005950 tx=0x7fc9200058e0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:29:07.999 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:07.997+0000 7fc931ffb700 1 -- 172.21.15.98:0/4260159366 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7fc9280a85c0 con 0x7fc9340ffc10 2023-07-14T19:29:08.228 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:08.226+0000 7fc93ac94700 1 -- 172.21.15.98:0/4260159366 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc93402c8e0 con 0x7fc9340ffc10 2023-07-14T19:29:08.228 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:08.227+0000 7fc931ffb700 1 -- 172.21.15.98:0/4260159366 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 22 v22) v1 ==== 94+0+10826 (secure 0 0 0) 0x7fc9280abbe0 con 0x7fc9340ffc10 2023-07-14T19:29:08.229 INFO:teuthology.orchestra.run.smithi098.stdout: 2023-07-14T19:29:08.229 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":22,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14565,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6835/586280675","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6834","nonce":586280675},{"type":"v1","addr":"172.21.15.98:6835","nonce":586280675}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":4},{"gid":14577,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6837/487577257","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6836","nonce":487577257},{"type":"v1","addr":"172.21.15.98:6837","nonce":487577257}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":5},{"gid":14634,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6839/4085731209","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6838","nonce":4085731209},{"type":"v1","addr":"172.21.15.98:6839","nonce":4085731209}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":11},{"gid":14646,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6841/1679318259","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6840","nonce":1679318259},{"type":"v1","addr":"172.21.15.98:6841","nonce":1679318259}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":12},{"gid":24421,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.139:6833/3505477820","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6832","nonce":3505477820},{"type":"v1","addr":"172.21.15.139:6833","nonce":3505477820}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":24437,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.101:6833/3474460171","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.101:6832","nonce":3474460171},{"type":"v1","addr":"172.21.15.101:6833","nonce":3474460171}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":22,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2023-07-14T19:29:01.033156+0000","modified":"2023-07-14T19:29:06.291916+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24521,"mds_1":24493,"mds_2":24479},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24455":{"gid":24455,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.101:6835/1583264648","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.101:6834","nonce":1583264648},{"type":"v1","addr":"172.21.15.101:6835","nonce":1583264648}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24460":{"gid":24460,"name":"f","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.139:6835/2315281724","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6834","nonce":2315281724},{"type":"v1","addr":"172.21.15.139:6835","nonce":2315281724}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24469":{"gid":24469,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.139:6837/3597660552","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6836","nonce":3597660552},{"type":"v1","addr":"172.21.15.139:6837","nonce":3597660552}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24479":{"gid":24479,"name":"h","rank":2,"incarnation":21,"state":"up:active","state_seq":6,"addr":"172.21.15.101:6837/485741528","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.101:6836","nonce":485741528},{"type":"v1","addr":"172.21.15.101:6837","nonce":485741528}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24493":{"gid":24493,"name":"i","rank":1,"incarnation":19,"state":"up:active","state_seq":5,"addr":"172.21.15.139:6839/1258854143","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6838","nonce":1258854143},{"type":"v1","addr":"172.21.15.139:6839","nonce":1258854143}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24521":{"gid":24521,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.101:6839/48393381","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.101:6838","nonce":48393381},{"type":"v1","addr":"172.21.15.101:6839","nonce":48393381}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-07-14T19:29:08.231 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:08.229+0000 7fc93ac94700 1 -- 172.21.15.98:0/4260159366 >> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] conn(0x7fc9240a1220 msgr2=0x7fc9240a36d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:29:08.231 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:08.229+0000 7fc93ac94700 1 --2- 172.21.15.98:0/4260159366 >> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] conn(0x7fc9240a1220 0x7fc9240a36d0 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7fc920005950 tx=0x7fc9200058e0 comp rx=0 tx=0).stop 2023-07-14T19:29:08.231 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:08.229+0000 7fc93ac94700 1 -- 172.21.15.98:0/4260159366 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc9340ffc10 msgr2=0x7fc934195990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:29:08.231 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:08.229+0000 7fc93ac94700 1 --2- 172.21.15.98:0/4260159366 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc9340ffc10 0x7fc934195990 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7fc928000c00 tx=0x7fc92800b770 comp rx=0 tx=0).stop 2023-07-14T19:29:08.231 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:08.229+0000 7fc93ac94700 1 -- 172.21.15.98:0/4260159366 shutdown_connections 2023-07-14T19:29:08.231 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:08.229+0000 7fc93ac94700 1 --2- 172.21.15.98:0/4260159366 >> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] conn(0x7fc9240a1220 0x7fc9240a36d0 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:29:08.232 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:08.229+0000 7fc93ac94700 1 --2- 172.21.15.98:0/4260159366 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fc934101790 0x7fc93419cf40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:29:08.232 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:08.229+0000 7fc93ac94700 1 --2- 172.21.15.98:0/4260159366 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc9340ffc10 0x7fc934195990 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:29:08.232 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:08.229+0000 7fc93ac94700 1 --2- 172.21.15.98:0/4260159366 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7fc934100e00 0x7fc934195ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:29:08.232 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:08.229+0000 7fc93ac94700 1 -- 172.21.15.98:0/4260159366 >> 172.21.15.98:0/4260159366 conn(0x7fc9340fb230 msgr2=0x7fc934104290 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:29:08.232 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:08.229+0000 7fc93ac94700 1 -- 172.21.15.98:0/4260159366 shutdown_connections 2023-07-14T19:29:08.232 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:08.229+0000 7fc93ac94700 1 -- 172.21.15.98:0/4260159366 wait complete. 2023-07-14T19:29:08.233 INFO:teuthology.orchestra.run.smithi098.stderr:dumped fsmap epoch 22 2023-07-14T19:29:08.243 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-07-14T19:29:08.364 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:08.362+0000 7faeb59fb700 1 -- 172.21.15.98:0/3971373846 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7faeb00ff820 msgr2=0x7faeb01070f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:29:08.364 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:08.362+0000 7faeb59fb700 1 --2- 172.21.15.98:0/3971373846 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7faeb00ff820 0x7faeb01070f0 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7faea800b0a0 tx=0x7faea800b3b0 comp rx=0 tx=0).stop 2023-07-14T19:29:08.365 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:08.363+0000 7faeb59fb700 1 -- 172.21.15.98:0/3971373846 shutdown_connections 2023-07-14T19:29:08.365 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:08.363+0000 7faeb59fb700 1 --2- 172.21.15.98:0/3971373846 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7faeb00ff820 0x7faeb01070f0 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:29:08.365 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:08.363+0000 7faeb59fb700 1 --2- 172.21.15.98:0/3971373846 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7faeb0100160 0x7faeb0100570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:29:08.365 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:08.363+0000 7faeb59fb700 1 --2- 172.21.15.98:0/3971373846 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7faeb00fee90 0x7faeb00ff2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:29:08.365 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:08.363+0000 7faeb59fb700 1 -- 172.21.15.98:0/3971373846 >> 172.21.15.98:0/3971373846 conn(0x7faeb00f9990 msgr2=0x7faeb00fbda0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:29:08.365 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:08.363+0000 7faeb59fb700 1 -- 172.21.15.98:0/3971373846 shutdown_connections 2023-07-14T19:29:08.365 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:08.363+0000 7faeb59fb700 1 -- 172.21.15.98:0/3971373846 wait complete. 2023-07-14T19:29:08.367 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:08.363+0000 7faeb59fb700 1 Processor -- start 2023-07-14T19:29:08.367 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:08.363+0000 7faeb59fb700 1 -- start start 2023-07-14T19:29:08.368 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:08.363+0000 7faeb59fb700 1 --2- >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7faeb00fee90 0x7faeb00fe910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:29:08.368 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:08.363+0000 7faeb59fb700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7faeb00ff820 0x7faeb019a6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:29:08.368 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:08.364+0000 7faeb59fb700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7faeb0100160 0x7faeb019cd30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:29:08.368 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:08.364+0000 7faeb59fb700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faeb019d500 con 0x7faeb00ff820 2023-07-14T19:29:08.368 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:08.364+0000 7faeb59fb700 1 -- --> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faeb019d670 con 0x7faeb00fee90 2023-07-14T19:29:08.368 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:08.364+0000 7faeb59fb700 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faeb019d830 con 0x7faeb0100160 2023-07-14T19:29:08.368 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:08.364+0000 7faeae7fc700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7faeb00ff820 0x7faeb019a6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:29:08.369 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:08.364+0000 7faeae7fc700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7faeb00ff820 0x7faeb019a6e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:45840/0 (socket says 172.21.15.98:45840) 2023-07-14T19:29:08.369 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:08.364+0000 7faeae7fc700 1 -- 172.21.15.98:0/2261216108 learned_addr learned my addr 172.21.15.98:0/2261216108 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-07-14T19:29:08.369 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:08.364+0000 7faeaf7fe700 1 --2- 172.21.15.98:0/2261216108 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7faeb0100160 0x7faeb019cd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:29:08.369 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:08.364+0000 7faeaeffd700 1 --2- 172.21.15.98:0/2261216108 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7faeb00fee90 0x7faeb00fe910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:29:08.369 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:08.364+0000 7faeae7fc700 1 -- 172.21.15.98:0/2261216108 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7faeb0100160 msgr2=0x7faeb019cd30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:29:08.369 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:08.364+0000 7faeae7fc700 1 --2- 172.21.15.98:0/2261216108 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7faeb0100160 0x7faeb019cd30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:29:08.369 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:08.364+0000 7faeae7fc700 1 -- 172.21.15.98:0/2261216108 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7faeb00fee90 msgr2=0x7faeb00fe910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:29:08.369 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:08.364+0000 7faeae7fc700 1 --2- 172.21.15.98:0/2261216108 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7faeb00fee90 0x7faeb00fe910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:29:08.370 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:08.364+0000 7faeae7fc700 1 -- 172.21.15.98:0/2261216108 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faea8009d00 con 0x7faeb00ff820 2023-07-14T19:29:08.370 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:08.364+0000 7faeaeffd700 1 --2- 172.21.15.98:0/2261216108 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7faeb00fee90 0x7faeb00fe910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-07-14T19:29:08.370 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:08.364+0000 7faeae7fc700 1 --2- 172.21.15.98:0/2261216108 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7faeb00ff820 0x7faeb019a6e0 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7faea000e9a0 tx=0x7faea000ecb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:29:08.370 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:08.364+0000 7faeb49f9700 1 -- 172.21.15.98:0/2261216108 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faea0009790 con 0x7faeb00ff820 2023-07-14T19:29:08.370 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:08.364+0000 7faeb49f9700 1 -- 172.21.15.98:0/2261216108 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7faea000cbd0 con 0x7faeb00ff820 2023-07-14T19:29:08.370 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:08.364+0000 7faeb59fb700 1 -- 172.21.15.98:0/2261216108 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faeb019db10 con 0x7faeb00ff820 2023-07-14T19:29:08.370 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:08.364+0000 7faeb49f9700 1 -- 172.21.15.98:0/2261216108 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faea0018680 con 0x7faeb00ff820 2023-07-14T19:29:08.371 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:08.364+0000 7faeb59fb700 1 -- 172.21.15.98:0/2261216108 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faeb019e020 con 0x7faeb00ff820 2023-07-14T19:29:08.371 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:08.365+0000 7faeb59fb700 1 -- 172.21.15.98:0/2261216108 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faeb010a3a0 con 0x7faeb00ff820 2023-07-14T19:29:08.371 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:08.366+0000 7faeb49f9700 1 -- 172.21.15.98:0/2261216108 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 17) v1 ==== 135766+0+0 (secure 0 0 0) 0x7faea000cd40 con 0x7faeb00ff820 2023-07-14T19:29:08.371 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:08.367+0000 7faeb49f9700 1 --2- 172.21.15.98:0/2261216108 >> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] conn(0x7fae980a1220 0x7fae980a36d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:29:08.372 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:08.367+0000 7faeb49f9700 1 -- 172.21.15.98:0/2261216108 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 7979+0+0 (secure 0 0 0) 0x7faea0014070 con 0x7faeb00ff820 2023-07-14T19:29:08.372 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:08.367+0000 7faeaeffd700 1 --2- 172.21.15.98:0/2261216108 >> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] conn(0x7fae980a1220 0x7fae980a36d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:29:08.372 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:08.367+0000 7faeaeffd700 1 --2- 172.21.15.98:0/2261216108 >> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] conn(0x7fae980a1220 0x7fae980a36d0 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7faea4005fd0 tx=0x7faea4005e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:29:08.372 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:08.371+0000 7faeb49f9700 1 -- 172.21.15.98:0/2261216108 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7faea0051780 con 0x7faeb00ff820 2023-07-14T19:29:08.565 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:08.561+0000 7faeb59fb700 1 -- 172.21.15.98:0/2261216108 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7faeb010a5c0 con 0x7faeb00ff820 2023-07-14T19:29:08.565 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:08.563+0000 7faeb49f9700 1 -- 172.21.15.98:0/2261216108 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v83) v1 ==== 74+0+22117 (secure 0 0 0) 0x7faea0096330 con 0x7faeb00ff820 2023-07-14T19:29:08.565 INFO:teuthology.orchestra.run.smithi098.stdout: 2023-07-14T19:29:08.566 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":83,"fsid":"9adea648-227b-11ee-9b33-001a4aab830c","created":"2023-07-14T19:22:07.178854+0000","modified":"2023-07-14T19:29:02.293099+0000","last_up_change":"2023-07-14T19:28:19.646827+0000","last_in_change":"2023-07-14T19:28:02.018325+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-07-14T19:25:09.845827+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2023-07-14T19:28:58.952921+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"82","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2023-07-14T19:29:00.228272+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"83","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"a3797e89-1754-41cf-9517-227e06ffdd72","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6802","nonce":120507228},{"type":"v1","addr":"172.21.15.98:6803","nonce":120507228}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6804","nonce":120507228},{"type":"v1","addr":"172.21.15.98:6805","nonce":120507228}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6808","nonce":120507228},{"type":"v1","addr":"172.21.15.98:6809","nonce":120507228}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6806","nonce":120507228},{"type":"v1","addr":"172.21.15.98:6807","nonce":120507228}]},"public_addr":"172.21.15.98:6803/120507228","cluster_addr":"172.21.15.98:6805/120507228","heartbeat_back_addr":"172.21.15.98:6809/120507228","heartbeat_front_addr":"172.21.15.98:6807/120507228","state":["exists","up"]},{"osd":1,"uuid":"2b001add-1aaa-4496-a403-7ef9725b80a0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.101:6800","nonce":2450584584},{"type":"v1","addr":"172.21.15.101:6801","nonce":2450584584}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.101:6802","nonce":2450584584},{"type":"v1","addr":"172.21.15.101:6803","nonce":2450584584}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.101:6806","nonce":2450584584},{"type":"v1","addr":"172.21.15.101:6807","nonce":2450584584}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.101:6804","nonce":2450584584},{"type":"v1","addr":"172.21.15.101:6805","nonce":2450584584}]},"public_addr":"172.21.15.101:6801/2450584584","cluster_addr":"172.21.15.101:6803/2450584584","heartbeat_back_addr":"172.21.15.101:6807/2450584584","heartbeat_front_addr":"172.21.15.101:6805/2450584584","state":["exists","up"]},{"osd":2,"uuid":"48e6d499-3ca9-424b-b0fb-3ba5f33b9fc9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":945038995},{"type":"v1","addr":"172.21.15.139:6801","nonce":945038995}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":945038995},{"type":"v1","addr":"172.21.15.139:6803","nonce":945038995}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":945038995},{"type":"v1","addr":"172.21.15.139:6807","nonce":945038995}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":945038995},{"type":"v1","addr":"172.21.15.139:6805","nonce":945038995}]},"public_addr":"172.21.15.139:6801/945038995","cluster_addr":"172.21.15.139:6803/945038995","heartbeat_back_addr":"172.21.15.139:6807/945038995","heartbeat_front_addr":"172.21.15.139:6805/945038995","state":["exists","up"]},{"osd":3,"uuid":"5530b265-31a5-447b-a1ea-17ebae9cfddf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6810","nonce":3095249665},{"type":"v1","addr":"172.21.15.98:6811","nonce":3095249665}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6812","nonce":3095249665},{"type":"v1","addr":"172.21.15.98:6813","nonce":3095249665}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6816","nonce":3095249665},{"type":"v1","addr":"172.21.15.98:6817","nonce":3095249665}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6814","nonce":3095249665},{"type":"v1","addr":"172.21.15.98:6815","nonce":3095249665}]},"public_addr":"172.21.15.98:6811/3095249665","cluster_addr":"172.21.15.98:6813/3095249665","heartbeat_back_addr":"172.21.15.98:6817/3095249665","heartbeat_front_addr":"172.21.15.98:6815/3095249665","state":["exists","up"]},{"osd":4,"uuid":"615d4642-a997-4f95-9e38-9fbdaa831400","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":32,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.101:6808","nonce":2844761406},{"type":"v1","addr":"172.21.15.101:6809","nonce":2844761406}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.101:6810","nonce":2844761406},{"type":"v1","addr":"172.21.15.101:6811","nonce":2844761406}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.101:6814","nonce":2844761406},{"type":"v1","addr":"172.21.15.101:6815","nonce":2844761406}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.101:6812","nonce":2844761406},{"type":"v1","addr":"172.21.15.101:6813","nonce":2844761406}]},"public_addr":"172.21.15.101:6809/2844761406","cluster_addr":"172.21.15.101:6811/2844761406","heartbeat_back_addr":"172.21.15.101:6815/2844761406","heartbeat_front_addr":"172.21.15.101:6813/2844761406","state":["exists","up"]},{"osd":5,"uuid":"415de5fa-607a-45a8-b732-28daf3be6dd3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":37,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":3532263042},{"type":"v1","addr":"172.21.15.139:6809","nonce":3532263042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6810","nonce":3532263042},{"type":"v1","addr":"172.21.15.139:6811","nonce":3532263042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6814","nonce":3532263042},{"type":"v1","addr":"172.21.15.139:6815","nonce":3532263042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":3532263042},{"type":"v1","addr":"172.21.15.139:6813","nonce":3532263042}]},"public_addr":"172.21.15.139:6809/3532263042","cluster_addr":"172.21.15.139:6811/3532263042","heartbeat_back_addr":"172.21.15.139:6815/3532263042","heartbeat_front_addr":"172.21.15.139:6813/3532263042","state":["exists","up"]},{"osd":6,"uuid":"d7f9d4ba-c309-4b75-a836-5a36384b528e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":45,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6818","nonce":658948743},{"type":"v1","addr":"172.21.15.98:6819","nonce":658948743}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6820","nonce":658948743},{"type":"v1","addr":"172.21.15.98:6821","nonce":658948743}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6824","nonce":658948743},{"type":"v1","addr":"172.21.15.98:6825","nonce":658948743}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6822","nonce":658948743},{"type":"v1","addr":"172.21.15.98:6823","nonce":658948743}]},"public_addr":"172.21.15.98:6819/658948743","cluster_addr":"172.21.15.98:6821/658948743","heartbeat_back_addr":"172.21.15.98:6825/658948743","heartbeat_front_addr":"172.21.15.98:6823/658948743","state":["exists","up"]},{"osd":7,"uuid":"eedb94d9-c5cd-4a93-8a8c-2648804b7195","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":50,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.101:6816","nonce":2881214066},{"type":"v1","addr":"172.21.15.101:6817","nonce":2881214066}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.101:6818","nonce":2881214066},{"type":"v1","addr":"172.21.15.101:6819","nonce":2881214066}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.101:6822","nonce":2881214066},{"type":"v1","addr":"172.21.15.101:6823","nonce":2881214066}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.101:6820","nonce":2881214066},{"type":"v1","addr":"172.21.15.101:6821","nonce":2881214066}]},"public_addr":"172.21.15.101:6817/2881214066","cluster_addr":"172.21.15.101:6819/2881214066","heartbeat_back_addr":"172.21.15.101:6823/2881214066","heartbeat_front_addr":"172.21.15.101:6821/2881214066","state":["exists","up"]},{"osd":8,"uuid":"6bfd77c2-5770-466e-aa6f-84b50c74ffb6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":57,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":3669514524},{"type":"v1","addr":"172.21.15.139:6817","nonce":3669514524}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":3669514524},{"type":"v1","addr":"172.21.15.139:6819","nonce":3669514524}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":3669514524},{"type":"v1","addr":"172.21.15.139:6823","nonce":3669514524}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6820","nonce":3669514524},{"type":"v1","addr":"172.21.15.139:6821","nonce":3669514524}]},"public_addr":"172.21.15.139:6817/3669514524","cluster_addr":"172.21.15.139:6819/3669514524","heartbeat_back_addr":"172.21.15.139:6823/3669514524","heartbeat_front_addr":"172.21.15.139:6821/3669514524","state":["exists","up"]},{"osd":9,"uuid":"19c5b54a-2242-4b23-b721-63be9559653d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":65,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6826","nonce":2893994080},{"type":"v1","addr":"172.21.15.98:6827","nonce":2893994080}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6828","nonce":2893994080},{"type":"v1","addr":"172.21.15.98:6829","nonce":2893994080}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6832","nonce":2893994080},{"type":"v1","addr":"172.21.15.98:6833","nonce":2893994080}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6830","nonce":2893994080},{"type":"v1","addr":"172.21.15.98:6831","nonce":2893994080}]},"public_addr":"172.21.15.98:6827/2893994080","cluster_addr":"172.21.15.98:6829/2893994080","heartbeat_back_addr":"172.21.15.98:6833/2893994080","heartbeat_front_addr":"172.21.15.98:6831/2893994080","state":["exists","up"]},{"osd":10,"uuid":"326f0209-ba9d-4b59-97cc-d61a758a8f29","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":70,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.101:6824","nonce":3545244225},{"type":"v1","addr":"172.21.15.101:6825","nonce":3545244225}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.101:6826","nonce":3545244225},{"type":"v1","addr":"172.21.15.101:6827","nonce":3545244225}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.101:6830","nonce":3545244225},{"type":"v1","addr":"172.21.15.101:6831","nonce":3545244225}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.101:6828","nonce":3545244225},{"type":"v1","addr":"172.21.15.101:6829","nonce":3545244225}]},"public_addr":"172.21.15.101:6825/3545244225","cluster_addr":"172.21.15.101:6827/3545244225","heartbeat_back_addr":"172.21.15.101:6831/3545244225","heartbeat_front_addr":"172.21.15.101:6829/3545244225","state":["exists","up"]},{"osd":11,"uuid":"c61c0320-1057-4f17-9bf1-7ad306ed7ed3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":78,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":2630023454},{"type":"v1","addr":"172.21.15.139:6825","nonce":2630023454}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6826","nonce":2630023454},{"type":"v1","addr":"172.21.15.139:6827","nonce":2630023454}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":2630023454},{"type":"v1","addr":"172.21.15.139:6831","nonce":2630023454}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":2630023454},{"type":"v1","addr":"172.21.15.139:6829","nonce":2630023454}]},"public_addr":"172.21.15.139:6825/2630023454","cluster_addr":"172.21.15.139:6827/2630023454","heartbeat_back_addr":"172.21.15.139:6831/2630023454","heartbeat_front_addr":"172.21.15.139:6829/2630023454","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T19:24:23.620028+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T19:24:45.779846+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T19:25:06.877949+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T19:25:27.994199+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T19:25:49.461139+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T19:26:10.409282+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T19:26:32.116390+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T19:26:53.536474+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T19:27:14.384839+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T19:27:35.635921+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T19:27:56.858744+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T19:28:18.342519+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.98:0/2164590253":"2023-07-15T19:23:01.520302+0000","172.21.15.98:0/2094734896":"2023-07-15T19:23:01.520302+0000","172.21.15.98:0/784745009":"2023-07-15T19:22:39.481502+0000","172.21.15.98:0/3657190818":"2023-07-15T19:22:39.481502+0000","172.21.15.98:6800/1732769087":"2023-07-15T19:23:01.520302+0000","172.21.15.98:0/3393542419":"2023-07-15T19:22:39.481502+0000","172.21.15.98:6800/749234665":"2023-07-15T19:22:39.481502+0000","172.21.15.98:6801/1732769087":"2023-07-15T19:23:01.520302+0000","172.21.15.98:6801/749234665":"2023-07-15T19:22:39.481502+0000","172.21.15.98:0/187822873":"2023-07-15T19:23:01.520302+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-07-14T19:29:08.567 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:08.564+0000 7faeb59fb700 1 -- 172.21.15.98:0/2261216108 >> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] conn(0x7fae980a1220 msgr2=0x7fae980a36d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:29:08.567 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:08.565+0000 7faeb59fb700 1 --2- 172.21.15.98:0/2261216108 >> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] conn(0x7fae980a1220 0x7fae980a36d0 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7faea4005fd0 tx=0x7faea4005e40 comp rx=0 tx=0).stop 2023-07-14T19:29:08.567 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:08.565+0000 7faeb59fb700 1 -- 172.21.15.98:0/2261216108 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7faeb00ff820 msgr2=0x7faeb019a6e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:29:08.567 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:08.565+0000 7faeb59fb700 1 --2- 172.21.15.98:0/2261216108 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7faeb00ff820 0x7faeb019a6e0 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7faea000e9a0 tx=0x7faea000ecb0 comp rx=0 tx=0).stop 2023-07-14T19:29:08.568 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:08.565+0000 7faeb59fb700 1 -- 172.21.15.98:0/2261216108 shutdown_connections 2023-07-14T19:29:08.568 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:08.565+0000 7faeb59fb700 1 --2- 172.21.15.98:0/2261216108 >> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] conn(0x7fae980a1220 0x7fae980a36d0 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:29:08.568 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:08.565+0000 7faeb59fb700 1 --2- 172.21.15.98:0/2261216108 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7faeb0100160 0x7faeb019cd30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:29:08.568 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:08.565+0000 7faeb59fb700 1 --2- 172.21.15.98:0/2261216108 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7faeb00fee90 0x7faeb00fe910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:29:08.568 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:08.565+0000 7faeb59fb700 1 --2- 172.21.15.98:0/2261216108 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7faeb00ff820 0x7faeb019a6e0 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:29:08.568 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:08.565+0000 7faeb59fb700 1 -- 172.21.15.98:0/2261216108 >> 172.21.15.98:0/2261216108 conn(0x7faeb00f9990 msgr2=0x7faeb00fba80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:29:08.568 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:08.565+0000 7faeb59fb700 1 -- 172.21.15.98:0/2261216108 shutdown_connections 2023-07-14T19:29:08.569 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:08.565+0000 7faeb59fb700 1 -- 172.21.15.98:0/2261216108 wait complete. 2023-07-14T19:29:08.579 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2023-07-14T19:29:08.579 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-07-14T19:29:08.704 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:08.700+0000 7f9afdcc0700 1 -- 172.21.15.98:0/1788995258 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9af80ffd10 msgr2=0x7f9af8100160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:29:08.704 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:08.700+0000 7f9afdcc0700 1 --2- 172.21.15.98:0/1788995258 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9af80ffd10 0x7f9af8100160 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f9aec009990 tx=0x7f9aec009ca0 comp rx=0 tx=0).stop 2023-07-14T19:29:08.704 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:08.700+0000 7f9afdcc0700 1 -- 172.21.15.98:0/1788995258 shutdown_connections 2023-07-14T19:29:08.705 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:08.700+0000 7f9afdcc0700 1 --2- 172.21.15.98:0/1788995258 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f9af81006a0 0x7f9af8106f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:29:08.705 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:08.700+0000 7f9afdcc0700 1 --2- 172.21.15.98:0/1788995258 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f9af80feb20 0x7f9af80fef30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:29:08.705 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:08.700+0000 7f9afdcc0700 1 --2- 172.21.15.98:0/1788995258 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9af80ffd10 0x7f9af8100160 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:29:08.705 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:08.700+0000 7f9afdcc0700 1 -- 172.21.15.98:0/1788995258 >> 172.21.15.98:0/1788995258 conn(0x7f9af80fa100 msgr2=0x7f9af80fc570 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:29:08.705 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:08.700+0000 7f9afdcc0700 1 -- 172.21.15.98:0/1788995258 shutdown_connections 2023-07-14T19:29:08.706 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:08.700+0000 7f9afdcc0700 1 -- 172.21.15.98:0/1788995258 wait complete. 2023-07-14T19:29:08.706 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:08.701+0000 7f9afdcc0700 1 Processor -- start 2023-07-14T19:29:08.706 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:08.701+0000 7f9afdcc0700 1 -- start start 2023-07-14T19:29:08.706 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:08.701+0000 7f9afdcc0700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9af80feb20 0x7f9af8069360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:29:08.706 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:08.701+0000 7f9afdcc0700 1 --2- >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f9af80ffd10 0x7f9af819a5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:29:08.706 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:08.701+0000 7f9afdcc0700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f9af81006a0 0x7f9af819cc40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:29:08.706 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:08.701+0000 7f9afdcc0700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9af819d410 con 0x7f9af80feb20 2023-07-14T19:29:08.707 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:08.701+0000 7f9afdcc0700 1 -- --> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9af819d580 con 0x7f9af80ffd10 2023-07-14T19:29:08.707 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:08.701+0000 7f9afdcc0700 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9af819d740 con 0x7f9af81006a0 2023-07-14T19:29:08.707 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:08.701+0000 7f9af77fe700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9af80feb20 0x7f9af8069360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:29:08.707 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:08.702+0000 7f9af77fe700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9af80feb20 0x7f9af8069360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:45850/0 (socket says 172.21.15.98:45850) 2023-07-14T19:29:08.707 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:08.702+0000 7f9af77fe700 1 -- 172.21.15.98:0/1702620960 learned_addr learned my addr 172.21.15.98:0/1702620960 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-07-14T19:29:08.708 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:08.702+0000 7f9af6ffd700 1 --2- 172.21.15.98:0/1702620960 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f9af80ffd10 0x7f9af819a5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:29:08.708 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:08.702+0000 7f9af7fff700 1 --2- 172.21.15.98:0/1702620960 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f9af81006a0 0x7f9af819cc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:29:08.708 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:08.702+0000 7f9af77fe700 1 -- 172.21.15.98:0/1702620960 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f9af81006a0 msgr2=0x7f9af819cc40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:29:08.708 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:08.702+0000 7f9af77fe700 1 --2- 172.21.15.98:0/1702620960 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f9af81006a0 0x7f9af819cc40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:29:08.708 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:08.702+0000 7f9af77fe700 1 -- 172.21.15.98:0/1702620960 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f9af80ffd10 msgr2=0x7f9af819a5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:29:08.709 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:08.702+0000 7f9af77fe700 1 --2- 172.21.15.98:0/1702620960 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f9af80ffd10 0x7f9af819a5f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:29:08.709 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:08.702+0000 7f9af77fe700 1 -- 172.21.15.98:0/1702620960 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9aec0095f0 con 0x7f9af80feb20 2023-07-14T19:29:08.709 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:08.702+0000 7f9af77fe700 1 --2- 172.21.15.98:0/1702620960 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9af80feb20 0x7f9af8069360 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f9ae800c990 tx=0x7f9ae800cca0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:29:08.709 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:08.702+0000 7f9af4ff9700 1 -- 172.21.15.98:0/1702620960 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ae8007940 con 0x7f9af80feb20 2023-07-14T19:29:08.709 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:08.702+0000 7f9afdcc0700 1 -- 172.21.15.98:0/1702620960 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9af819da20 con 0x7f9af80feb20 2023-07-14T19:29:08.709 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:08.702+0000 7f9af4ff9700 1 -- 172.21.15.98:0/1702620960 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f9ae800f450 con 0x7f9af80feb20 2023-07-14T19:29:08.710 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:08.702+0000 7f9af4ff9700 1 -- 172.21.15.98:0/1702620960 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ae8018660 con 0x7f9af80feb20 2023-07-14T19:29:08.710 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:08.702+0000 7f9afdcc0700 1 -- 172.21.15.98:0/1702620960 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9af819df30 con 0x7f9af80feb20 2023-07-14T19:29:08.711 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:08.703+0000 7f9afdcc0700 1 -- 172.21.15.98:0/1702620960 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9af8105910 con 0x7f9af80feb20 2023-07-14T19:29:08.712 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:08.704+0000 7f9af4ff9700 1 -- 172.21.15.98:0/1702620960 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 17) v1 ==== 135766+0+0 (secure 0 0 0) 0x7f9ae800f8c0 con 0x7f9af80feb20 2023-07-14T19:29:08.712 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:08.704+0000 7f9af4ff9700 1 --2- 172.21.15.98:0/1702620960 >> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] conn(0x7f9ae00a1220 0x7f9ae00a36d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:29:08.712 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:08.705+0000 7f9af6ffd700 1 --2- 172.21.15.98:0/1702620960 >> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] conn(0x7f9ae00a1220 0x7f9ae00a36d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:29:08.713 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:08.705+0000 7f9af4ff9700 1 -- 172.21.15.98:0/1702620960 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 7979+0+0 (secure 0 0 0) 0x7f9ae80cc920 con 0x7f9af80feb20 2023-07-14T19:29:08.713 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:08.705+0000 7f9af6ffd700 1 --2- 172.21.15.98:0/1702620960 >> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] conn(0x7f9ae00a1220 0x7f9ae00a36d0 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f9af80690d0 tx=0x7f9aec005fb0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:29:08.713 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:08.708+0000 7f9af4ff9700 1 -- 172.21.15.98:0/1702620960 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f9ae8096050 con 0x7f9af80feb20 2023-07-14T19:29:08.896 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:08.894+0000 7f9afdcc0700 1 -- 172.21.15.98:0/1702620960 --> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f9af8060a80 con 0x7f9ae00a1220 2023-07-14T19:29:08.906 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:08.904+0000 7f9af4ff9700 1 -- 172.21.15.98:0/1702620960 <== mgr.14152 v2:172.21.15.98:6800/452042164 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+336085 (secure 0 0 0) 0x7f9af8060a80 con 0x7f9ae00a1220 2023-07-14T19:29:08.912 INFO:teuthology.orchestra.run.smithi098.stdout: 2023-07-14T19:29:08.916 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:08.908+0000 7f9afdcc0700 1 -- 172.21.15.98:0/1702620960 >> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] conn(0x7f9ae00a1220 msgr2=0x7f9ae00a36d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:29:08.917 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:08.908+0000 7f9afdcc0700 1 --2- 172.21.15.98:0/1702620960 >> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] conn(0x7f9ae00a1220 0x7f9ae00a36d0 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f9af80690d0 tx=0x7f9aec005fb0 comp rx=0 tx=0).stop 2023-07-14T19:29:08.917 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:08.908+0000 7f9afdcc0700 1 -- 172.21.15.98:0/1702620960 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9af80feb20 msgr2=0x7f9af8069360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:29:08.917 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:08.908+0000 7f9afdcc0700 1 --2- 172.21.15.98:0/1702620960 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9af80feb20 0x7f9af8069360 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f9ae800c990 tx=0x7f9ae800cca0 comp rx=0 tx=0).stop 2023-07-14T19:29:08.917 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:08.908+0000 7f9afdcc0700 1 -- 172.21.15.98:0/1702620960 shutdown_connections 2023-07-14T19:29:08.917 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:08.908+0000 7f9afdcc0700 1 --2- 172.21.15.98:0/1702620960 >> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] conn(0x7f9ae00a1220 0x7f9ae00a36d0 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:29:08.917 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:08.908+0000 7f9afdcc0700 1 --2- 172.21.15.98:0/1702620960 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f9af81006a0 0x7f9af819cc40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:29:08.918 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:08.908+0000 7f9afdcc0700 1 --2- 172.21.15.98:0/1702620960 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9af80feb20 0x7f9af8069360 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:29:08.918 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:08.908+0000 7f9afdcc0700 1 --2- 172.21.15.98:0/1702620960 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f9af80ffd10 0x7f9af819a5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:29:08.918 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:08.908+0000 7f9afdcc0700 1 -- 172.21.15.98:0/1702620960 >> 172.21.15.98:0/1702620960 conn(0x7f9af80fa100 msgr2=0x7f9af80fc290 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:29:08.918 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:08.908+0000 7f9afdcc0700 1 -- 172.21.15.98:0/1702620960 shutdown_connections 2023-07-14T19:29:08.918 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:08.908+0000 7f9afdcc0700 1 -- 172.21.15.98:0/1702620960 wait complete. 2023-07-14T19:29:08.918 INFO:teuthology.orchestra.run.smithi098.stderr:dumped all 2023-07-14T19:29:08.921 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:29:08 smithi098 ceph-mon[127486]: from='client.14754 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "sub_name": "sv_0", "target": ["mon-mgr", ""]}]: dispatch 2023-07-14T19:29:08.921 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:29:08 smithi098 ceph-mon[127486]: from='client.14760 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_1", "group_name": "", "target": ["mon-mgr", ""]}]: dispatch 2023-07-14T19:29:08.921 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:29:08 smithi098 ceph-mon[127486]: pgmap v256: 97 pgs: 97 active+clean; 581 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 586 B/s rd, 5.0 KiB/s wr, 11 op/s 2023-07-14T19:29:08.922 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:29:08 smithi098 ceph-mon[127486]: mgrmap e17: x(active, since 6m), standbys: z, y 2023-07-14T19:29:08.922 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:29:08 smithi098 ceph-mon[127486]: from='client.14766 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "sub_name": "sv_1", "target": ["mon-mgr", ""]}]: dispatch 2023-07-14T19:29:08.922 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:29:08 smithi098 ceph-mon[127486]: from='client.? 172.21.15.98:0/4260159366' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-07-14T19:29:08.923 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:29:08 smithi098 ceph-mon[127486]: from='client.? 172.21.15.98:0/2261216108' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-07-14T19:29:08.924 INFO:teuthology.orchestra.run.smithi098.stdout:{"pg_ready":true,"pg_map":{"version":256,"stamp":"2023-07-14T19:29:07.592616+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":594984,"num_objects":41,"num_object_clones":0,"num_object_copies":123,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":41,"num_whiteouts":0,"num_read":141,"num_read_kb":123,"num_write":258,"num_write_kb":2973,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":208,"ondisk_log_size":208,"up":291,"acting":291,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":269,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":336572,"kb_used_data":12312,"kb_used_omap":20,"kb_used_meta":324075,"kb_avail":1124605252,"statfs":{"total":1151940427776,"available":1151595778048,"internally_reserved":0,"allocated":12607488,"data_stored":6912997,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":21372,"internal_metadata":331852932},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":14,"apply_latency_ms":14,"commit_latency_ns":14000000,"apply_latency_ns":14000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":4616,"num_objects":39,"num_object_clones":0,"num_object_copies":117,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":39,"num_whiteouts":0,"num_read":5,"num_read_kb":4,"num_write":77,"num_write_kb":35,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.985142"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":12,"reported_epoch":83,"state":"active+clean","last_fresh":"2023-07-14T19:29:02.573808+0000","last_change":"2023-07-14T19:29:01.313813+0000","last_active":"2023-07-14T19:29:02.573808+0000","last_peered":"2023-07-14T19:29:02.573808+0000","last_clean":"2023-07-14T19:29:02.573808+0000","last_became_active":"2023-07-14T19:29:01.313514+0000","last_became_peered":"2023-07-14T19:29:01.313514+0000","last_unstale":"2023-07-14T19:29:02.573808+0000","last_undegraded":"2023-07-14T19:29:02.573808+0000","last_fullsized":"2023-07-14T19:29:02.573808+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:29:00.604983+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:29:00.604983+0000","last_clean_scrub_stamp":"2023-07-14T19:29:00.604983+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T04:19:18.663334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":12,"reported_epoch":83,"state":"active+clean","last_fresh":"2023-07-14T19:29:02.603793+0000","last_change":"2023-07-14T19:29:01.316899+0000","last_active":"2023-07-14T19:29:02.603793+0000","last_peered":"2023-07-14T19:29:02.603793+0000","last_clean":"2023-07-14T19:29:02.603793+0000","last_became_active":"2023-07-14T19:29:01.316603+0000","last_became_peered":"2023-07-14T19:29:01.316603+0000","last_unstale":"2023-07-14T19:29:02.603793+0000","last_undegraded":"2023-07-14T19:29:02.603793+0000","last_fullsized":"2023-07-14T19:29:02.603793+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:29:00.604983+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:29:00.604983+0000","last_clean_scrub_stamp":"2023-07-14T19:29:00.604983+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T01:32:43.783686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":12,"reported_epoch":83,"state":"active+clean","last_fresh":"2023-07-14T19:29:02.603746+0000","last_change":"2023-07-14T19:29:01.316557+0000","last_active":"2023-07-14T19:29:02.603746+0000","last_peered":"2023-07-14T19:29:02.603746+0000","last_clean":"2023-07-14T19:29:02.603746+0000","last_became_active":"2023-07-14T19:29:01.316079+0000","last_became_peered":"2023-07-14T19:29:01.316079+0000","last_unstale":"2023-07-14T19:29:02.603746+0000","last_undegraded":"2023-07-14T19:29:02.603746+0000","last_fullsized":"2023-07-14T19:29:02.603746+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:29:00.604983+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:29:00.604983+0000","last_clean_scrub_stamp":"2023-07-14T19:29:00.604983+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T05:43:38.465282+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":12,"reported_epoch":83,"state":"active+clean","last_fresh":"2023-07-14T19:29:03.339617+0000","last_change":"2023-07-14T19:29:01.307992+0000","last_active":"2023-07-14T19:29:03.339617+0000","last_peered":"2023-07-14T19:29:03.339617+0000","last_clean":"2023-07-14T19:29:03.339617+0000","last_became_active":"2023-07-14T19:29:01.307687+0000","last_became_peered":"2023-07-14T19:29:01.307687+0000","last_unstale":"2023-07-14T19:29:03.339617+0000","last_undegraded":"2023-07-14T19:29:03.339617+0000","last_fullsized":"2023-07-14T19:29:03.339617+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:29:00.604983+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:29:00.604983+0000","last_clean_scrub_stamp":"2023-07-14T19:29:00.604983+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T03:04:56.480796+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":12,"reported_epoch":83,"state":"active+clean","last_fresh":"2023-07-14T19:29:02.660819+0000","last_change":"2023-07-14T19:29:01.311036+0000","last_active":"2023-07-14T19:29:02.660819+0000","last_peered":"2023-07-14T19:29:02.660819+0000","last_clean":"2023-07-14T19:29:02.660819+0000","last_became_active":"2023-07-14T19:29:01.310723+0000","last_became_peered":"2023-07-14T19:29:01.310723+0000","last_unstale":"2023-07-14T19:29:02.660819+0000","last_undegraded":"2023-07-14T19:29:02.660819+0000","last_fullsized":"2023-07-14T19:29:02.660819+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:29:00.604983+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:29:00.604983+0000","last_clean_scrub_stamp":"2023-07-14T19:29:00.604983+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T23:51:40.842901+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":12,"reported_epoch":83,"state":"active+clean","last_fresh":"2023-07-14T19:29:03.339712+0000","last_change":"2023-07-14T19:29:01.315246+0000","last_active":"2023-07-14T19:29:03.339712+0000","last_peered":"2023-07-14T19:29:03.339712+0000","last_clean":"2023-07-14T19:29:03.339712+0000","last_became_active":"2023-07-14T19:29:01.314885+0000","last_became_peered":"2023-07-14T19:29:01.314885+0000","last_unstale":"2023-07-14T19:29:03.339712+0000","last_undegraded":"2023-07-14T19:29:03.339712+0000","last_fullsized":"2023-07-14T19:29:03.339712+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:29:00.604983+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:29:00.604983+0000","last_clean_scrub_stamp":"2023-07-14T19:29:00.604983+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T04:22:25.924419+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":12,"reported_epoch":83,"state":"active+clean","last_fresh":"2023-07-14T19:29:03.338493+0000","last_change":"2023-07-14T19:29:01.306322+0000","last_active":"2023-07-14T19:29:03.338493+0000","last_peered":"2023-07-14T19:29:03.338493+0000","last_clean":"2023-07-14T19:29:03.338493+0000","last_became_active":"2023-07-14T19:29:01.306024+0000","last_became_peered":"2023-07-14T19:29:01.306024+0000","last_unstale":"2023-07-14T19:29:03.338493+0000","last_undegraded":"2023-07-14T19:29:03.338493+0000","last_fullsized":"2023-07-14T19:29:03.338493+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:29:00.604983+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:29:00.604983+0000","last_clean_scrub_stamp":"2023-07-14T19:29:00.604983+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T04:13:45.893986+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":12,"reported_epoch":83,"state":"active+clean","last_fresh":"2023-07-14T19:29:02.298956+0000","last_change":"2023-07-14T19:29:01.312514+0000","last_active":"2023-07-14T19:29:02.298956+0000","last_peered":"2023-07-14T19:29:02.298956+0000","last_clean":"2023-07-14T19:29:02.298956+0000","last_became_active":"2023-07-14T19:29:01.312260+0000","last_became_peered":"2023-07-14T19:29:01.312260+0000","last_unstale":"2023-07-14T19:29:02.298956+0000","last_undegraded":"2023-07-14T19:29:02.298956+0000","last_fullsized":"2023-07-14T19:29:02.298956+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:29:00.604983+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:29:00.604983+0000","last_clean_scrub_stamp":"2023-07-14T19:29:00.604983+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T21:31:03.757922+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":12,"reported_epoch":83,"state":"active+clean","last_fresh":"2023-07-14T19:29:02.300275+0000","last_change":"2023-07-14T19:29:01.312344+0000","last_active":"2023-07-14T19:29:02.300275+0000","last_peered":"2023-07-14T19:29:02.300275+0000","last_clean":"2023-07-14T19:29:02.300275+0000","last_became_active":"2023-07-14T19:29:01.311960+0000","last_became_peered":"2023-07-14T19:29:01.311960+0000","last_unstale":"2023-07-14T19:29:02.300275+0000","last_undegraded":"2023-07-14T19:29:02.300275+0000","last_fullsized":"2023-07-14T19:29:02.300275+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:29:00.604983+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:29:00.604983+0000","last_clean_scrub_stamp":"2023-07-14T19:29:00.604983+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T23:15:52.484949+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":12,"reported_epoch":83,"state":"active+clean","last_fresh":"2023-07-14T19:29:03.339513+0000","last_change":"2023-07-14T19:29:01.314868+0000","last_active":"2023-07-14T19:29:03.339513+0000","last_peered":"2023-07-14T19:29:03.339513+0000","last_clean":"2023-07-14T19:29:03.339513+0000","last_became_active":"2023-07-14T19:29:01.314617+0000","last_became_peered":"2023-07-14T19:29:01.314617+0000","last_unstale":"2023-07-14T19:29:03.339513+0000","last_undegraded":"2023-07-14T19:29:03.339513+0000","last_fullsized":"2023-07-14T19:29:03.339513+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:29:00.604983+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:29:00.604983+0000","last_clean_scrub_stamp":"2023-07-14T19:29:00.604983+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T20:33:35.726050+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":12,"reported_epoch":83,"state":"active+clean","last_fresh":"2023-07-14T19:29:03.339870+0000","last_change":"2023-07-14T19:29:01.319124+0000","last_active":"2023-07-14T19:29:03.339870+0000","last_peered":"2023-07-14T19:29:03.339870+0000","last_clean":"2023-07-14T19:29:03.339870+0000","last_became_active":"2023-07-14T19:29:01.318392+0000","last_became_peered":"2023-07-14T19:29:01.318392+0000","last_unstale":"2023-07-14T19:29:03.339870+0000","last_undegraded":"2023-07-14T19:29:03.339870+0000","last_fullsized":"2023-07-14T19:29:03.339870+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:29:00.604983+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:29:00.604983+0000","last_clean_scrub_stamp":"2023-07-14T19:29:00.604983+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T06:19:35.297680+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":12,"reported_epoch":83,"state":"active+clean","last_fresh":"2023-07-14T19:29:03.339129+0000","last_change":"2023-07-14T19:29:01.319003+0000","last_active":"2023-07-14T19:29:03.339129+0000","last_peered":"2023-07-14T19:29:03.339129+0000","last_clean":"2023-07-14T19:29:03.339129+0000","last_became_active":"2023-07-14T19:29:01.316550+0000","last_became_peered":"2023-07-14T19:29:01.316550+0000","last_unstale":"2023-07-14T19:29:03.339129+0000","last_undegraded":"2023-07-14T19:29:03.339129+0000","last_fullsized":"2023-07-14T19:29:03.339129+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:29:00.604983+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:29:00.604983+0000","last_clean_scrub_stamp":"2023-07-14T19:29:00.604983+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T22:06:18.517102+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":12,"reported_epoch":83,"state":"active+clean","last_fresh":"2023-07-14T19:29:02.444425+0000","last_change":"2023-07-14T19:29:01.318596+0000","last_active":"2023-07-14T19:29:02.444425+0000","last_peered":"2023-07-14T19:29:02.444425+0000","last_clean":"2023-07-14T19:29:02.444425+0000","last_became_active":"2023-07-14T19:29:01.318278+0000","last_became_peered":"2023-07-14T19:29:01.318278+0000","last_unstale":"2023-07-14T19:29:02.444425+0000","last_undegraded":"2023-07-14T19:29:02.444425+0000","last_fullsized":"2023-07-14T19:29:02.444425+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:29:00.604983+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:29:00.604983+0000","last_clean_scrub_stamp":"2023-07-14T19:29:00.604983+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T06:00:40.438416+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":12,"reported_epoch":83,"state":"active+clean","last_fresh":"2023-07-14T19:29:02.603553+0000","last_change":"2023-07-14T19:29:01.314344+0000","last_active":"2023-07-14T19:29:02.603553+0000","last_peered":"2023-07-14T19:29:02.603553+0000","last_clean":"2023-07-14T19:29:02.603553+0000","last_became_active":"2023-07-14T19:29:01.314054+0000","last_became_peered":"2023-07-14T19:29:01.314054+0000","last_unstale":"2023-07-14T19:29:02.603553+0000","last_undegraded":"2023-07-14T19:29:02.603553+0000","last_fullsized":"2023-07-14T19:29:02.603553+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:29:00.604983+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:29:00.604983+0000","last_clean_scrub_stamp":"2023-07-14T19:29:00.604983+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T23:03:20.122826+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":12,"reported_epoch":83,"state":"active+clean","last_fresh":"2023-07-14T19:29:02.603428+0000","last_change":"2023-07-14T19:29:01.314703+0000","last_active":"2023-07-14T19:29:02.603428+0000","last_peered":"2023-07-14T19:29:02.603428+0000","last_clean":"2023-07-14T19:29:02.603428+0000","last_became_active":"2023-07-14T19:29:01.314425+0000","last_became_peered":"2023-07-14T19:29:01.314425+0000","last_unstale":"2023-07-14T19:29:02.603428+0000","last_undegraded":"2023-07-14T19:29:02.603428+0000","last_fullsized":"2023-07-14T19:29:02.603428+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:29:00.604983+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:29:00.604983+0000","last_clean_scrub_stamp":"2023-07-14T19:29:00.604983+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T02:24:08.467559+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":12,"reported_epoch":83,"state":"active+clean","last_fresh":"2023-07-14T19:29:02.299763+0000","last_change":"2023-07-14T19:29:01.308769+0000","last_active":"2023-07-14T19:29:02.299763+0000","last_peered":"2023-07-14T19:29:02.299763+0000","last_clean":"2023-07-14T19:29:02.299763+0000","last_became_active":"2023-07-14T19:29:01.308450+0000","last_became_peered":"2023-07-14T19:29:01.308450+0000","last_unstale":"2023-07-14T19:29:02.299763+0000","last_undegraded":"2023-07-14T19:29:02.299763+0000","last_fullsized":"2023-07-14T19:29:02.299763+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:29:00.604983+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:29:00.604983+0000","last_clean_scrub_stamp":"2023-07-14T19:29:00.604983+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T06:36:10.772949+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":12,"reported_epoch":83,"state":"active+clean","last_fresh":"2023-07-14T19:29:02.573896+0000","last_change":"2023-07-14T19:29:01.313928+0000","last_active":"2023-07-14T19:29:02.573896+0000","last_peered":"2023-07-14T19:29:02.573896+0000","last_clean":"2023-07-14T19:29:02.573896+0000","last_became_active":"2023-07-14T19:29:01.313528+0000","last_became_peered":"2023-07-14T19:29:01.313528+0000","last_unstale":"2023-07-14T19:29:02.573896+0000","last_undegraded":"2023-07-14T19:29:02.573896+0000","last_fullsized":"2023-07-14T19:29:02.573896+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:29:00.604983+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:29:00.604983+0000","last_clean_scrub_stamp":"2023-07-14T19:29:00.604983+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T20:08:14.603380+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":12,"reported_epoch":83,"state":"active+clean","last_fresh":"2023-07-14T19:29:02.864370+0000","last_change":"2023-07-14T19:29:01.318903+0000","last_active":"2023-07-14T19:29:02.864370+0000","last_peered":"2023-07-14T19:29:02.864370+0000","last_clean":"2023-07-14T19:29:02.864370+0000","last_became_active":"2023-07-14T19:29:01.318610+0000","last_became_peered":"2023-07-14T19:29:01.318610+0000","last_unstale":"2023-07-14T19:29:02.864370+0000","last_undegraded":"2023-07-14T19:29:02.864370+0000","last_fullsized":"2023-07-14T19:29:02.864370+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:29:00.604983+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:29:00.604983+0000","last_clean_scrub_stamp":"2023-07-14T19:29:00.604983+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T20:01:45.469683+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":12,"reported_epoch":83,"state":"active+clean","last_fresh":"2023-07-14T19:29:03.222037+0000","last_change":"2023-07-14T19:29:01.315405+0000","last_active":"2023-07-14T19:29:03.222037+0000","last_peered":"2023-07-14T19:29:03.222037+0000","last_clean":"2023-07-14T19:29:03.222037+0000","last_became_active":"2023-07-14T19:29:01.315156+0000","last_became_peered":"2023-07-14T19:29:01.315156+0000","last_unstale":"2023-07-14T19:29:03.222037+0000","last_undegraded":"2023-07-14T19:29:03.222037+0000","last_fullsized":"2023-07-14T19:29:03.222037+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:29:00.604983+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:29:00.604983+0000","last_clean_scrub_stamp":"2023-07-14T19:29:00.604983+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T02:34:33.971469+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"82'4","reported_seq":18,"reported_epoch":83,"state":"active+clean","last_fresh":"2023-07-14T19:29:02.864366+0000","last_change":"2023-07-14T19:29:00.618385+0000","last_active":"2023-07-14T19:29:02.864366+0000","last_peered":"2023-07-14T19:29:02.864366+0000","last_clean":"2023-07-14T19:29:02.864366+0000","last_became_active":"2023-07-14T19:29:00.618081+0000","last_became_peered":"2023-07-14T19:29:00.618081+0000","last_unstale":"2023-07-14T19:29:02.864366+0000","last_undegraded":"2023-07-14T19:29:02.864366+0000","last_fullsized":"2023-07-14T19:29:02.864366+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:28:59.600128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:28:59.600128+0000","last_clean_scrub_stamp":"2023-07-14T19:28:59.600128+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T06:55:30.739187+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":12,"reported_epoch":83,"state":"active+clean","last_fresh":"2023-07-14T19:29:02.299334+0000","last_change":"2023-07-14T19:29:01.305197+0000","last_active":"2023-07-14T19:29:02.299334+0000","last_peered":"2023-07-14T19:29:02.299334+0000","last_clean":"2023-07-14T19:29:02.299334+0000","last_became_active":"2023-07-14T19:29:01.304902+0000","last_became_peered":"2023-07-14T19:29:01.304902+0000","last_unstale":"2023-07-14T19:29:02.299334+0000","last_undegraded":"2023-07-14T19:29:02.299334+0000","last_fullsized":"2023-07-14T19:29:02.299334+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:29:00.604983+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:29:00.604983+0000","last_clean_scrub_stamp":"2023-07-14T19:29:00.604983+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T20:54:29.407321+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1e","version":"82'1","reported_seq":15,"reported_epoch":83,"state":"active+clean","last_fresh":"2023-07-14T19:29:02.299472+0000","last_change":"2023-07-14T19:29:00.626316+0000","last_active":"2023-07-14T19:29:02.299472+0000","last_peered":"2023-07-14T19:29:02.299472+0000","last_clean":"2023-07-14T19:29:02.299472+0000","last_became_active":"2023-07-14T19:29:00.626035+0000","last_became_peered":"2023-07-14T19:29:00.626035+0000","last_unstale":"2023-07-14T19:29:02.299472+0000","last_undegraded":"2023-07-14T19:29:02.299472+0000","last_fullsized":"2023-07-14T19:29:02.299472+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:28:59.600128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:28:59.600128+0000","last_clean_scrub_stamp":"2023-07-14T19:28:59.600128+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T23:59:08.083433+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":12,"reported_epoch":83,"state":"active+clean","last_fresh":"2023-07-14T19:29:02.603883+0000","last_change":"2023-07-14T19:29:01.317118+0000","last_active":"2023-07-14T19:29:02.603883+0000","last_peered":"2023-07-14T19:29:02.603883+0000","last_clean":"2023-07-14T19:29:02.603883+0000","last_became_active":"2023-07-14T19:29:01.316805+0000","last_became_peered":"2023-07-14T19:29:01.316805+0000","last_unstale":"2023-07-14T19:29:02.603883+0000","last_undegraded":"2023-07-14T19:29:02.603883+0000","last_fullsized":"2023-07-14T19:29:02.603883+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:29:00.604983+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:29:00.604983+0000","last_clean_scrub_stamp":"2023-07-14T19:29:00.604983+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T04:47:32.634271+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1d","version":"83'4","reported_seq":18,"reported_epoch":83,"state":"active+clean","last_fresh":"2023-07-14T19:29:05.679368+0000","last_change":"2023-07-14T19:29:00.619435+0000","last_active":"2023-07-14T19:29:05.679368+0000","last_peered":"2023-07-14T19:29:05.679368+0000","last_clean":"2023-07-14T19:29:05.679368+0000","last_became_active":"2023-07-14T19:29:00.619131+0000","last_became_peered":"2023-07-14T19:29:00.619131+0000","last_unstale":"2023-07-14T19:29:05.679368+0000","last_undegraded":"2023-07-14T19:29:05.679368+0000","last_fullsized":"2023-07-14T19:29:05.679368+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:28:59.600128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:28:59.600128+0000","last_clean_scrub_stamp":"2023-07-14T19:28:59.600128+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T04:51:10.432113+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":12,"reported_epoch":83,"state":"active+clean","last_fresh":"2023-07-14T19:29:02.299884+0000","last_change":"2023-07-14T19:29:01.317947+0000","last_active":"2023-07-14T19:29:02.299884+0000","last_peered":"2023-07-14T19:29:02.299884+0000","last_clean":"2023-07-14T19:29:02.299884+0000","last_became_active":"2023-07-14T19:29:01.317633+0000","last_became_peered":"2023-07-14T19:29:01.317633+0000","last_unstale":"2023-07-14T19:29:02.299884+0000","last_undegraded":"2023-07-14T19:29:02.299884+0000","last_fullsized":"2023-07-14T19:29:02.299884+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:29:00.604983+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:29:00.604983+0000","last_clean_scrub_stamp":"2023-07-14T19:29:00.604983+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T00:25:43.200161+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1c","version":"82'2","reported_seq":16,"reported_epoch":83,"state":"active+clean","last_fresh":"2023-07-14T19:29:02.661063+0000","last_change":"2023-07-14T19:29:00.620401+0000","last_active":"2023-07-14T19:29:02.661063+0000","last_peered":"2023-07-14T19:29:02.661063+0000","last_clean":"2023-07-14T19:29:02.661063+0000","last_became_active":"2023-07-14T19:29:00.620134+0000","last_became_peered":"2023-07-14T19:29:00.620134+0000","last_unstale":"2023-07-14T19:29:02.661063+0000","last_undegraded":"2023-07-14T19:29:02.661063+0000","last_fullsized":"2023-07-14T19:29:02.661063+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:28:59.600128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:28:59.600128+0000","last_clean_scrub_stamp":"2023-07-14T19:28:59.600128+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T06:39:10.382383+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":12,"reported_epoch":83,"state":"active+clean","last_fresh":"2023-07-14T19:29:02.299819+0000","last_change":"2023-07-14T19:29:01.312174+0000","last_active":"2023-07-14T19:29:02.299819+0000","last_peered":"2023-07-14T19:29:02.299819+0000","last_clean":"2023-07-14T19:29:02.299819+0000","last_became_active":"2023-07-14T19:29:01.311891+0000","last_became_peered":"2023-07-14T19:29:01.311891+0000","last_unstale":"2023-07-14T19:29:02.299819+0000","last_undegraded":"2023-07-14T19:29:02.299819+0000","last_fullsized":"2023-07-14T19:29:02.299819+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:29:00.604983+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:29:00.604983+0000","last_clean_scrub_stamp":"2023-07-14T19:29:00.604983+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T19:35:58.889437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1b","version":"83'6","reported_seq":23,"reported_epoch":83,"state":"active+clean","last_fresh":"2023-07-14T19:29:06.303726+0000","last_change":"2023-07-14T19:29:00.622257+0000","last_active":"2023-07-14T19:29:06.303726+0000","last_peered":"2023-07-14T19:29:06.303726+0000","last_clean":"2023-07-14T19:29:06.303726+0000","last_became_active":"2023-07-14T19:29:00.621984+0000","last_became_peered":"2023-07-14T19:29:00.621984+0000","last_unstale":"2023-07-14T19:29:06.303726+0000","last_undegraded":"2023-07-14T19:29:06.303726+0000","last_fullsized":"2023-07-14T19:29:06.303726+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:28:59.600128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:28:59.600128+0000","last_clean_scrub_stamp":"2023-07-14T19:28:59.600128+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T01:01:52.888462+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":964,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":3,"num_read_kb":2,"num_write":3,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":12,"reported_epoch":83,"state":"active+clean","last_fresh":"2023-07-14T19:29:02.660873+0000","last_change":"2023-07-14T19:29:01.306274+0000","last_active":"2023-07-14T19:29:02.660873+0000","last_peered":"2023-07-14T19:29:02.660873+0000","last_clean":"2023-07-14T19:29:02.660873+0000","last_became_active":"2023-07-14T19:29:01.305863+0000","last_became_peered":"2023-07-14T19:29:01.305863+0000","last_unstale":"2023-07-14T19:29:02.660873+0000","last_undegraded":"2023-07-14T19:29:02.660873+0000","last_fullsized":"2023-07-14T19:29:02.660873+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:29:00.604983+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:29:00.604983+0000","last_clean_scrub_stamp":"2023-07-14T19:29:00.604983+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T00:42:02.531260+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.1a","version":"82'1","reported_seq":15,"reported_epoch":83,"state":"active+clean","last_fresh":"2023-07-14T19:29:03.339637+0000","last_change":"2023-07-14T19:29:00.619509+0000","last_active":"2023-07-14T19:29:03.339637+0000","last_peered":"2023-07-14T19:29:03.339637+0000","last_clean":"2023-07-14T19:29:03.339637+0000","last_became_active":"2023-07-14T19:29:00.619193+0000","last_became_peered":"2023-07-14T19:29:00.619193+0000","last_unstale":"2023-07-14T19:29:03.339637+0000","last_undegraded":"2023-07-14T19:29:03.339637+0000","last_fullsized":"2023-07-14T19:29:03.339637+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:28:59.600128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:28:59.600128+0000","last_clean_scrub_stamp":"2023-07-14T19:28:59.600128+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T01:07:45.846978+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":12,"reported_epoch":83,"state":"active+clean","last_fresh":"2023-07-14T19:29:02.299714+0000","last_change":"2023-07-14T19:29:01.307535+0000","last_active":"2023-07-14T19:29:02.299714+0000","last_peered":"2023-07-14T19:29:02.299714+0000","last_clean":"2023-07-14T19:29:02.299714+0000","last_became_active":"2023-07-14T19:29:01.307209+0000","last_became_peered":"2023-07-14T19:29:01.307209+0000","last_unstale":"2023-07-14T19:29:02.299714+0000","last_undegraded":"2023-07-14T19:29:02.299714+0000","last_fullsized":"2023-07-14T19:29:02.299714+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:29:00.604983+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:29:00.604983+0000","last_clean_scrub_stamp":"2023-07-14T19:29:00.604983+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T03:49:13.610038+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":14,"reported_epoch":83,"state":"active+clean","last_fresh":"2023-07-14T19:29:02.444474+0000","last_change":"2023-07-14T19:29:00.620671+0000","last_active":"2023-07-14T19:29:02.444474+0000","last_peered":"2023-07-14T19:29:02.444474+0000","last_clean":"2023-07-14T19:29:02.444474+0000","last_became_active":"2023-07-14T19:29:00.619786+0000","last_became_peered":"2023-07-14T19:29:00.619786+0000","last_unstale":"2023-07-14T19:29:02.444474+0000","last_undegraded":"2023-07-14T19:29:02.444474+0000","last_fullsized":"2023-07-14T19:29:02.444474+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:28:59.600128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:28:59.600128+0000","last_clean_scrub_stamp":"2023-07-14T19:28:59.600128+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T06:56:01.025242+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":12,"reported_epoch":83,"state":"active+clean","last_fresh":"2023-07-14T19:29:02.298842+0000","last_change":"2023-07-14T19:29:01.316248+0000","last_active":"2023-07-14T19:29:02.298842+0000","last_peered":"2023-07-14T19:29:02.298842+0000","last_clean":"2023-07-14T19:29:02.298842+0000","last_became_active":"2023-07-14T19:29:01.315974+0000","last_became_peered":"2023-07-14T19:29:01.315974+0000","last_unstale":"2023-07-14T19:29:02.298842+0000","last_undegraded":"2023-07-14T19:29:02.298842+0000","last_fullsized":"2023-07-14T19:29:02.298842+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:29:00.604983+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:29:00.604983+0000","last_clean_scrub_stamp":"2023-07-14T19:29:00.604983+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T01:48:48.147093+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"83'1","reported_seq":15,"reported_epoch":83,"state":"active+clean","last_fresh":"2023-07-14T19:29:02.299408+0000","last_change":"2023-07-14T19:29:00.624210+0000","last_active":"2023-07-14T19:29:02.299408+0000","last_peered":"2023-07-14T19:29:02.299408+0000","last_clean":"2023-07-14T19:29:02.299408+0000","last_became_active":"2023-07-14T19:29:00.623918+0000","last_became_peered":"2023-07-14T19:29:00.623918+0000","last_unstale":"2023-07-14T19:29:02.299408+0000","last_undegraded":"2023-07-14T19:29:02.299408+0000","last_fullsized":"2023-07-14T19:29:02.299408+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:28:59.600128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:28:59.600128+0000","last_clean_scrub_stamp":"2023-07-14T19:28:59.600128+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T01:38:53.165557+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":12,"reported_epoch":83,"state":"active+clean","last_fresh":"2023-07-14T19:29:02.864241+0000","last_change":"2023-07-14T19:29:01.313787+0000","last_active":"2023-07-14T19:29:02.864241+0000","last_peered":"2023-07-14T19:29:02.864241+0000","last_clean":"2023-07-14T19:29:02.864241+0000","last_became_active":"2023-07-14T19:29:01.313479+0000","last_became_peered":"2023-07-14T19:29:01.313479+0000","last_unstale":"2023-07-14T19:29:02.864241+0000","last_undegraded":"2023-07-14T19:29:02.864241+0000","last_fullsized":"2023-07-14T19:29:02.864241+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:29:00.604983+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:29:00.604983+0000","last_clean_scrub_stamp":"2023-07-14T19:29:00.604983+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T22:33:21.348977+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":14,"reported_epoch":83,"state":"active+clean","last_fresh":"2023-07-14T19:29:03.339281+0000","last_change":"2023-07-14T19:29:00.623053+0000","last_active":"2023-07-14T19:29:03.339281+0000","last_peered":"2023-07-14T19:29:03.339281+0000","last_clean":"2023-07-14T19:29:03.339281+0000","last_became_active":"2023-07-14T19:29:00.622760+0000","last_became_peered":"2023-07-14T19:29:00.622760+0000","last_unstale":"2023-07-14T19:29:03.339281+0000","last_undegraded":"2023-07-14T19:29:03.339281+0000","last_fullsized":"2023-07-14T19:29:03.339281+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:28:59.600128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:28:59.600128+0000","last_clean_scrub_stamp":"2023-07-14T19:28:59.600128+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T21:15:50.464654+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":12,"reported_epoch":83,"state":"active+clean","last_fresh":"2023-07-14T19:29:02.299982+0000","last_change":"2023-07-14T19:29:01.317509+0000","last_active":"2023-07-14T19:29:02.299982+0000","last_peered":"2023-07-14T19:29:02.299982+0000","last_clean":"2023-07-14T19:29:02.299982+0000","last_became_active":"2023-07-14T19:29:01.317238+0000","last_became_peered":"2023-07-14T19:29:01.317238+0000","last_unstale":"2023-07-14T19:29:02.299982+0000","last_undegraded":"2023-07-14T19:29:02.299982+0000","last_fullsized":"2023-07-14T19:29:02.299982+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:29:00.604983+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:29:00.604983+0000","last_clean_scrub_stamp":"2023-07-14T19:29:00.604983+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T04:59:01.463446+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.16","version":"83'2","reported_seq":16,"reported_epoch":83,"state":"active+clean","last_fresh":"2023-07-14T19:29:04.671803+0000","last_change":"2023-07-14T19:29:00.624024+0000","last_active":"2023-07-14T19:29:04.671803+0000","last_peered":"2023-07-14T19:29:04.671803+0000","last_clean":"2023-07-14T19:29:04.671803+0000","last_became_active":"2023-07-14T19:29:00.623638+0000","last_became_peered":"2023-07-14T19:29:00.623638+0000","last_unstale":"2023-07-14T19:29:04.671803+0000","last_undegraded":"2023-07-14T19:29:04.671803+0000","last_fullsized":"2023-07-14T19:29:04.671803+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:28:59.600128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:28:59.600128+0000","last_clean_scrub_stamp":"2023-07-14T19:28:59.600128+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T03:50:19.311702+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":12,"reported_epoch":83,"state":"active+clean","last_fresh":"2023-07-14T19:29:02.603804+0000","last_change":"2023-07-14T19:29:01.309164+0000","last_active":"2023-07-14T19:29:02.603804+0000","last_peered":"2023-07-14T19:29:02.603804+0000","last_clean":"2023-07-14T19:29:02.603804+0000","last_became_active":"2023-07-14T19:29:01.308797+0000","last_became_peered":"2023-07-14T19:29:01.308797+0000","last_unstale":"2023-07-14T19:29:02.603804+0000","last_undegraded":"2023-07-14T19:29:02.603804+0000","last_fullsized":"2023-07-14T19:29:02.603804+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:29:00.604983+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:29:00.604983+0000","last_clean_scrub_stamp":"2023-07-14T19:29:00.604983+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T23:09:54.074320+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"83'4","reported_seq":19,"reported_epoch":83,"state":"active+clean","last_fresh":"2023-07-14T19:29:04.671757+0000","last_change":"2023-07-14T19:29:00.625435+0000","last_active":"2023-07-14T19:29:04.671757+0000","last_peered":"2023-07-14T19:29:04.671757+0000","last_clean":"2023-07-14T19:29:04.671757+0000","last_became_active":"2023-07-14T19:29:00.625092+0000","last_became_peered":"2023-07-14T19:29:00.625092+0000","last_unstale":"2023-07-14T19:29:04.671757+0000","last_undegraded":"2023-07-14T19:29:04.671757+0000","last_fullsized":"2023-07-14T19:29:04.671757+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:28:59.600128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:28:59.600128+0000","last_clean_scrub_stamp":"2023-07-14T19:28:59.600128+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T02:44:25.187321+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":12,"reported_epoch":83,"state":"active+clean","last_fresh":"2023-07-14T19:29:02.300025+0000","last_change":"2023-07-14T19:29:01.315672+0000","last_active":"2023-07-14T19:29:02.300025+0000","last_peered":"2023-07-14T19:29:02.300025+0000","last_clean":"2023-07-14T19:29:02.300025+0000","last_became_active":"2023-07-14T19:29:01.315331+0000","last_became_peered":"2023-07-14T19:29:01.315331+0000","last_unstale":"2023-07-14T19:29:02.300025+0000","last_undegraded":"2023-07-14T19:29:02.300025+0000","last_fullsized":"2023-07-14T19:29:02.300025+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:29:00.604983+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:29:00.604983+0000","last_clean_scrub_stamp":"2023-07-14T19:29:00.604983+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T02:46:22.959698+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.14","version":"82'6","reported_seq":20,"reported_epoch":83,"state":"active+clean","last_fresh":"2023-07-14T19:29:03.338983+0000","last_change":"2023-07-14T19:29:00.622910+0000","last_active":"2023-07-14T19:29:03.338983+0000","last_peered":"2023-07-14T19:29:03.338983+0000","last_clean":"2023-07-14T19:29:03.338983+0000","last_became_active":"2023-07-14T19:29:00.622582+0000","last_became_peered":"2023-07-14T19:29:00.622582+0000","last_unstale":"2023-07-14T19:29:03.338983+0000","last_undegraded":"2023-07-14T19:29:03.338983+0000","last_fullsized":"2023-07-14T19:29:03.338983+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:28:59.600128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:28:59.600128+0000","last_clean_scrub_stamp":"2023-07-14T19:28:59.600128+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T20:35:52.757984+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1564,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":12,"reported_epoch":83,"state":"active+clean","last_fresh":"2023-07-14T19:29:02.660955+0000","last_change":"2023-07-14T19:29:01.306169+0000","last_active":"2023-07-14T19:29:02.660955+0000","last_peered":"2023-07-14T19:29:02.660955+0000","last_clean":"2023-07-14T19:29:02.660955+0000","last_became_active":"2023-07-14T19:29:01.305830+0000","last_became_peered":"2023-07-14T19:29:01.305830+0000","last_unstale":"2023-07-14T19:29:02.660955+0000","last_undegraded":"2023-07-14T19:29:02.660955+0000","last_fullsized":"2023-07-14T19:29:02.660955+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:29:00.604983+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:29:00.604983+0000","last_clean_scrub_stamp":"2023-07-14T19:29:00.604983+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T00:12:18.670125+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.13","version":"82'2","reported_seq":16,"reported_epoch":83,"state":"active+clean","last_fresh":"2023-07-14T19:29:03.221973+0000","last_change":"2023-07-14T19:29:00.626500+0000","last_active":"2023-07-14T19:29:03.221973+0000","last_peered":"2023-07-14T19:29:03.221973+0000","last_clean":"2023-07-14T19:29:03.221973+0000","last_became_active":"2023-07-14T19:29:00.626130+0000","last_became_peered":"2023-07-14T19:29:00.626130+0000","last_unstale":"2023-07-14T19:29:03.221973+0000","last_undegraded":"2023-07-14T19:29:03.221973+0000","last_fullsized":"2023-07-14T19:29:03.221973+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:28:59.600128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:28:59.600128+0000","last_clean_scrub_stamp":"2023-07-14T19:28:59.600128+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T01:08:40.766374+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":12,"reported_epoch":83,"state":"active+clean","last_fresh":"2023-07-14T19:29:02.444400+0000","last_change":"2023-07-14T19:29:01.317902+0000","last_active":"2023-07-14T19:29:02.444400+0000","last_peered":"2023-07-14T19:29:02.444400+0000","last_clean":"2023-07-14T19:29:02.444400+0000","last_became_active":"2023-07-14T19:29:01.317587+0000","last_became_peered":"2023-07-14T19:29:01.317587+0000","last_unstale":"2023-07-14T19:29:02.444400+0000","last_undegraded":"2023-07-14T19:29:02.444400+0000","last_fullsized":"2023-07-14T19:29:02.444400+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:29:00.604983+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:29:00.604983+0000","last_clean_scrub_stamp":"2023-07-14T19:29:00.604983+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T03:39:39.441094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.12","version":"83'3","reported_seq":17,"reported_epoch":83,"state":"active+clean","last_fresh":"2023-07-14T19:29:04.671869+0000","last_change":"2023-07-14T19:29:00.626345+0000","last_active":"2023-07-14T19:29:04.671869+0000","last_peered":"2023-07-14T19:29:04.671869+0000","last_clean":"2023-07-14T19:29:04.671869+0000","last_became_active":"2023-07-14T19:29:00.626059+0000","last_became_peered":"2023-07-14T19:29:00.626059+0000","last_unstale":"2023-07-14T19:29:04.671869+0000","last_undegraded":"2023-07-14T19:29:04.671869+0000","last_fullsized":"2023-07-14T19:29:04.671869+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:28:59.600128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:28:59.600128+0000","last_clean_scrub_stamp":"2023-07-14T19:28:59.600128+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T21:18:49.525099+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":12,"reported_epoch":83,"state":"active+clean","last_fresh":"2023-07-14T19:29:03.339214+0000","last_change":"2023-07-14T19:29:01.310518+0000","last_active":"2023-07-14T19:29:03.339214+0000","last_peered":"2023-07-14T19:29:03.339214+0000","last_clean":"2023-07-14T19:29:03.339214+0000","last_became_active":"2023-07-14T19:29:01.310213+0000","last_became_peered":"2023-07-14T19:29:01.310213+0000","last_unstale":"2023-07-14T19:29:03.339214+0000","last_undegraded":"2023-07-14T19:29:03.339214+0000","last_fullsized":"2023-07-14T19:29:03.339214+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:29:00.604983+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:29:00.604983+0000","last_clean_scrub_stamp":"2023-07-14T19:29:00.604983+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T01:11:48.079392+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"83'3","reported_seq":17,"reported_epoch":83,"state":"active+clean","last_fresh":"2023-07-14T19:29:05.678131+0000","last_change":"2023-07-14T19:29:00.620450+0000","last_active":"2023-07-14T19:29:05.678131+0000","last_peered":"2023-07-14T19:29:05.678131+0000","last_clean":"2023-07-14T19:29:05.678131+0000","last_became_active":"2023-07-14T19:29:00.620100+0000","last_became_peered":"2023-07-14T19:29:00.620100+0000","last_unstale":"2023-07-14T19:29:05.678131+0000","last_undegraded":"2023-07-14T19:29:05.678131+0000","last_fullsized":"2023-07-14T19:29:05.678131+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:28:59.600128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:28:59.600128+0000","last_clean_scrub_stamp":"2023-07-14T19:28:59.600128+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T23:01:06.885919+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":540,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":12,"reported_epoch":83,"state":"active+clean","last_fresh":"2023-07-14T19:29:03.338482+0000","last_change":"2023-07-14T19:29:01.320612+0000","last_active":"2023-07-14T19:29:03.338482+0000","last_peered":"2023-07-14T19:29:03.338482+0000","last_clean":"2023-07-14T19:29:03.338482+0000","last_became_active":"2023-07-14T19:29:01.319569+0000","last_became_peered":"2023-07-14T19:29:01.319569+0000","last_unstale":"2023-07-14T19:29:03.338482+0000","last_undegraded":"2023-07-14T19:29:03.338482+0000","last_fullsized":"2023-07-14T19:29:03.338482+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:29:00.604983+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:29:00.604983+0000","last_clean_scrub_stamp":"2023-07-14T19:29:00.604983+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T05:46:45.094405+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.10","version":"83'15","reported_seq":29,"reported_epoch":83,"state":"active+clean","last_fresh":"2023-07-14T19:29:05.679927+0000","last_change":"2023-07-14T19:29:00.618619+0000","last_active":"2023-07-14T19:29:05.679927+0000","last_peered":"2023-07-14T19:29:05.679927+0000","last_clean":"2023-07-14T19:29:05.679927+0000","last_became_active":"2023-07-14T19:29:00.617735+0000","last_became_peered":"2023-07-14T19:29:00.617735+0000","last_unstale":"2023-07-14T19:29:05.679927+0000","last_undegraded":"2023-07-14T19:29:05.679927+0000","last_fullsized":"2023-07-14T19:29:05.679927+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:28:59.600128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:28:59.600128+0000","last_clean_scrub_stamp":"2023-07-14T19:28:59.600128+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T22:03:33.860671+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":12,"reported_epoch":83,"state":"active+clean","last_fresh":"2023-07-14T19:29:02.299414+0000","last_change":"2023-07-14T19:29:01.303053+0000","last_active":"2023-07-14T19:29:02.299414+0000","last_peered":"2023-07-14T19:29:02.299414+0000","last_clean":"2023-07-14T19:29:02.299414+0000","last_became_active":"2023-07-14T19:29:01.302679+0000","last_became_peered":"2023-07-14T19:29:01.302679+0000","last_unstale":"2023-07-14T19:29:02.299414+0000","last_undegraded":"2023-07-14T19:29:02.299414+0000","last_fullsized":"2023-07-14T19:29:02.299414+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:29:00.604983+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:29:00.604983+0000","last_clean_scrub_stamp":"2023-07-14T19:29:00.604983+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T03:30:02.623040+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"82'1","reported_seq":15,"reported_epoch":83,"state":"active+clean","last_fresh":"2023-07-14T19:29:02.661220+0000","last_change":"2023-07-14T19:29:00.622494+0000","last_active":"2023-07-14T19:29:02.661220+0000","last_peered":"2023-07-14T19:29:02.661220+0000","last_clean":"2023-07-14T19:29:02.661220+0000","last_became_active":"2023-07-14T19:29:00.622195+0000","last_became_peered":"2023-07-14T19:29:00.622195+0000","last_unstale":"2023-07-14T19:29:02.661220+0000","last_undegraded":"2023-07-14T19:29:02.661220+0000","last_fullsized":"2023-07-14T19:29:02.661220+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:28:59.600128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:28:59.600128+0000","last_clean_scrub_stamp":"2023-07-14T19:28:59.600128+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T01:34:47.084212+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":12,"reported_epoch":83,"state":"active+clean","last_fresh":"2023-07-14T19:29:02.573720+0000","last_change":"2023-07-14T19:29:01.319375+0000","last_active":"2023-07-14T19:29:02.573720+0000","last_peered":"2023-07-14T19:29:02.573720+0000","last_clean":"2023-07-14T19:29:02.573720+0000","last_became_active":"2023-07-14T19:29:01.319068+0000","last_became_peered":"2023-07-14T19:29:01.319068+0000","last_unstale":"2023-07-14T19:29:02.573720+0000","last_undegraded":"2023-07-14T19:29:02.573720+0000","last_fullsized":"2023-07-14T19:29:02.573720+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:29:00.604983+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:29:00.604983+0000","last_clean_scrub_stamp":"2023-07-14T19:29:00.604983+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T06:31:40.249974+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":14,"reported_epoch":83,"state":"active+clean","last_fresh":"2023-07-14T19:29:02.864257+0000","last_change":"2023-07-14T19:29:00.622364+0000","last_active":"2023-07-14T19:29:02.864257+0000","last_peered":"2023-07-14T19:29:02.864257+0000","last_clean":"2023-07-14T19:29:02.864257+0000","last_became_active":"2023-07-14T19:29:00.622073+0000","last_became_peered":"2023-07-14T19:29:00.622073+0000","last_unstale":"2023-07-14T19:29:02.864257+0000","last_undegraded":"2023-07-14T19:29:02.864257+0000","last_fullsized":"2023-07-14T19:29:02.864257+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:28:59.600128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:28:59.600128+0000","last_clean_scrub_stamp":"2023-07-14T19:28:59.600128+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T20:41:09.339550+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":12,"reported_epoch":83,"state":"active+clean","last_fresh":"2023-07-14T19:29:03.339325+0000","last_change":"2023-07-14T19:29:01.314076+0000","last_active":"2023-07-14T19:29:03.339325+0000","last_peered":"2023-07-14T19:29:03.339325+0000","last_clean":"2023-07-14T19:29:03.339325+0000","last_became_active":"2023-07-14T19:29:01.313769+0000","last_became_peered":"2023-07-14T19:29:01.313769+0000","last_unstale":"2023-07-14T19:29:03.339325+0000","last_undegraded":"2023-07-14T19:29:03.339325+0000","last_fullsized":"2023-07-14T19:29:03.339325+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:29:00.604983+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:29:00.604983+0000","last_clean_scrub_stamp":"2023-07-14T19:29:00.604983+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T02:26:15.354399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":14,"reported_epoch":83,"state":"active+clean","last_fresh":"2023-07-14T19:29:03.338900+0000","last_change":"2023-07-14T19:29:00.622928+0000","last_active":"2023-07-14T19:29:03.338900+0000","last_peered":"2023-07-14T19:29:03.338900+0000","last_clean":"2023-07-14T19:29:03.338900+0000","last_became_active":"2023-07-14T19:29:00.622628+0000","last_became_peered":"2023-07-14T19:29:00.622628+0000","last_unstale":"2023-07-14T19:29:03.338900+0000","last_undegraded":"2023-07-14T19:29:03.338900+0000","last_fullsized":"2023-07-14T19:29:03.338900+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:28:59.600128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:28:59.600128+0000","last_clean_scrub_stamp":"2023-07-14T19:28:59.600128+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T21:41:30.895447+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":12,"reported_epoch":83,"state":"active+clean","last_fresh":"2023-07-14T19:29:03.339904+0000","last_change":"2023-07-14T19:29:01.322519+0000","last_active":"2023-07-14T19:29:03.339904+0000","last_peered":"2023-07-14T19:29:03.339904+0000","last_clean":"2023-07-14T19:29:03.339904+0000","last_became_active":"2023-07-14T19:29:01.322233+0000","last_became_peered":"2023-07-14T19:29:01.322233+0000","last_unstale":"2023-07-14T19:29:03.339904+0000","last_undegraded":"2023-07-14T19:29:03.339904+0000","last_fullsized":"2023-07-14T19:29:03.339904+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:29:00.604983+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:29:00.604983+0000","last_clean_scrub_stamp":"2023-07-14T19:29:00.604983+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T23:32:02.997469+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":12,"reported_epoch":83,"state":"active+clean","last_fresh":"2023-07-14T19:29:02.299512+0000","last_change":"2023-07-14T19:29:01.305439+0000","last_active":"2023-07-14T19:29:02.299512+0000","last_peered":"2023-07-14T19:29:02.299512+0000","last_clean":"2023-07-14T19:29:02.299512+0000","last_became_active":"2023-07-14T19:29:01.305060+0000","last_became_peered":"2023-07-14T19:29:01.305060+0000","last_unstale":"2023-07-14T19:29:02.299512+0000","last_undegraded":"2023-07-14T19:29:02.299512+0000","last_fullsized":"2023-07-14T19:29:02.299512+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:29:00.604983+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:29:00.604983+0000","last_clean_scrub_stamp":"2023-07-14T19:29:00.604983+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T06:07:47.229804+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":14,"reported_epoch":83,"state":"active+clean","last_fresh":"2023-07-14T19:29:02.444211+0000","last_change":"2023-07-14T19:29:00.620867+0000","last_active":"2023-07-14T19:29:02.444211+0000","last_peered":"2023-07-14T19:29:02.444211+0000","last_clean":"2023-07-14T19:29:02.444211+0000","last_became_active":"2023-07-14T19:29:00.620370+0000","last_became_peered":"2023-07-14T19:29:00.620370+0000","last_unstale":"2023-07-14T19:29:02.444211+0000","last_undegraded":"2023-07-14T19:29:02.444211+0000","last_fullsized":"2023-07-14T19:29:02.444211+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:28:59.600128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:28:59.600128+0000","last_clean_scrub_stamp":"2023-07-14T19:28:59.600128+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T05:53:10.376311+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":12,"reported_epoch":83,"state":"active+clean","last_fresh":"2023-07-14T19:29:02.299185+0000","last_change":"2023-07-14T19:29:01.306289+0000","last_active":"2023-07-14T19:29:02.299185+0000","last_peered":"2023-07-14T19:29:02.299185+0000","last_clean":"2023-07-14T19:29:02.299185+0000","last_became_active":"2023-07-14T19:29:01.305987+0000","last_became_peered":"2023-07-14T19:29:01.305987+0000","last_unstale":"2023-07-14T19:29:02.299185+0000","last_undegraded":"2023-07-14T19:29:02.299185+0000","last_fullsized":"2023-07-14T19:29:02.299185+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:29:00.604983+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:29:00.604983+0000","last_clean_scrub_stamp":"2023-07-14T19:29:00.604983+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T07:21:20.067549+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"83'3","reported_seq":17,"reported_epoch":83,"state":"active+clean","last_fresh":"2023-07-14T19:29:04.670894+0000","last_change":"2023-07-14T19:29:00.620937+0000","last_active":"2023-07-14T19:29:04.670894+0000","last_peered":"2023-07-14T19:29:04.670894+0000","last_clean":"2023-07-14T19:29:04.670894+0000","last_became_active":"2023-07-14T19:29:00.620616+0000","last_became_peered":"2023-07-14T19:29:00.620616+0000","last_unstale":"2023-07-14T19:29:04.670894+0000","last_undegraded":"2023-07-14T19:29:04.670894+0000","last_fullsized":"2023-07-14T19:29:04.670894+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:28:59.600128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:28:59.600128+0000","last_clean_scrub_stamp":"2023-07-14T19:28:59.600128+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T20:32:12.990940+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"22'137","reported_seq":341,"reported_epoch":83,"state":"active+clean","last_fresh":"2023-07-14T19:29:03.338586+0000","last_change":"2023-07-14T19:28:01.948805+0000","last_active":"2023-07-14T19:29:03.338586+0000","last_peered":"2023-07-14T19:29:03.338586+0000","last_clean":"2023-07-14T19:29:03.338586+0000","last_became_active":"2023-07-14T19:28:01.948295+0000","last_became_peered":"2023-07-14T19:28:01.948295+0000","last_unstale":"2023-07-14T19:29:03.338586+0000","last_undegraded":"2023-07-14T19:29:03.338586+0000","last_fullsized":"2023-07-14T19:29:03.338586+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:25:10.240820+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:25:10.240820+0000","last_clean_scrub_stamp":"2023-07-14T19:25:10.240820+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T23:39:19.337174+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":12,"reported_epoch":83,"state":"active+clean","last_fresh":"2023-07-14T19:29:02.444463+0000","last_change":"2023-07-14T19:29:01.321025+0000","last_active":"2023-07-14T19:29:02.444463+0000","last_peered":"2023-07-14T19:29:02.444463+0000","last_clean":"2023-07-14T19:29:02.444463+0000","last_became_active":"2023-07-14T19:29:01.320787+0000","last_became_peered":"2023-07-14T19:29:01.320787+0000","last_unstale":"2023-07-14T19:29:02.444463+0000","last_undegraded":"2023-07-14T19:29:02.444463+0000","last_fullsized":"2023-07-14T19:29:02.444463+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:29:00.604983+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:29:00.604983+0000","last_clean_scrub_stamp":"2023-07-14T19:29:00.604983+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T02:26:20.883377+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"83'1","reported_seq":15,"reported_epoch":83,"state":"active+clean","last_fresh":"2023-07-14T19:29:03.338874+0000","last_change":"2023-07-14T19:29:00.623891+0000","last_active":"2023-07-14T19:29:03.338874+0000","last_peered":"2023-07-14T19:29:03.338874+0000","last_clean":"2023-07-14T19:29:03.338874+0000","last_became_active":"2023-07-14T19:29:00.623562+0000","last_became_peered":"2023-07-14T19:29:00.623562+0000","last_unstale":"2023-07-14T19:29:03.338874+0000","last_undegraded":"2023-07-14T19:29:03.338874+0000","last_fullsized":"2023-07-14T19:29:03.338874+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:28:59.600128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:28:59.600128+0000","last_clean_scrub_stamp":"2023-07-14T19:28:59.600128+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T19:35:24.864363+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.5","version":"83'1","reported_seq":15,"reported_epoch":83,"state":"active+clean","last_fresh":"2023-07-14T19:29:02.661060+0000","last_change":"2023-07-14T19:29:00.620282+0000","last_active":"2023-07-14T19:29:02.661060+0000","last_peered":"2023-07-14T19:29:02.661060+0000","last_clean":"2023-07-14T19:29:02.661060+0000","last_became_active":"2023-07-14T19:29:00.620005+0000","last_became_peered":"2023-07-14T19:29:00.620005+0000","last_unstale":"2023-07-14T19:29:02.661060+0000","last_undegraded":"2023-07-14T19:29:02.661060+0000","last_fullsized":"2023-07-14T19:29:02.661060+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:28:59.600128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:28:59.600128+0000","last_clean_scrub_stamp":"2023-07-14T19:28:59.600128+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T03:20:25.333743+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":12,"reported_epoch":83,"state":"active+clean","last_fresh":"2023-07-14T19:29:02.444350+0000","last_change":"2023-07-14T19:29:01.311708+0000","last_active":"2023-07-14T19:29:02.444350+0000","last_peered":"2023-07-14T19:29:02.444350+0000","last_clean":"2023-07-14T19:29:02.444350+0000","last_became_active":"2023-07-14T19:29:01.311400+0000","last_became_peered":"2023-07-14T19:29:01.311400+0000","last_unstale":"2023-07-14T19:29:02.444350+0000","last_undegraded":"2023-07-14T19:29:02.444350+0000","last_fullsized":"2023-07-14T19:29:02.444350+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:29:00.604983+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:29:00.604983+0000","last_clean_scrub_stamp":"2023-07-14T19:29:00.604983+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T23:01:26.499242+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":14,"reported_epoch":83,"state":"active+clean","last_fresh":"2023-07-14T19:29:02.864292+0000","last_change":"2023-07-14T19:29:00.617411+0000","last_active":"2023-07-14T19:29:02.864292+0000","last_peered":"2023-07-14T19:29:02.864292+0000","last_clean":"2023-07-14T19:29:02.864292+0000","last_became_active":"2023-07-14T19:29:00.617078+0000","last_became_peered":"2023-07-14T19:29:00.617078+0000","last_unstale":"2023-07-14T19:29:02.864292+0000","last_undegraded":"2023-07-14T19:29:02.864292+0000","last_fullsized":"2023-07-14T19:29:02.864292+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:28:59.600128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:28:59.600128+0000","last_clean_scrub_stamp":"2023-07-14T19:28:59.600128+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T21:38:43.985609+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":12,"reported_epoch":83,"state":"active+clean","last_fresh":"2023-07-14T19:29:02.299351+0000","last_change":"2023-07-14T19:29:01.307282+0000","last_active":"2023-07-14T19:29:02.299351+0000","last_peered":"2023-07-14T19:29:02.299351+0000","last_clean":"2023-07-14T19:29:02.299351+0000","last_became_active":"2023-07-14T19:29:01.306966+0000","last_became_peered":"2023-07-14T19:29:01.306966+0000","last_unstale":"2023-07-14T19:29:02.299351+0000","last_undegraded":"2023-07-14T19:29:02.299351+0000","last_fullsized":"2023-07-14T19:29:02.299351+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:29:00.604983+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:29:00.604983+0000","last_clean_scrub_stamp":"2023-07-14T19:29:00.604983+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T23:06:00.918796+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"83'4","reported_seq":18,"reported_epoch":83,"state":"active+clean","last_fresh":"2023-07-14T19:29:04.671123+0000","last_change":"2023-07-14T19:29:00.624599+0000","last_active":"2023-07-14T19:29:04.671123+0000","last_peered":"2023-07-14T19:29:04.671123+0000","last_clean":"2023-07-14T19:29:04.671123+0000","last_became_active":"2023-07-14T19:29:00.624272+0000","last_became_peered":"2023-07-14T19:29:00.624272+0000","last_unstale":"2023-07-14T19:29:04.671123+0000","last_undegraded":"2023-07-14T19:29:04.671123+0000","last_fullsized":"2023-07-14T19:29:04.671123+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:28:59.600128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:28:59.600128+0000","last_clean_scrub_stamp":"2023-07-14T19:28:59.600128+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T20:00:31.406685+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":12,"reported_epoch":83,"state":"active+clean","last_fresh":"2023-07-14T19:29:02.603663+0000","last_change":"2023-07-14T19:29:01.309411+0000","last_active":"2023-07-14T19:29:02.603663+0000","last_peered":"2023-07-14T19:29:02.603663+0000","last_clean":"2023-07-14T19:29:02.603663+0000","last_became_active":"2023-07-14T19:29:01.309113+0000","last_became_peered":"2023-07-14T19:29:01.309113+0000","last_unstale":"2023-07-14T19:29:02.603663+0000","last_undegraded":"2023-07-14T19:29:02.603663+0000","last_fullsized":"2023-07-14T19:29:02.603663+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:29:00.604983+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:29:00.604983+0000","last_clean_scrub_stamp":"2023-07-14T19:29:00.604983+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T07:09:16.642605+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":14,"reported_epoch":83,"state":"active+clean","last_fresh":"2023-07-14T19:29:02.299764+0000","last_change":"2023-07-14T19:29:00.622758+0000","last_active":"2023-07-14T19:29:02.299764+0000","last_peered":"2023-07-14T19:29:02.299764+0000","last_clean":"2023-07-14T19:29:02.299764+0000","last_became_active":"2023-07-14T19:29:00.622460+0000","last_became_peered":"2023-07-14T19:29:00.622460+0000","last_unstale":"2023-07-14T19:29:02.299764+0000","last_undegraded":"2023-07-14T19:29:02.299764+0000","last_fullsized":"2023-07-14T19:29:02.299764+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:28:59.600128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:28:59.600128+0000","last_clean_scrub_stamp":"2023-07-14T19:28:59.600128+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T23:30:15.407153+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":12,"reported_epoch":83,"state":"active+clean","last_fresh":"2023-07-14T19:29:02.660717+0000","last_change":"2023-07-14T19:29:01.317046+0000","last_active":"2023-07-14T19:29:02.660717+0000","last_peered":"2023-07-14T19:29:02.660717+0000","last_clean":"2023-07-14T19:29:02.660717+0000","last_became_active":"2023-07-14T19:29:01.316653+0000","last_became_peered":"2023-07-14T19:29:01.316653+0000","last_unstale":"2023-07-14T19:29:02.660717+0000","last_undegraded":"2023-07-14T19:29:02.660717+0000","last_fullsized":"2023-07-14T19:29:02.660717+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:29:00.604983+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:29:00.604983+0000","last_clean_scrub_stamp":"2023-07-14T19:29:00.604983+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T04:50:19.726545+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":14,"reported_epoch":83,"state":"active+clean","last_fresh":"2023-07-14T19:29:02.299046+0000","last_change":"2023-07-14T19:29:00.619474+0000","last_active":"2023-07-14T19:29:02.299046+0000","last_peered":"2023-07-14T19:29:02.299046+0000","last_clean":"2023-07-14T19:29:02.299046+0000","last_became_active":"2023-07-14T19:29:00.619240+0000","last_became_peered":"2023-07-14T19:29:00.619240+0000","last_unstale":"2023-07-14T19:29:02.299046+0000","last_undegraded":"2023-07-14T19:29:02.299046+0000","last_fullsized":"2023-07-14T19:29:02.299046+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:28:59.600128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:28:59.600128+0000","last_clean_scrub_stamp":"2023-07-14T19:28:59.600128+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T02:51:09.423733+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":12,"reported_epoch":83,"state":"active+clean","last_fresh":"2023-07-14T19:29:02.573459+0000","last_change":"2023-07-14T19:29:01.308964+0000","last_active":"2023-07-14T19:29:02.573459+0000","last_peered":"2023-07-14T19:29:02.573459+0000","last_clean":"2023-07-14T19:29:02.573459+0000","last_became_active":"2023-07-14T19:29:01.308638+0000","last_became_peered":"2023-07-14T19:29:01.308638+0000","last_unstale":"2023-07-14T19:29:02.573459+0000","last_undegraded":"2023-07-14T19:29:02.573459+0000","last_fullsized":"2023-07-14T19:29:02.573459+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:29:00.604983+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:29:00.604983+0000","last_clean_scrub_stamp":"2023-07-14T19:29:00.604983+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T03:51:09.875511+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":14,"reported_epoch":83,"state":"active+clean","last_fresh":"2023-07-14T19:29:03.221784+0000","last_change":"2023-07-14T19:29:00.617894+0000","last_active":"2023-07-14T19:29:03.221784+0000","last_peered":"2023-07-14T19:29:03.221784+0000","last_clean":"2023-07-14T19:29:03.221784+0000","last_became_active":"2023-07-14T19:29:00.617584+0000","last_became_peered":"2023-07-14T19:29:00.617584+0000","last_unstale":"2023-07-14T19:29:03.221784+0000","last_undegraded":"2023-07-14T19:29:03.221784+0000","last_fullsized":"2023-07-14T19:29:03.221784+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:28:59.600128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:28:59.600128+0000","last_clean_scrub_stamp":"2023-07-14T19:28:59.600128+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T23:24:30.467604+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":12,"reported_epoch":83,"state":"active+clean","last_fresh":"2023-07-14T19:29:02.299239+0000","last_change":"2023-07-14T19:29:01.317153+0000","last_active":"2023-07-14T19:29:02.299239+0000","last_peered":"2023-07-14T19:29:02.299239+0000","last_clean":"2023-07-14T19:29:02.299239+0000","last_became_active":"2023-07-14T19:29:01.316800+0000","last_became_peered":"2023-07-14T19:29:01.316800+0000","last_unstale":"2023-07-14T19:29:02.299239+0000","last_undegraded":"2023-07-14T19:29:02.299239+0000","last_fullsized":"2023-07-14T19:29:02.299239+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:29:00.604983+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:29:00.604983+0000","last_clean_scrub_stamp":"2023-07-14T19:29:00.604983+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T04:58:24.726933+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"83'4","reported_seq":18,"reported_epoch":83,"state":"active+clean","last_fresh":"2023-07-14T19:29:04.671987+0000","last_change":"2023-07-14T19:29:00.618469+0000","last_active":"2023-07-14T19:29:04.671987+0000","last_peered":"2023-07-14T19:29:04.671987+0000","last_clean":"2023-07-14T19:29:04.671987+0000","last_became_active":"2023-07-14T19:29:00.618218+0000","last_became_peered":"2023-07-14T19:29:00.618218+0000","last_unstale":"2023-07-14T19:29:04.671987+0000","last_undegraded":"2023-07-14T19:29:04.671987+0000","last_fullsized":"2023-07-14T19:29:04.671987+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:28:59.600128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:28:59.600128+0000","last_clean_scrub_stamp":"2023-07-14T19:28:59.600128+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T22:41:36.925396+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":12,"reported_epoch":83,"state":"active+clean","last_fresh":"2023-07-14T19:29:03.339251+0000","last_change":"2023-07-14T19:29:01.319023+0000","last_active":"2023-07-14T19:29:03.339251+0000","last_peered":"2023-07-14T19:29:03.339251+0000","last_clean":"2023-07-14T19:29:03.339251+0000","last_became_active":"2023-07-14T19:29:01.311530+0000","last_became_peered":"2023-07-14T19:29:01.311530+0000","last_unstale":"2023-07-14T19:29:03.339251+0000","last_undegraded":"2023-07-14T19:29:03.339251+0000","last_fullsized":"2023-07-14T19:29:03.339251+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:29:00.604983+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:29:00.604983+0000","last_clean_scrub_stamp":"2023-07-14T19:29:00.604983+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T01:46:30.175019+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":14,"reported_epoch":83,"state":"active+clean","last_fresh":"2023-07-14T19:29:03.222071+0000","last_change":"2023-07-14T19:29:00.624227+0000","last_active":"2023-07-14T19:29:03.222071+0000","last_peered":"2023-07-14T19:29:03.222071+0000","last_clean":"2023-07-14T19:29:03.222071+0000","last_became_active":"2023-07-14T19:29:00.623953+0000","last_became_peered":"2023-07-14T19:29:00.623953+0000","last_unstale":"2023-07-14T19:29:03.222071+0000","last_undegraded":"2023-07-14T19:29:03.222071+0000","last_fullsized":"2023-07-14T19:29:03.222071+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:28:59.600128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:28:59.600128+0000","last_clean_scrub_stamp":"2023-07-14T19:28:59.600128+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T04:31:39.691751+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":12,"reported_epoch":83,"state":"active+clean","last_fresh":"2023-07-14T19:29:03.340024+0000","last_change":"2023-07-14T19:29:01.319936+0000","last_active":"2023-07-14T19:29:03.340024+0000","last_peered":"2023-07-14T19:29:03.340024+0000","last_clean":"2023-07-14T19:29:03.340024+0000","last_became_active":"2023-07-14T19:29:01.319437+0000","last_became_peered":"2023-07-14T19:29:01.319437+0000","last_unstale":"2023-07-14T19:29:03.340024+0000","last_undegraded":"2023-07-14T19:29:03.340024+0000","last_fullsized":"2023-07-14T19:29:03.340024+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:29:00.604983+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:29:00.604983+0000","last_clean_scrub_stamp":"2023-07-14T19:29:00.604983+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T07:14:58.343588+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"82'1","reported_seq":16,"reported_epoch":83,"state":"active+clean","last_fresh":"2023-07-14T19:29:02.299764+0000","last_change":"2023-07-14T19:29:00.628103+0000","last_active":"2023-07-14T19:29:02.299764+0000","last_peered":"2023-07-14T19:29:02.299764+0000","last_clean":"2023-07-14T19:29:02.299764+0000","last_became_active":"2023-07-14T19:29:00.627689+0000","last_became_peered":"2023-07-14T19:29:00.627689+0000","last_unstale":"2023-07-14T19:29:02.299764+0000","last_undegraded":"2023-07-14T19:29:02.299764+0000","last_fullsized":"2023-07-14T19:29:02.299764+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:28:59.600128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:28:59.600128+0000","last_clean_scrub_stamp":"2023-07-14T19:28:59.600128+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T23:24:22.985508+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":12,"reported_epoch":83,"state":"active+clean","last_fresh":"2023-07-14T19:29:03.338902+0000","last_change":"2023-07-14T19:29:01.319519+0000","last_active":"2023-07-14T19:29:03.338902+0000","last_peered":"2023-07-14T19:29:03.338902+0000","last_clean":"2023-07-14T19:29:03.338902+0000","last_became_active":"2023-07-14T19:29:01.318900+0000","last_became_peered":"2023-07-14T19:29:01.318900+0000","last_unstale":"2023-07-14T19:29:03.338902+0000","last_undegraded":"2023-07-14T19:29:03.338902+0000","last_fullsized":"2023-07-14T19:29:03.338902+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:29:00.604983+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:29:00.604983+0000","last_clean_scrub_stamp":"2023-07-14T19:29:00.604983+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T20:31:55.947593+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"82'2","reported_seq":16,"reported_epoch":83,"state":"active+clean","last_fresh":"2023-07-14T19:29:02.444432+0000","last_change":"2023-07-14T19:29:00.619280+0000","last_active":"2023-07-14T19:29:02.444432+0000","last_peered":"2023-07-14T19:29:02.444432+0000","last_clean":"2023-07-14T19:29:02.444432+0000","last_became_active":"2023-07-14T19:29:00.618980+0000","last_became_peered":"2023-07-14T19:29:00.618980+0000","last_unstale":"2023-07-14T19:29:02.444432+0000","last_undegraded":"2023-07-14T19:29:02.444432+0000","last_fullsized":"2023-07-14T19:29:02.444432+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:28:59.600128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:28:59.600128+0000","last_clean_scrub_stamp":"2023-07-14T19:28:59.600128+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T22:09:48.530415+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":12,"reported_epoch":83,"state":"active+clean","last_fresh":"2023-07-14T19:29:03.339356+0000","last_change":"2023-07-14T19:29:01.320160+0000","last_active":"2023-07-14T19:29:03.339356+0000","last_peered":"2023-07-14T19:29:03.339356+0000","last_clean":"2023-07-14T19:29:03.339356+0000","last_became_active":"2023-07-14T19:29:01.319850+0000","last_became_peered":"2023-07-14T19:29:01.319850+0000","last_unstale":"2023-07-14T19:29:03.339356+0000","last_undegraded":"2023-07-14T19:29:03.339356+0000","last_fullsized":"2023-07-14T19:29:03.339356+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:29:00.604983+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:29:00.604983+0000","last_clean_scrub_stamp":"2023-07-14T19:29:00.604983+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T04:25:40.648850+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":12,"reported_epoch":83,"state":"active+clean","last_fresh":"2023-07-14T19:29:02.573560+0000","last_change":"2023-07-14T19:29:01.309770+0000","last_active":"2023-07-14T19:29:02.573560+0000","last_peered":"2023-07-14T19:29:02.573560+0000","last_clean":"2023-07-14T19:29:02.573560+0000","last_became_active":"2023-07-14T19:29:01.308677+0000","last_became_peered":"2023-07-14T19:29:01.308677+0000","last_unstale":"2023-07-14T19:29:02.573560+0000","last_undegraded":"2023-07-14T19:29:02.573560+0000","last_fullsized":"2023-07-14T19:29:02.573560+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:29:00.604983+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:29:00.604983+0000","last_clean_scrub_stamp":"2023-07-14T19:29:00.604983+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T21:27:03.743070+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":12,"reported_epoch":83,"state":"active+clean","last_fresh":"2023-07-14T19:29:03.221821+0000","last_change":"2023-07-14T19:29:01.319447+0000","last_active":"2023-07-14T19:29:03.221821+0000","last_peered":"2023-07-14T19:29:03.221821+0000","last_clean":"2023-07-14T19:29:03.221821+0000","last_became_active":"2023-07-14T19:29:01.319120+0000","last_became_peered":"2023-07-14T19:29:01.319120+0000","last_unstale":"2023-07-14T19:29:03.221821+0000","last_undegraded":"2023-07-14T19:29:03.221821+0000","last_fullsized":"2023-07-14T19:29:03.221821+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:29:00.604983+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:29:00.604983+0000","last_clean_scrub_stamp":"2023-07-14T19:29:00.604983+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T07:03:28.055942+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":12,"reported_epoch":83,"state":"active+clean","last_fresh":"2023-07-14T19:29:03.221929+0000","last_change":"2023-07-14T19:29:01.310554+0000","last_active":"2023-07-14T19:29:03.221929+0000","last_peered":"2023-07-14T19:29:03.221929+0000","last_clean":"2023-07-14T19:29:03.221929+0000","last_became_active":"2023-07-14T19:29:01.310302+0000","last_became_peered":"2023-07-14T19:29:01.310302+0000","last_unstale":"2023-07-14T19:29:03.221929+0000","last_undegraded":"2023-07-14T19:29:03.221929+0000","last_fullsized":"2023-07-14T19:29:03.221929+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:29:00.604983+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:29:00.604983+0000","last_clean_scrub_stamp":"2023-07-14T19:29:00.604983+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T04:50:18.928722+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":12,"reported_epoch":83,"state":"active+clean","last_fresh":"2023-07-14T19:29:02.603764+0000","last_change":"2023-07-14T19:29:01.318753+0000","last_active":"2023-07-14T19:29:02.603764+0000","last_peered":"2023-07-14T19:29:02.603764+0000","last_clean":"2023-07-14T19:29:02.603764+0000","last_became_active":"2023-07-14T19:29:01.318476+0000","last_became_peered":"2023-07-14T19:29:01.318476+0000","last_unstale":"2023-07-14T19:29:02.603764+0000","last_undegraded":"2023-07-14T19:29:02.603764+0000","last_fullsized":"2023-07-14T19:29:02.603764+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:29:00.604983+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:29:00.604983+0000","last_clean_scrub_stamp":"2023-07-14T19:29:00.604983+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T07:27:40.189788+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":12,"reported_epoch":83,"state":"active+clean","last_fresh":"2023-07-14T19:29:02.444422+0000","last_change":"2023-07-14T19:29:01.317843+0000","last_active":"2023-07-14T19:29:02.444422+0000","last_peered":"2023-07-14T19:29:02.444422+0000","last_clean":"2023-07-14T19:29:02.444422+0000","last_became_active":"2023-07-14T19:29:01.317550+0000","last_became_peered":"2023-07-14T19:29:01.317550+0000","last_unstale":"2023-07-14T19:29:02.444422+0000","last_undegraded":"2023-07-14T19:29:02.444422+0000","last_fullsized":"2023-07-14T19:29:02.444422+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:29:00.604983+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:29:00.604983+0000","last_clean_scrub_stamp":"2023-07-14T19:29:00.604983+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T22:59:15.139142+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":12,"reported_epoch":83,"state":"active+clean","last_fresh":"2023-07-14T19:29:02.573731+0000","last_change":"2023-07-14T19:29:01.309103+0000","last_active":"2023-07-14T19:29:02.573731+0000","last_peered":"2023-07-14T19:29:02.573731+0000","last_clean":"2023-07-14T19:29:02.573731+0000","last_became_active":"2023-07-14T19:29:01.308786+0000","last_became_peered":"2023-07-14T19:29:01.308786+0000","last_unstale":"2023-07-14T19:29:02.573731+0000","last_undegraded":"2023-07-14T19:29:02.573731+0000","last_fullsized":"2023-07-14T19:29:02.573731+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:29:00.604983+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:29:00.604983+0000","last_clean_scrub_stamp":"2023-07-14T19:29:00.604983+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T02:55:34.529448+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":12,"reported_epoch":83,"state":"active+clean","last_fresh":"2023-07-14T19:29:03.222213+0000","last_change":"2023-07-14T19:29:01.309940+0000","last_active":"2023-07-14T19:29:03.222213+0000","last_peered":"2023-07-14T19:29:03.222213+0000","last_clean":"2023-07-14T19:29:03.222213+0000","last_became_active":"2023-07-14T19:29:01.309638+0000","last_became_peered":"2023-07-14T19:29:01.309638+0000","last_unstale":"2023-07-14T19:29:03.222213+0000","last_undegraded":"2023-07-14T19:29:03.222213+0000","last_fullsized":"2023-07-14T19:29:03.222213+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:29:00.604983+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:29:00.604983+0000","last_clean_scrub_stamp":"2023-07-14T19:29:00.604983+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T03:28:26.563593+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":12,"reported_epoch":83,"state":"active+clean","last_fresh":"2023-07-14T19:29:02.299283+0000","last_change":"2023-07-14T19:29:01.305957+0000","last_active":"2023-07-14T19:29:02.299283+0000","last_peered":"2023-07-14T19:29:02.299283+0000","last_clean":"2023-07-14T19:29:02.299283+0000","last_became_active":"2023-07-14T19:29:01.305564+0000","last_became_peered":"2023-07-14T19:29:01.305564+0000","last_unstale":"2023-07-14T19:29:02.299283+0000","last_undegraded":"2023-07-14T19:29:02.299283+0000","last_fullsized":"2023-07-14T19:29:02.299283+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:29:00.604983+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:29:00.604983+0000","last_clean_scrub_stamp":"2023-07-14T19:29:00.604983+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T21:47:06.297622+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":12,"reported_epoch":83,"state":"active+clean","last_fresh":"2023-07-14T19:29:02.299375+0000","last_change":"2023-07-14T19:29:01.310521+0000","last_active":"2023-07-14T19:29:02.299375+0000","last_peered":"2023-07-14T19:29:02.299375+0000","last_clean":"2023-07-14T19:29:02.299375+0000","last_became_active":"2023-07-14T19:29:01.310211+0000","last_became_peered":"2023-07-14T19:29:01.310211+0000","last_unstale":"2023-07-14T19:29:02.299375+0000","last_undegraded":"2023-07-14T19:29:02.299375+0000","last_fullsized":"2023-07-14T19:29:02.299375+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:29:00.604983+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:29:00.604983+0000","last_clean_scrub_stamp":"2023-07-14T19:29:00.604983+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T19:41:09.317667+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":12,"reported_epoch":83,"state":"active+clean","last_fresh":"2023-07-14T19:29:02.298983+0000","last_change":"2023-07-14T19:29:01.308033+0000","last_active":"2023-07-14T19:29:02.298983+0000","last_peered":"2023-07-14T19:29:02.298983+0000","last_clean":"2023-07-14T19:29:02.298983+0000","last_became_active":"2023-07-14T19:29:01.307166+0000","last_became_peered":"2023-07-14T19:29:01.307166+0000","last_unstale":"2023-07-14T19:29:02.298983+0000","last_undegraded":"2023-07-14T19:29:02.298983+0000","last_fullsized":"2023-07-14T19:29:02.298983+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:29:00.604983+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:29:00.604983+0000","last_clean_scrub_stamp":"2023-07-14T19:29:00.604983+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T06:29:57.949181+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":12,"reported_epoch":83,"state":"active+clean","last_fresh":"2023-07-14T19:29:02.299844+0000","last_change":"2023-07-14T19:29:01.308122+0000","last_active":"2023-07-14T19:29:02.299844+0000","last_peered":"2023-07-14T19:29:02.299844+0000","last_clean":"2023-07-14T19:29:02.299844+0000","last_became_active":"2023-07-14T19:29:01.307568+0000","last_became_peered":"2023-07-14T19:29:01.307568+0000","last_unstale":"2023-07-14T19:29:02.299844+0000","last_undegraded":"2023-07-14T19:29:02.299844+0000","last_fullsized":"2023-07-14T19:29:02.299844+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:29:00.604983+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:29:00.604983+0000","last_clean_scrub_stamp":"2023-07-14T19:29:00.604983+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T04:08:56.378075+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":12,"reported_epoch":83,"state":"active+clean","last_fresh":"2023-07-14T19:29:02.444527+0000","last_change":"2023-07-14T19:29:01.314261+0000","last_active":"2023-07-14T19:29:02.444527+0000","last_peered":"2023-07-14T19:29:02.444527+0000","last_clean":"2023-07-14T19:29:02.444527+0000","last_became_active":"2023-07-14T19:29:01.313967+0000","last_became_peered":"2023-07-14T19:29:01.313967+0000","last_unstale":"2023-07-14T19:29:02.444527+0000","last_undegraded":"2023-07-14T19:29:02.444527+0000","last_fullsized":"2023-07-14T19:29:02.444527+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:29:00.604983+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:29:00.604983+0000","last_clean_scrub_stamp":"2023-07-14T19:29:00.604983+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T04:19:07.923742+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":12,"reported_epoch":83,"state":"active+clean","last_fresh":"2023-07-14T19:29:02.573578+0000","last_change":"2023-07-14T19:29:01.319605+0000","last_active":"2023-07-14T19:29:02.573578+0000","last_peered":"2023-07-14T19:29:02.573578+0000","last_clean":"2023-07-14T19:29:02.573578+0000","last_became_active":"2023-07-14T19:29:01.319310+0000","last_became_peered":"2023-07-14T19:29:01.319310+0000","last_unstale":"2023-07-14T19:29:02.573578+0000","last_undegraded":"2023-07-14T19:29:02.573578+0000","last_fullsized":"2023-07-14T19:29:02.573578+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:29:00.604983+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:29:00.604983+0000","last_clean_scrub_stamp":"2023-07-14T19:29:00.604983+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T20:14:29.714925+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":4751360,"data_stored":4722944,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":4616,"num_objects":39,"num_object_clones":0,"num_object_copies":117,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":39,"num_whiteouts":0,"num_read":5,"num_read_kb":4,"num_write":77,"num_write_kb":35,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":167936,"data_stored":11392,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":71,"ondisk_log_size":71,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":78,"seq":335007449099,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27892,"kb_used_data":872,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717260,"statfs":{"total":95995035648,"available":95966474240,"internally_reserved":0,"allocated":892928,"data_stored":428457,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":27654411},"hb_peers":[0,1,2,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.5}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.77200000000000002}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.749}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.37}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.68700000000000006}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.66700000000000004}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.72599999999999998}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.92300000000000004}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.89500000000000002}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.872}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.84199999999999997}]}]},{"osd":10,"up_from":70,"seq":300647710736,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28488,"kb_used_data":1460,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716664,"statfs":{"total":95995035648,"available":95965863936,"internally_reserved":0,"allocated":1495040,"data_stored":1011682,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":27654411},"hb_peers":[1,2,3,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.52600000000000002}]},{"osd":1,"last update":"Fri Jul 14 19:29:00 2023","interfaces":[{"interface":"back","average":{"1min":0.44800000000000001,"5min":0.44800000000000001,"15min":0.44800000000000001},"min":{"1min":0.27000000000000002,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":0.73299999999999998,"5min":0.73299999999999998,"15min":0.73299999999999998},"last":0.33900000000000002},{"interface":"front","average":{"1min":0.46800000000000003,"5min":0.46800000000000003,"15min":0.46800000000000003},"min":{"1min":0.28599999999999998,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":0.80800000000000005,"5min":0.80800000000000005,"15min":0.80800000000000005},"last":0.40100000000000002}]},{"osd":2,"last update":"Fri Jul 14 19:29:00 2023","interfaces":[{"interface":"back","average":{"1min":0.59899999999999998,"5min":0.59899999999999998,"15min":0.59899999999999998},"min":{"1min":0.40200000000000002,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":0.83699999999999997,"5min":0.83699999999999997,"15min":0.83699999999999997},"last":0.68700000000000006},{"interface":"front","average":{"1min":0.64300000000000002,"5min":0.64300000000000002,"15min":0.64300000000000002},"min":{"1min":0.46800000000000003,"5min":0.46800000000000003,"15min":0.46800000000000003},"max":{"1min":0.89100000000000001,"5min":0.89100000000000001,"15min":0.89100000000000001},"last":0.75600000000000001}]},{"osd":3,"last update":"Fri Jul 14 19:29:00 2023","interfaces":[{"interface":"back","average":{"1min":0.58199999999999996,"5min":0.58199999999999996,"15min":0.58199999999999996},"min":{"1min":0.35999999999999999,"5min":0.35999999999999999,"15min":0.35999999999999999},"max":{"1min":0.77800000000000002,"5min":0.77800000000000002,"15min":0.77800000000000002},"last":0.56999999999999995},{"interface":"front","average":{"1min":0.63200000000000001,"5min":0.63200000000000001,"15min":0.63200000000000001},"min":{"1min":0.45200000000000001,"5min":0.45200000000000001,"15min":0.45200000000000001},"max":{"1min":0.89700000000000002,"5min":0.89700000000000002,"15min":0.89700000000000002},"last":0.47899999999999998}]},{"osd":4,"last update":"Fri Jul 14 19:29:00 2023","interfaces":[{"interface":"back","average":{"1min":0.55600000000000005,"5min":0.55600000000000005,"15min":0.55600000000000005},"min":{"1min":0.316,"5min":0.316,"15min":0.316},"max":{"1min":0.99399999999999999,"5min":0.99399999999999999,"15min":0.99399999999999999},"last":0.37},{"interface":"front","average":{"1min":0.54800000000000004,"5min":0.54800000000000004,"15min":0.54800000000000004},"min":{"1min":0.29299999999999998,"5min":0.29299999999999998,"15min":0.29299999999999998},"max":{"1min":0.94999999999999996,"5min":0.94999999999999996,"15min":0.94999999999999996},"last":0.42399999999999999}]},{"osd":5,"last update":"Fri Jul 14 19:29:00 2023","interfaces":[{"interface":"back","average":{"1min":0.68600000000000005,"5min":0.68600000000000005,"15min":0.68600000000000005},"min":{"1min":0.45300000000000001,"5min":0.45300000000000001,"15min":0.45300000000000001},"max":{"1min":1.0129999999999999,"5min":1.0129999999999999,"15min":1.0129999999999999},"last":0.77500000000000002},{"interface":"front","average":{"1min":0.67100000000000004,"5min":0.67100000000000004,"15min":0.67100000000000004},"min":{"1min":0.374,"5min":0.374,"15min":0.374},"max":{"1min":0.96499999999999997,"5min":0.96499999999999997,"15min":0.96499999999999997},"last":0.60699999999999998}]},{"osd":6,"last update":"Fri Jul 14 19:29:00 2023","interfaces":[{"interface":"back","average":{"1min":0.65700000000000003,"5min":0.65700000000000003,"15min":0.65700000000000003},"min":{"1min":0.40400000000000003,"5min":0.40400000000000003,"15min":0.40400000000000003},"max":{"1min":1.032,"5min":1.032,"15min":1.032},"last":0.51800000000000002},{"interface":"front","average":{"1min":0.68799999999999994,"5min":0.68799999999999994,"15min":0.68799999999999994},"min":{"1min":0.45700000000000002,"5min":0.45700000000000002,"15min":0.45700000000000002},"max":{"1min":1.0649999999999999,"5min":1.0649999999999999,"15min":1.0649999999999999},"last":0.50800000000000001}]},{"osd":7,"last update":"Fri Jul 14 19:29:00 2023","interfaces":[{"interface":"back","average":{"1min":0.66000000000000003,"5min":0.66000000000000003,"15min":0.66000000000000003},"min":{"1min":0.34200000000000003,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":1.0960000000000001,"5min":1.0960000000000001,"15min":1.0960000000000001},"last":0.46000000000000002},{"interface":"front","average":{"1min":0.69399999999999995,"5min":0.69399999999999995,"15min":0.69399999999999995},"min":{"1min":0.44600000000000001,"5min":0.44600000000000001,"15min":0.44600000000000001},"max":{"1min":0.97899999999999998,"5min":0.97899999999999998,"15min":0.97899999999999998},"last":0.44600000000000001}]},{"osd":8,"last update":"Fri Jul 14 19:29:00 2023","interfaces":[{"interface":"back","average":{"1min":0.75900000000000001,"5min":0.75900000000000001,"15min":0.75900000000000001},"min":{"1min":0.52800000000000002,"5min":0.52800000000000002,"15min":0.52800000000000002},"max":{"1min":1.143,"5min":1.143,"15min":1.143},"last":0.65200000000000002},{"interface":"front","average":{"1min":0.77600000000000002,"5min":0.77600000000000002,"15min":0.77600000000000002},"min":{"1min":0.47599999999999998,"5min":0.47599999999999998,"15min":0.47599999999999998},"max":{"1min":1.1060000000000001,"5min":1.1060000000000001,"15min":1.1060000000000001},"last":0.71199999999999997}]},{"osd":9,"last update":"Fri Jul 14 19:29:00 2023","interfaces":[{"interface":"back","average":{"1min":0.754,"5min":0.754,"15min":0.754},"min":{"1min":0.54500000000000004,"5min":0.54500000000000004,"15min":0.54500000000000004},"max":{"1min":1.115,"5min":1.115,"15min":1.115},"last":0.55200000000000005},{"interface":"front","average":{"1min":0.77000000000000002,"5min":0.77000000000000002,"15min":0.77000000000000002},"min":{"1min":0.49299999999999999,"5min":0.49299999999999999,"15min":0.49299999999999999},"max":{"1min":1.127,"5min":1.127,"15min":1.127},"last":0.49299999999999999}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.69699999999999995}]}]},{"osd":9,"up_from":65,"seq":279172874260,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27896,"kb_used_data":876,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717256,"statfs":{"total":95995035648,"available":95966470144,"internally_reserved":0,"allocated":897024,"data_stored":429039,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":27654411},"hb_peers":[0,1,2,3,4,5,6,7,8,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jul 14 19:28:37 2023","interfaces":[{"interface":"back","average":{"1min":0.434,"5min":0.434,"15min":0.434},"min":{"1min":0.28000000000000003,"5min":0.28000000000000003,"15min":0.28000000000000003},"max":{"1min":0.54800000000000004,"5min":0.54800000000000004,"15min":0.54800000000000004},"last":0.35399999999999998},{"interface":"front","average":{"1min":0.47199999999999998,"5min":0.47199999999999998,"15min":0.47199999999999998},"min":{"1min":0.26100000000000001,"5min":0.26100000000000001,"15min":0.26100000000000001},"max":{"1min":0.76400000000000001,"5min":0.76400000000000001,"15min":0.76400000000000001},"last":0.39500000000000002}]},{"osd":1,"last update":"Fri Jul 14 19:28:37 2023","interfaces":[{"interface":"back","average":{"1min":0.63200000000000001,"5min":0.63200000000000001,"15min":0.63200000000000001},"min":{"1min":0.45000000000000001,"5min":0.45000000000000001,"15min":0.45000000000000001},"max":{"1min":0.82199999999999995,"5min":0.82199999999999995,"15min":0.82199999999999995},"last":0.55000000000000004},{"interface":"front","average":{"1min":0.54800000000000004,"5min":0.54800000000000004,"15min":0.54800000000000004},"min":{"1min":0.437,"5min":0.437,"15min":0.437},"max":{"1min":0.70899999999999996,"5min":0.70899999999999996,"15min":0.70899999999999996},"last":0.57899999999999996}]},{"osd":2,"last update":"Fri Jul 14 19:28:37 2023","interfaces":[{"interface":"back","average":{"1min":0.63,"5min":0.63,"15min":0.63},"min":{"1min":0.49099999999999999,"5min":0.49099999999999999,"15min":0.49099999999999999},"max":{"1min":0.75800000000000001,"5min":0.75800000000000001,"15min":0.75800000000000001},"last":0.49399999999999999},{"interface":"front","average":{"1min":0.58899999999999997,"5min":0.58899999999999997,"15min":0.58899999999999997},"min":{"1min":0.373,"5min":0.373,"15min":0.373},"max":{"1min":0.72399999999999998,"5min":0.72399999999999998,"15min":0.72399999999999998},"last":0.378}]},{"osd":3,"last update":"Fri Jul 14 19:28:37 2023","interfaces":[{"interface":"back","average":{"1min":0.51900000000000002,"5min":0.51900000000000002,"15min":0.51900000000000002},"min":{"1min":0.35099999999999998,"5min":0.35099999999999998,"15min":0.35099999999999998},"max":{"1min":0.71399999999999997,"5min":0.71399999999999997,"15min":0.71399999999999997},"last":0.51100000000000001},{"interface":"front","average":{"1min":0.58399999999999996,"5min":0.58399999999999996,"15min":0.58399999999999996},"min":{"1min":0.33900000000000002,"5min":0.33900000000000002,"15min":0.33900000000000002},"max":{"1min":0.85699999999999998,"5min":0.85699999999999998,"15min":0.85699999999999998},"last":0.318}]},{"osd":4,"last update":"Fri Jul 14 19:28:37 2023","interfaces":[{"interface":"back","average":{"1min":0.59599999999999997,"5min":0.59599999999999997,"15min":0.59599999999999997},"min":{"1min":0.499,"5min":0.499,"15min":0.499},"max":{"1min":0.77200000000000002,"5min":0.77200000000000002,"15min":0.77200000000000002},"last":0.45300000000000001},{"interface":"front","average":{"1min":0.66000000000000003,"5min":0.66000000000000003,"15min":0.66000000000000003},"min":{"1min":0.47699999999999998,"5min":0.47699999999999998,"15min":0.47699999999999998},"max":{"1min":0.84499999999999997,"5min":0.84499999999999997,"15min":0.84499999999999997},"last":0.628}]},{"osd":5,"last update":"Fri Jul 14 19:28:37 2023","interfaces":[{"interface":"back","average":{"1min":0.67200000000000004,"5min":0.67200000000000004,"15min":0.67200000000000004},"min":{"1min":0.54000000000000004,"5min":0.54000000000000004,"15min":0.54000000000000004},"max":{"1min":0.872,"5min":0.872,"15min":0.872},"last":0.56399999999999995},{"interface":"front","average":{"1min":0.73899999999999999,"5min":0.73899999999999999,"15min":0.73899999999999999},"min":{"1min":0.52500000000000002,"5min":0.52500000000000002,"15min":0.52500000000000002},"max":{"1min":0.90600000000000003,"5min":0.90600000000000003,"15min":0.90600000000000003},"last":0.46500000000000002}]},{"osd":6,"last update":"Fri Jul 14 19:28:37 2023","interfaces":[{"interface":"back","average":{"1min":0.69399999999999995,"5min":0.69399999999999995,"15min":0.69399999999999995},"min":{"1min":0.40699999999999997,"5min":0.40699999999999997,"15min":0.40699999999999997},"max":{"1min":0.89900000000000002,"5min":0.89900000000000002,"15min":0.89900000000000002},"last":0.42599999999999999},{"interface":"front","average":{"1min":0.60399999999999998,"5min":0.60399999999999998,"15min":0.60399999999999998},"min":{"1min":0.38800000000000001,"5min":0.38800000000000001,"15min":0.38800000000000001},"max":{"1min":0.86399999999999999,"5min":0.86399999999999999,"15min":0.86399999999999999},"last":0.41499999999999998}]},{"osd":7,"last update":"Fri Jul 14 19:28:37 2023","interfaces":[{"interface":"back","average":{"1min":0.75800000000000001,"5min":0.75800000000000001,"15min":0.75800000000000001},"min":{"1min":0.58199999999999996,"5min":0.58199999999999996,"15min":0.58199999999999996},"max":{"1min":0.96799999999999997,"5min":0.96799999999999997,"15min":0.96799999999999997},"last":0.65500000000000003},{"interface":"front","average":{"1min":0.77500000000000002,"5min":0.77500000000000002,"15min":0.77500000000000002},"min":{"1min":0.52700000000000002,"5min":0.52700000000000002,"15min":0.52700000000000002},"max":{"1min":0.95299999999999996,"5min":0.95299999999999996,"15min":0.95299999999999996},"last":0.61699999999999999}]},{"osd":8,"last update":"Fri Jul 14 19:28:37 2023","interfaces":[{"interface":"back","average":{"1min":0.71699999999999997,"5min":0.71699999999999997,"15min":0.71699999999999997},"min":{"1min":0.53700000000000003,"5min":0.53700000000000003,"15min":0.53700000000000003},"max":{"1min":0.91300000000000003,"5min":0.91300000000000003,"15min":0.91300000000000003},"last":0.53800000000000003},{"interface":"front","average":{"1min":0.74199999999999999,"5min":0.74199999999999999,"15min":0.74199999999999999},"min":{"1min":0.54500000000000004,"5min":0.54500000000000004,"15min":0.54500000000000004},"max":{"1min":0.89500000000000002,"5min":0.89500000000000002,"15min":0.89500000000000002},"last":0.68400000000000005}]},{"osd":10,"last update":"Fri Jul 14 19:29:00 2023","interfaces":[{"interface":"back","average":{"1min":0.70299999999999996,"5min":0.70299999999999996,"15min":0.70299999999999996},"min":{"1min":0.439,"5min":0.439,"15min":0.439},"max":{"1min":0.95199999999999996,"5min":0.95199999999999996,"15min":0.95199999999999996},"last":0.60499999999999998},{"interface":"front","average":{"1min":0.73799999999999999,"5min":0.73799999999999999,"15min":0.73799999999999999},"min":{"1min":0.47799999999999998,"5min":0.47799999999999998,"15min":0.47799999999999998},"max":{"1min":1.016,"5min":1.016,"15min":1.016},"last":0.72699999999999998}]}]},{"osd":8,"up_from":57,"seq":244813135896,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27896,"kb_used_data":880,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717256,"statfs":{"total":95995035648,"available":95966470144,"internally_reserved":0,"allocated":901120,"data_stored":428802,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":27654411},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jul 14 19:28:18 2023","interfaces":[{"interface":"back","average":{"1min":0.55500000000000005,"5min":0.55500000000000005,"15min":0.55500000000000005},"min":{"1min":0.30099999999999999,"5min":0.30099999999999999,"15min":0.30099999999999999},"max":{"1min":0.73699999999999999,"5min":0.73699999999999999,"15min":0.73699999999999999},"last":0.39300000000000002},{"interface":"front","average":{"1min":0.54500000000000004,"5min":0.54500000000000004,"15min":0.54500000000000004},"min":{"1min":0.32100000000000001,"5min":0.32100000000000001,"15min":0.32100000000000001},"max":{"1min":0.755,"5min":0.755,"15min":0.755},"last":0.46100000000000002}]},{"osd":1,"last update":"Fri Jul 14 19:28:18 2023","interfaces":[{"interface":"back","average":{"1min":0.58199999999999996,"5min":0.58199999999999996,"15min":0.58199999999999996},"min":{"1min":0.35199999999999998,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":0.85499999999999998,"5min":0.85499999999999998,"15min":0.85499999999999998},"last":0.66100000000000003},{"interface":"front","average":{"1min":0.60399999999999998,"5min":0.60399999999999998,"15min":0.60399999999999998},"min":{"1min":0.34499999999999997,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":0.93400000000000005,"5min":0.93400000000000005,"15min":0.93400000000000005},"last":0.55900000000000005}]},{"osd":2,"last update":"Fri Jul 14 19:28:18 2023","interfaces":[{"interface":"back","average":{"1min":0.47199999999999998,"5min":0.47199999999999998,"15min":0.47199999999999998},"min":{"1min":0.27700000000000002,"5min":0.27700000000000002,"15min":0.27700000000000002},"max":{"1min":0.67200000000000004,"5min":0.67200000000000004,"15min":0.67200000000000004},"last":0.42999999999999999},{"interface":"front","average":{"1min":0.45500000000000002,"5min":0.45500000000000002,"15min":0.45500000000000002},"min":{"1min":0.25600000000000001,"5min":0.25600000000000001,"15min":0.25600000000000001},"max":{"1min":0.61399999999999999,"5min":0.61399999999999999,"15min":0.61399999999999999},"last":0.48699999999999999}]},{"osd":3,"last update":"Fri Jul 14 19:28:18 2023","interfaces":[{"interface":"back","average":{"1min":0.57199999999999995,"5min":0.57199999999999995,"15min":0.57199999999999995},"min":{"1min":0.36199999999999999,"5min":0.36199999999999999,"15min":0.36199999999999999},"max":{"1min":0.77000000000000002,"5min":0.77000000000000002,"15min":0.77000000000000002},"last":0.68799999999999994},{"interface":"front","average":{"1min":0.58399999999999996,"5min":0.58399999999999996,"15min":0.58399999999999996},"min":{"1min":0.41499999999999998,"5min":0.41499999999999998,"15min":0.41499999999999998},"max":{"1min":0.77700000000000002,"5min":0.77700000000000002,"15min":0.77700000000000002},"last":0.54300000000000004}]},{"osd":4,"last update":"Fri Jul 14 19:28:18 2023","interfaces":[{"interface":"back","average":{"1min":0.60399999999999998,"5min":0.60399999999999998,"15min":0.60399999999999998},"min":{"1min":0.41699999999999998,"5min":0.41699999999999998,"15min":0.41699999999999998},"max":{"1min":0.86399999999999999,"5min":0.86399999999999999,"15min":0.86399999999999999},"last":0.78800000000000003},{"interface":"front","average":{"1min":0.63700000000000001,"5min":0.63700000000000001,"15min":0.63700000000000001},"min":{"1min":0.39500000000000002,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":0.88,"5min":0.88,"15min":0.88},"last":0.56899999999999995}]},{"osd":5,"last update":"Fri Jul 14 19:28:18 2023","interfaces":[{"interface":"back","average":{"1min":0.55100000000000005,"5min":0.55100000000000005,"15min":0.55100000000000005},"min":{"1min":0.311,"5min":0.311,"15min":0.311},"max":{"1min":0.81000000000000005,"5min":0.81000000000000005,"15min":0.81000000000000005},"last":0.52800000000000002},{"interface":"front","average":{"1min":0.56599999999999995,"5min":0.56599999999999995,"15min":0.56599999999999995},"min":{"1min":0.34799999999999998,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":0.90900000000000003,"5min":0.90900000000000003,"15min":0.90900000000000003},"last":0.752}]},{"osd":6,"last update":"Fri Jul 14 19:28:18 2023","interfaces":[{"interface":"back","average":{"1min":0.65400000000000003,"5min":0.65400000000000003,"15min":0.65400000000000003},"min":{"1min":0.436,"5min":0.436,"15min":0.436},"max":{"1min":0.873,"5min":0.873,"15min":0.873},"last":0.72699999999999998},{"interface":"front","average":{"1min":0.68000000000000005,"5min":0.68000000000000005,"15min":0.68000000000000005},"min":{"1min":0.45600000000000002,"5min":0.45600000000000002,"15min":0.45600000000000002},"max":{"1min":1.1479999999999999,"5min":1.1479999999999999,"15min":1.1479999999999999},"last":0.59499999999999997}]},{"osd":7,"last update":"Fri Jul 14 19:28:18 2023","interfaces":[{"interface":"back","average":{"1min":0.72599999999999998,"5min":0.72599999999999998,"15min":0.72599999999999998},"min":{"1min":0.53400000000000003,"5min":0.53400000000000003,"15min":0.53400000000000003},"max":{"1min":1.2070000000000001,"5min":1.2070000000000001,"15min":1.2070000000000001},"last":0.63700000000000001},{"interface":"front","average":{"1min":0.71199999999999997,"5min":0.71199999999999997,"15min":0.71199999999999997},"min":{"1min":0.47499999999999998,"5min":0.47499999999999998,"15min":0.47499999999999998},"max":{"1min":1.127,"5min":1.127,"15min":1.127},"last":0.83799999999999997}]},{"osd":9,"last update":"Fri Jul 14 19:28:39 2023","interfaces":[{"interface":"back","average":{"1min":0.68000000000000005,"5min":0.68000000000000005,"15min":0.68000000000000005},"min":{"1min":0.497,"5min":0.497,"15min":0.497},"max":{"1min":0.997,"5min":0.997,"15min":0.997},"last":0.81499999999999995},{"interface":"front","average":{"1min":0.68400000000000005,"5min":0.68400000000000005,"15min":0.68400000000000005},"min":{"1min":0.48299999999999998,"5min":0.48299999999999998,"15min":0.48299999999999998},"max":{"1min":0.86299999999999999,"5min":0.86299999999999999,"15min":0.86299999999999999},"last":0.77000000000000002}]},{"osd":10,"last update":"Fri Jul 14 19:28:58 2023","interfaces":[{"interface":"back","average":{"1min":0.69599999999999995,"5min":0.69599999999999995,"15min":0.69599999999999995},"min":{"1min":0.55800000000000005,"5min":0.55800000000000005,"15min":0.55800000000000005},"max":{"1min":0.94799999999999995,"5min":0.94799999999999995,"15min":0.94799999999999995},"last":0.88900000000000001},{"interface":"front","average":{"1min":0.72299999999999998,"5min":0.72299999999999998,"15min":0.72299999999999998},"min":{"1min":0.55200000000000005,"5min":0.55200000000000005,"15min":0.55200000000000005},"max":{"1min":0.91100000000000003,"5min":0.91100000000000003,"15min":0.91100000000000003},"last":0.82799999999999996}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.86399999999999999}]}]},{"osd":7,"up_from":50,"seq":214748364828,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28484,"kb_used_data":1464,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716668,"statfs":{"total":95995035648,"available":95965868032,"internally_reserved":0,"allocated":1499136,"data_stored":1019981,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":27654411},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jul 14 19:28:59 2023","interfaces":[{"interface":"back","average":{"1min":0.60799999999999998,"5min":0.57299999999999995,"15min":0.56699999999999995},"min":{"1min":0.41599999999999998,"5min":0.40400000000000003,"15min":0.40400000000000003},"max":{"1min":0.95899999999999996,"5min":1.0349999999999999,"15min":1.0349999999999999},"last":0.45100000000000001},{"interface":"front","average":{"1min":0.56499999999999995,"5min":0.54200000000000004,"15min":0.53800000000000003},"min":{"1min":0.34899999999999998,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":1.0089999999999999,"5min":1.0089999999999999,"15min":1.0089999999999999},"last":0.40999999999999998}]},{"osd":1,"last update":"Fri Jul 14 19:28:59 2023","interfaces":[{"interface":"back","average":{"1min":0.47299999999999998,"5min":0.45600000000000002,"15min":0.45300000000000001},"min":{"1min":0.27600000000000002,"5min":0.27600000000000002,"15min":0.27600000000000002},"max":{"1min":0.76000000000000001,"5min":0.76000000000000001,"15min":0.76000000000000001},"last":0.35599999999999998},{"interface":"front","average":{"1min":0.48899999999999999,"5min":0.47699999999999998,"15min":0.47499999999999998},"min":{"1min":0.251,"5min":0.251,"15min":0.251},"max":{"1min":0.81999999999999995,"5min":0.81999999999999995,"15min":0.81999999999999995},"last":0.48199999999999998}]},{"osd":2,"last update":"Fri Jul 14 19:28:59 2023","interfaces":[{"interface":"back","average":{"1min":0.64700000000000002,"5min":0.621,"15min":0.61699999999999999},"min":{"1min":0.41399999999999998,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":1.091,"5min":1.091,"15min":1.091},"last":0.64400000000000002},{"interface":"front","average":{"1min":0.63800000000000001,"5min":0.61799999999999999,"15min":0.61499999999999999},"min":{"1min":0.35199999999999998,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":1.1040000000000001,"5min":1.1040000000000001,"15min":1.1040000000000001},"last":0.67700000000000005}]},{"osd":3,"last update":"Fri Jul 14 19:28:59 2023","interfaces":[{"interface":"back","average":{"1min":0.624,"5min":0.57999999999999996,"15min":0.57299999999999995},"min":{"1min":0.45100000000000001,"5min":0.38800000000000001,"15min":0.38800000000000001},"max":{"1min":0.90200000000000002,"5min":1.0109999999999999,"15min":1.0109999999999999},"last":0.52900000000000003},{"interface":"front","average":{"1min":0.64900000000000002,"5min":0.621,"15min":0.61599999999999999},"min":{"1min":0.495,"5min":0.47299999999999998,"15min":0.47299999999999998},"max":{"1min":0.88500000000000001,"5min":0.88600000000000001,"15min":0.88600000000000001},"last":0.56200000000000006}]},{"osd":4,"last update":"Fri Jul 14 19:28:59 2023","interfaces":[{"interface":"back","average":{"1min":0.60799999999999998,"5min":0.58899999999999997,"15min":0.58599999999999997},"min":{"1min":0.36099999999999999,"5min":0.36099999999999999,"15min":0.36099999999999999},"max":{"1min":1.125,"5min":1.125,"15min":1.125},"last":0.41399999999999998},{"interface":"front","average":{"1min":0.58799999999999997,"5min":0.54200000000000004,"15min":0.53400000000000003},"min":{"1min":0.33600000000000002,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":0.879,"5min":0.879,"15min":0.879},"last":0.39500000000000002}]},{"osd":5,"last update":"Fri Jul 14 19:28:59 2023","interfaces":[{"interface":"back","average":{"1min":0.69899999999999995,"5min":0.68300000000000005,"15min":0.68000000000000005},"min":{"1min":0.38300000000000001,"5min":0.38300000000000001,"15min":0.38300000000000001},"max":{"1min":1.173,"5min":1.173,"15min":1.173},"last":0.66500000000000004},{"interface":"front","average":{"1min":0.69299999999999995,"5min":0.69299999999999995,"15min":0.69299999999999995},"min":{"1min":0.40200000000000002,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":1.0329999999999999,"5min":1.0329999999999999,"15min":1.0329999999999999},"last":0.628}]},{"osd":6,"last update":"Fri Jul 14 19:28:59 2023","interfaces":[{"interface":"back","average":{"1min":0.72699999999999998,"5min":0.68799999999999994,"15min":0.68200000000000005},"min":{"1min":0.48499999999999999,"5min":0.42799999999999999,"15min":0.42799999999999999},"max":{"1min":1.0309999999999999,"5min":1.0309999999999999,"15min":1.0309999999999999},"last":0.60199999999999998},{"interface":"front","average":{"1min":0.71899999999999997,"5min":0.69299999999999995,"15min":0.68899999999999995},"min":{"1min":0.50800000000000001,"5min":0.443,"15min":0.443},"max":{"1min":0.93999999999999995,"5min":1.0209999999999999,"15min":1.0209999999999999},"last":0.45300000000000001}]},{"osd":8,"last update":"Fri Jul 14 19:28:18 2023","interfaces":[{"interface":"back","average":{"1min":0.754,"5min":0.754,"15min":0.754},"min":{"1min":0.56499999999999995,"5min":0.56499999999999995,"15min":0.56499999999999995},"max":{"1min":1.0720000000000001,"5min":1.0720000000000001,"15min":1.0720000000000001},"last":0.73599999999999999},{"interface":"front","average":{"1min":0.753,"5min":0.753,"15min":0.753},"min":{"1min":0.55400000000000005,"5min":0.55400000000000005,"15min":0.55400000000000005},"max":{"1min":1.141,"5min":1.141,"15min":1.141},"last":0.61699999999999999}]},{"osd":9,"last update":"Fri Jul 14 19:28:39 2023","interfaces":[{"interface":"back","average":{"1min":0.74099999999999999,"5min":0.74099999999999999,"15min":0.74099999999999999},"min":{"1min":0.53400000000000003,"5min":0.53400000000000003,"15min":0.53400000000000003},"max":{"1min":1.0489999999999999,"5min":1.0489999999999999,"15min":1.0489999999999999},"last":0.505},{"interface":"front","average":{"1min":0.75900000000000001,"5min":0.75900000000000001,"15min":0.75900000000000001},"min":{"1min":0.53300000000000003,"5min":0.53300000000000003,"15min":0.53300000000000003},"max":{"1min":1.1519999999999999,"5min":1.1519999999999999,"15min":1.1519999999999999},"last":0.58299999999999996}]},{"osd":10,"last update":"Fri Jul 14 19:28:59 2023","interfaces":[{"interface":"back","average":{"1min":0.72999999999999998,"5min":0.72999999999999998,"15min":0.72999999999999998},"min":{"1min":0.38700000000000001,"5min":0.38700000000000001,"15min":0.38700000000000001},"max":{"1min":1.077,"5min":1.077,"15min":1.077},"last":0.69199999999999995},{"interface":"front","average":{"1min":0.74199999999999999,"5min":0.74199999999999999,"15min":0.74199999999999999},"min":{"1min":0.38500000000000001,"5min":0.38500000000000001,"15min":0.38500000000000001},"max":{"1min":1.117,"5min":1.117,"15min":1.117},"last":0.65200000000000002}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.76600000000000001}]}]},{"osd":6,"up_from":45,"seq":193273528353,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27892,"kb_used_data":872,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717260,"statfs":{"total":95995035648,"available":95966474240,"internally_reserved":0,"allocated":892928,"data_stored":428457,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":27654411},"hb_peers":[0,1,2,3,4,5,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jul 14 19:28:36 2023","interfaces":[{"interface":"back","average":{"1min":0.42099999999999999,"5min":0.38200000000000001,"15min":0.376},"min":{"1min":0.29099999999999998,"5min":0.23400000000000001,"15min":0.23400000000000001},"max":{"1min":0.63,"5min":0.63,"15min":0.63},"last":0.34399999999999997},{"interface":"front","average":{"1min":0.42699999999999999,"5min":0.38900000000000001,"15min":0.38300000000000001},"min":{"1min":0.32000000000000001,"5min":0.249,"15min":0.249},"max":{"1min":0.63900000000000001,"5min":0.63900000000000001,"15min":0.63900000000000001},"last":0.307}]},{"osd":1,"last update":"Fri Jul 14 19:28:36 2023","interfaces":[{"interface":"back","average":{"1min":0.66400000000000003,"5min":0.53600000000000003,"15min":0.51400000000000001},"min":{"1min":0.41699999999999998,"5min":0.29899999999999999,"15min":0.29899999999999999},"max":{"1min":0.83999999999999997,"5min":0.83999999999999997,"15min":0.83999999999999997},"last":0.51500000000000001},{"interface":"front","average":{"1min":0.58199999999999996,"5min":0.505,"15min":0.49199999999999999},"min":{"1min":0.36699999999999999,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":0.872,"5min":0.872,"15min":0.872},"last":0.435}]},{"osd":2,"last update":"Fri Jul 14 19:28:36 2023","interfaces":[{"interface":"back","average":{"1min":0.58399999999999996,"5min":0.58799999999999997,"15min":0.58899999999999997},"min":{"1min":0.40600000000000003,"5min":0.27600000000000002,"15min":0.27600000000000002},"max":{"1min":0.68799999999999994,"5min":0.76800000000000002,"15min":0.76800000000000002},"last":0.46899999999999997},{"interface":"front","average":{"1min":0.628,"5min":0.61199999999999999,"15min":0.60899999999999999},"min":{"1min":0.46500000000000002,"5min":0.378,"15min":0.378},"max":{"1min":0.83299999999999996,"5min":0.83299999999999996,"15min":0.83299999999999996},"last":0.621}]},{"osd":3,"last update":"Fri Jul 14 19:28:36 2023","interfaces":[{"interface":"back","average":{"1min":0.47399999999999998,"5min":0.47599999999999998,"15min":0.47599999999999998},"min":{"1min":0.30099999999999999,"5min":0.30099999999999999,"15min":0.30099999999999999},"max":{"1min":0.64200000000000002,"5min":0.78500000000000003,"15min":0.78500000000000003},"last":0.55700000000000005},{"interface":"front","average":{"1min":0.45100000000000001,"5min":0.441,"15min":0.439},"min":{"1min":0.315,"5min":0.28999999999999998,"15min":0.28999999999999998},"max":{"1min":0.75700000000000001,"5min":0.75700000000000001,"15min":0.75700000000000001},"last":0.35799999999999998}]},{"osd":4,"last update":"Fri Jul 14 19:28:36 2023","interfaces":[{"interface":"back","average":{"1min":0.61099999999999999,"5min":0.53500000000000003,"15min":0.52200000000000002},"min":{"1min":0.39500000000000002,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":0.82899999999999996,"5min":0.82899999999999996,"15min":0.82899999999999996},"last":0.40300000000000002},{"interface":"front","average":{"1min":0.67300000000000004,"5min":0.62,"15min":0.61099999999999999},"min":{"1min":0.38700000000000001,"5min":0.38700000000000001,"15min":0.38700000000000001},"max":{"1min":0.96899999999999997,"5min":0.96899999999999997,"15min":0.96899999999999997},"last":0.45300000000000001}]},{"osd":5,"last update":"Fri Jul 14 19:28:36 2023","interfaces":[{"interface":"back","average":{"1min":0.56799999999999995,"5min":0.57399999999999995,"15min":0.57499999999999996},"min":{"1min":0.38300000000000001,"5min":0.38300000000000001,"15min":0.38300000000000001},"max":{"1min":0.752,"5min":0.81799999999999995,"15min":0.81799999999999995},"last":0.56899999999999995},{"interface":"front","average":{"1min":0.64400000000000002,"5min":0.63500000000000001,"15min":0.63300000000000001},"min":{"1min":0.47399999999999998,"5min":0.46000000000000002,"15min":0.46000000000000002},"max":{"1min":0.81399999999999995,"5min":0.88500000000000001,"15min":0.88500000000000001},"last":0.54000000000000004}]},{"osd":7,"last update":"Fri Jul 14 19:28:59 2023","interfaces":[{"interface":"back","average":{"1min":0.68600000000000005,"5min":0.68100000000000005,"15min":0.68000000000000005},"min":{"1min":0.47599999999999998,"5min":0.44500000000000001,"15min":0.44500000000000001},"max":{"1min":1.0029999999999999,"5min":1.0029999999999999,"15min":1.0029999999999999},"last":0.48699999999999999},{"interface":"front","average":{"1min":0.69699999999999995,"5min":0.69699999999999995,"15min":0.69699999999999995},"min":{"1min":0.50900000000000001,"5min":0.46700000000000003,"15min":0.46700000000000003},"max":{"1min":0.86599999999999999,"5min":1.0189999999999999,"15min":1.0189999999999999},"last":0.41899999999999998}]},{"osd":8,"last update":"Fri Jul 14 19:28:17 2023","interfaces":[{"interface":"back","average":{"1min":0.70999999999999996,"5min":0.70999999999999996,"15min":0.70999999999999996},"min":{"1min":0.50900000000000001,"5min":0.50900000000000001,"15min":0.50900000000000001},"max":{"1min":1.1000000000000001,"5min":1.1000000000000001,"15min":1.1000000000000001},"last":0.60299999999999998},{"interface":"front","average":{"1min":0.59799999999999998,"5min":0.59799999999999998,"15min":0.59799999999999998},"min":{"1min":0.434,"5min":0.434,"15min":0.434},"max":{"1min":0.754,"5min":0.754,"15min":0.754},"last":0.53300000000000003}]},{"osd":9,"last update":"Fri Jul 14 19:28:41 2023","interfaces":[{"interface":"back","average":{"1min":0.65100000000000002,"5min":0.65100000000000002,"15min":0.65100000000000002},"min":{"1min":0.434,"5min":0.434,"15min":0.434},"max":{"1min":0.90600000000000003,"5min":0.90600000000000003,"15min":0.90600000000000003},"last":0.58699999999999997},{"interface":"front","average":{"1min":0.76900000000000002,"5min":0.76900000000000002,"15min":0.76900000000000002},"min":{"1min":0.51100000000000001,"5min":0.51100000000000001,"15min":0.51100000000000001},"max":{"1min":1.1379999999999999,"5min":1.1379999999999999,"15min":1.1379999999999999},"last":0.67100000000000004}]},{"osd":10,"last update":"Fri Jul 14 19:28:59 2023","interfaces":[{"interface":"back","average":{"1min":0.63900000000000001,"5min":0.63900000000000001,"15min":0.63900000000000001},"min":{"1min":0.505,"5min":0.505,"15min":0.505},"max":{"1min":0.82999999999999996,"5min":0.82999999999999996,"15min":0.82999999999999996},"last":0.498},{"interface":"front","average":{"1min":0.76800000000000002,"5min":0.76800000000000002,"15min":0.76800000000000002},"min":{"1min":0.58999999999999997,"5min":0.58999999999999997,"15min":0.58999999999999997},"max":{"1min":1.0329999999999999,"5min":1.0329999999999999,"15min":1.0329999999999999},"last":0.64100000000000001}]}]},{"osd":1,"up_from":13,"seq":55834574902,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27912,"kb_used_data":884,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717240,"statfs":{"total":95995035648,"available":95966453760,"internally_reserved":0,"allocated":905216,"data_stored":430021,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":27654411},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jul 14 19:28:58 2023","interfaces":[{"interface":"back","average":{"1min":0.53800000000000003,"5min":0.498,"15min":0.47099999999999997},"min":{"1min":0.43099999999999999,"5min":0.215,"15min":0.215},"max":{"1min":0.82399999999999995,"5min":1.6970000000000001,"15min":1.6970000000000001},"last":0.42699999999999999},{"interface":"front","average":{"1min":0.58599999999999997,"5min":0.48699999999999999,"15min":0.46100000000000002},"min":{"1min":0.39100000000000001,"5min":0.23699999999999999,"15min":0.23699999999999999},"max":{"1min":0.82899999999999996,"5min":0.82899999999999996,"15min":0.82899999999999996},"last":0.51000000000000001}]},{"osd":2,"last update":"Fri Jul 14 19:28:13 2023","interfaces":[{"interface":"back","average":{"1min":0.53900000000000003,"5min":0.496,"15min":0.48699999999999999},"min":{"1min":0.34799999999999998,"5min":0.31,"15min":0.31},"max":{"1min":0.77500000000000002,"5min":0.77500000000000002,"15min":0.77500000000000002},"last":0.40600000000000003},{"interface":"front","average":{"1min":0.55900000000000005,"5min":0.52400000000000002,"15min":0.498},"min":{"1min":0.317,"5min":0.317,"15min":0.317},"max":{"1min":0.76700000000000002,"5min":1.74,"15min":1.74},"last":0.55300000000000005}]},{"osd":3,"last update":"Fri Jul 14 19:28:33 2023","interfaces":[{"interface":"back","average":{"1min":0.59099999999999997,"5min":0.55400000000000005,"15min":0.53600000000000003},"min":{"1min":0.35499999999999998,"5min":0.32200000000000001,"15min":0.32200000000000001},"max":{"1min":0.73899999999999999,"5min":1.8460000000000001,"15min":1.8460000000000001},"last":0.53300000000000003},{"interface":"front","average":{"1min":0.62,"5min":0.55900000000000005,"15min":0.54800000000000004},"min":{"1min":0.47499999999999998,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.81499999999999995,"5min":0.81499999999999995,"15min":0.81499999999999995},"last":0.626}]},{"osd":4,"last update":"Fri Jul 14 19:28:58 2023","interfaces":[{"interface":"back","average":{"1min":0.51400000000000001,"5min":0.49199999999999999,"15min":0.502},"min":{"1min":0.314,"5min":0.29899999999999999,"15min":0.29899999999999999},"max":{"1min":0.78100000000000003,"5min":2.2029999999999998,"15min":2.2029999999999998},"last":0.379},{"interface":"front","average":{"1min":0.53700000000000003,"5min":0.495,"15min":0.49199999999999999},"min":{"1min":0.309,"5min":0.26400000000000001,"15min":0.26400000000000001},"max":{"1min":0.90400000000000003,"5min":1.901,"15min":1.901},"last":0.65400000000000003}]},{"osd":5,"last update":"Fri Jul 14 19:28:13 2023","interfaces":[{"interface":"back","average":{"1min":0.626,"5min":0.57399999999999995,"15min":0.56499999999999995},"min":{"1min":0.308,"5min":0.308,"15min":0.308},"max":{"1min":0.78500000000000003,"5min":0.78500000000000003,"15min":0.78500000000000003},"last":0.57199999999999995},{"interface":"front","average":{"1min":0.67700000000000005,"5min":0.65600000000000003,"15min":0.65200000000000002},"min":{"1min":0.442,"5min":0.42699999999999999,"15min":0.42699999999999999},"max":{"1min":0.84299999999999997,"5min":1.8,"15min":1.8},"last":0.64000000000000001}]},{"osd":6,"last update":"Fri Jul 14 19:28:39 2023","interfaces":[{"interface":"back","average":{"1min":0.64700000000000002,"5min":0.63500000000000001,"15min":0.63300000000000001},"min":{"1min":0.45000000000000001,"5min":0.40999999999999998,"15min":0.40999999999999998},"max":{"1min":0.91500000000000004,"5min":1.8300000000000001,"15min":1.8300000000000001},"last":0.59299999999999997},{"interface":"front","average":{"1min":0.68500000000000005,"5min":0.66100000000000003,"15min":0.65700000000000003},"min":{"1min":0.46600000000000003,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":1.026,"5min":1.875,"15min":1.875},"last":0.70599999999999996}]},{"osd":7,"last update":"Fri Jul 14 19:28:58 2023","interfaces":[{"interface":"back","average":{"1min":0.67400000000000004,"5min":0.56599999999999995,"15min":0.54800000000000004},"min":{"1min":0.36199999999999999,"5min":0.36199999999999999,"15min":0.36199999999999999},"max":{"1min":0.95399999999999996,"5min":0.95399999999999996,"15min":0.95399999999999996},"last":0.49099999999999999},{"interface":"front","average":{"1min":0.70199999999999996,"5min":0.60799999999999998,"15min":0.59199999999999997},"min":{"1min":0.39200000000000002,"5min":0.38400000000000001,"15min":0.38400000000000001},"max":{"1min":1.24,"5min":1.24,"15min":1.24},"last":0.60899999999999999}]},{"osd":8,"last update":"Fri Jul 14 19:28:16 2023","interfaces":[{"interface":"back","average":{"1min":0.69199999999999995,"5min":0.69199999999999995,"15min":0.69199999999999995},"min":{"1min":0.42199999999999999,"5min":0.42199999999999999,"15min":0.42199999999999999},"max":{"1min":0.93100000000000005,"5min":0.93100000000000005,"15min":0.93100000000000005},"last":0.66200000000000003},{"interface":"front","average":{"1min":0.73999999999999999,"5min":0.73999999999999999,"15min":0.73999999999999999},"min":{"1min":0.505,"5min":0.505,"15min":0.505},"max":{"1min":1.0449999999999999,"5min":1.0449999999999999,"15min":1.0449999999999999},"last":0.79800000000000004}]},{"osd":9,"last update":"Fri Jul 14 19:28:39 2023","interfaces":[{"interface":"back","average":{"1min":0.77600000000000002,"5min":0.77600000000000002,"15min":0.77600000000000002},"min":{"1min":0.52100000000000002,"5min":0.52100000000000002,"15min":0.52100000000000002},"max":{"1min":1.042,"5min":1.042,"15min":1.042},"last":0.72299999999999998},{"interface":"front","average":{"1min":0.76100000000000001,"5min":0.76100000000000001,"15min":0.76100000000000001},"min":{"1min":0.51200000000000001,"5min":0.51200000000000001,"15min":0.51200000000000001},"max":{"1min":1.175,"5min":1.175,"15min":1.175},"last":0.751}]},{"osd":10,"last update":"Fri Jul 14 19:28:58 2023","interfaces":[{"interface":"back","average":{"1min":0.73699999999999999,"5min":0.73699999999999999,"15min":0.73699999999999999},"min":{"1min":0.44800000000000001,"5min":0.44800000000000001,"15min":0.44800000000000001},"max":{"1min":1.079,"5min":1.079,"15min":1.079},"last":0.69299999999999995},{"interface":"front","average":{"1min":0.81799999999999995,"5min":0.81799999999999995,"15min":0.81799999999999995},"min":{"1min":0.58799999999999997,"5min":0.58799999999999997,"15min":0.58799999999999997},"max":{"1min":1.127,"5min":1.127,"15min":1.127},"last":0.67700000000000005}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.73599999999999999}]}]},{"osd":0,"up_from":8,"seq":34359738426,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27904,"kb_used_data":884,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717248,"statfs":{"total":95995035648,"available":95966461952,"internally_reserved":0,"allocated":905216,"data_stored":429151,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":27654411},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Jul 14 19:28:52 2023","interfaces":[{"interface":"back","average":{"1min":0.629,"5min":0.52600000000000002,"15min":0.503},"min":{"1min":0.35999999999999999,"5min":0.28000000000000003,"15min":0.28000000000000003},"max":{"1min":1.79,"5min":1.79,"15min":1.79},"last":0.41199999999999998},{"interface":"front","average":{"1min":0.57199999999999995,"5min":0.51700000000000002,"15min":0.505},"min":{"1min":0.32000000000000001,"5min":0.313,"15min":0.313},"max":{"1min":1.605,"5min":1.605,"15min":1.605},"last":0.40200000000000002}]},{"osd":2,"last update":"Fri Jul 14 19:28:17 2023","interfaces":[{"interface":"back","average":{"1min":0.64700000000000002,"5min":0.54500000000000004,"15min":0.51700000000000002},"min":{"1min":0.40699999999999997,"5min":0.34599999999999997,"15min":0.34599999999999997},"max":{"1min":0.77700000000000002,"5min":0.77700000000000002,"15min":0.77700000000000002},"last":0.47699999999999998},{"interface":"front","average":{"1min":0.60799999999999998,"5min":0.54100000000000004,"15min":0.52100000000000002},"min":{"1min":0.35899999999999999,"5min":0.29999999999999999,"15min":0.29999999999999999},"max":{"1min":0.81399999999999995,"5min":0.81399999999999995,"15min":0.81399999999999995},"last":0.38600000000000001}]},{"osd":3,"last update":"Fri Jul 14 19:27:35 2023","interfaces":[{"interface":"back","average":{"1min":0.439,"5min":0.41099999999999998,"15min":0.40600000000000003},"min":{"1min":0.29399999999999998,"5min":0.26400000000000001,"15min":0.26400000000000001},"max":{"1min":0.72299999999999998,"5min":0.72299999999999998,"15min":0.72299999999999998},"last":0.36299999999999999},{"interface":"front","average":{"1min":0.44900000000000001,"5min":0.437,"15min":0.435},"min":{"1min":0.245,"5min":0.24099999999999999,"15min":0.24099999999999999},"max":{"1min":0.61499999999999999,"5min":0.61499999999999999,"15min":0.61499999999999999},"last":0.32900000000000001}]},{"osd":4,"last update":"Fri Jul 14 19:29:02 2023","interfaces":[{"interface":"back","average":{"1min":0.57099999999999995,"5min":0.53900000000000003,"15min":0.52500000000000002},"min":{"1min":0.38800000000000001,"5min":0.377,"15min":0.377},"max":{"1min":1.139,"5min":1.139,"15min":1.139},"last":0.44800000000000001},{"interface":"front","average":{"1min":0.58499999999999996,"5min":0.55000000000000004,"15min":0.53700000000000003},"min":{"1min":0.34799999999999998,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":1.2190000000000001,"5min":1.2190000000000001,"15min":1.2190000000000001},"last":0.46000000000000002}]},{"osd":5,"last update":"Fri Jul 14 19:28:17 2023","interfaces":[{"interface":"back","average":{"1min":0.626,"5min":0.60499999999999998,"15min":0.60099999999999998},"min":{"1min":0.47999999999999998,"5min":0.45900000000000002,"15min":0.45900000000000002},"max":{"1min":0.81799999999999995,"5min":0.81799999999999995,"15min":0.81799999999999995},"last":0.57099999999999995},{"interface":"front","average":{"1min":0.64400000000000002,"5min":0.61599999999999999,"15min":0.61099999999999999},"min":{"1min":0.45600000000000002,"5min":0.436,"15min":0.436},"max":{"1min":0.85499999999999998,"5min":0.85499999999999998,"15min":0.85499999999999998},"last":0.624}]},{"osd":6,"last update":"Fri Jul 14 19:28:36 2023","interfaces":[{"interface":"back","average":{"1min":0.48899999999999999,"5min":0.495,"15min":0.496},"min":{"1min":0.25900000000000001,"5min":0.25900000000000001,"15min":0.25900000000000001},"max":{"1min":0.80000000000000004,"5min":0.80000000000000004,"15min":0.80000000000000004},"last":0.50700000000000001},{"interface":"front","average":{"1min":0.55300000000000005,"5min":0.51400000000000001,"15min":0.50800000000000001},"min":{"1min":0.32500000000000001,"5min":0.32500000000000001,"15min":0.32500000000000001},"max":{"1min":0.74199999999999999,"5min":0.76300000000000001,"15min":0.76300000000000001},"last":0.42799999999999999}]},{"osd":7,"last update":"Fri Jul 14 19:27:57 2023","interfaces":[{"interface":"back","average":{"1min":0.60299999999999998,"5min":0.60299999999999998,"15min":0.60299999999999998},"min":{"1min":0.41999999999999998,"5min":0.41999999999999998,"15min":0.41999999999999998},"max":{"1min":0.83599999999999997,"5min":0.83599999999999997,"15min":0.83599999999999997},"last":0.52100000000000002},{"interface":"front","average":{"1min":0.65200000000000002,"5min":0.65200000000000002,"15min":0.65200000000000002},"min":{"1min":0.47299999999999998,"5min":0.47299999999999998,"15min":0.47299999999999998},"max":{"1min":0.89500000000000002,"5min":0.89500000000000002,"15min":0.89500000000000002},"last":0.40799999999999997}]},{"osd":8,"last update":"Fri Jul 14 19:28:17 2023","interfaces":[{"interface":"back","average":{"1min":0.69299999999999995,"5min":0.69299999999999995,"15min":0.69299999999999995},"min":{"1min":0.57599999999999996,"5min":0.57599999999999996,"15min":0.57599999999999996},"max":{"1min":0.80700000000000005,"5min":0.80700000000000005,"15min":0.80700000000000005},"last":0.53500000000000003},{"interface":"front","average":{"1min":0.71399999999999997,"5min":0.71399999999999997,"15min":0.71399999999999997},"min":{"1min":0.53400000000000003,"5min":0.53400000000000003,"15min":0.53400000000000003},"max":{"1min":0.86499999999999999,"5min":0.86499999999999999,"15min":0.86499999999999999},"last":0.55500000000000005}]},{"osd":9,"last update":"Fri Jul 14 19:28:39 2023","interfaces":[{"interface":"back","average":{"1min":0.63500000000000001,"5min":0.63500000000000001,"15min":0.63500000000000001},"min":{"1min":0.36499999999999999,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":0.85299999999999998,"5min":0.85299999999999998,"15min":0.85299999999999998},"last":0.54500000000000004},{"interface":"front","average":{"1min":0.64200000000000002,"5min":0.64200000000000002,"15min":0.64200000000000002},"min":{"1min":0.32300000000000001,"5min":0.32300000000000001,"15min":0.32300000000000001},"max":{"1min":0.96099999999999997,"5min":0.96099999999999997,"15min":0.96099999999999997},"last":0.48899999999999999}]},{"osd":10,"last update":"Fri Jul 14 19:29:02 2023","interfaces":[{"interface":"back","average":{"1min":0.754,"5min":0.754,"15min":0.754},"min":{"1min":0.45900000000000002,"5min":0.45900000000000002,"15min":0.45900000000000002},"max":{"1min":1.776,"5min":1.776,"15min":1.776},"last":0.52400000000000002},{"interface":"front","average":{"1min":0.72499999999999998,"5min":0.72499999999999998,"15min":0.72499999999999998},"min":{"1min":0.502,"5min":0.502,"15min":0.502},"max":{"1min":1.522,"5min":1.522,"15min":1.522},"last":0.58099999999999996}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.60399999999999998}]}]},{"osd":2,"up_from":18,"seq":77309411377,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27916,"kb_used_data":888,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717236,"statfs":{"total":95995035648,"available":95966449664,"internally_reserved":0,"allocated":909312,"data_stored":430055,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":27654411},"hb_peers":[0,1,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jul 14 19:28:16 2023","interfaces":[{"interface":"back","average":{"1min":0.53600000000000003,"5min":0.503,"15min":0.49199999999999999},"min":{"1min":0.34399999999999997,"5min":0.312,"15min":0.312},"max":{"1min":0.89000000000000001,"5min":0.89000000000000001,"15min":0.89000000000000001},"last":0.75600000000000001},{"interface":"front","average":{"1min":0.51300000000000001,"5min":0.49099999999999999,"15min":0.48699999999999999},"min":{"1min":0.31900000000000001,"5min":0.31900000000000001,"15min":0.31900000000000001},"max":{"1min":0.65100000000000002,"5min":0.69699999999999995,"15min":0.69699999999999995},"last":0.47699999999999998}]},{"osd":1,"last update":"Fri Jul 14 19:28:16 2023","interfaces":[{"interface":"back","average":{"1min":0.628,"5min":0.53300000000000003,"15min":0.51500000000000001},"min":{"1min":0.40000000000000002,"5min":0.316,"15min":0.316},"max":{"1min":0.89300000000000002,"5min":0.89300000000000002,"15min":0.89300000000000002},"last":0.54000000000000004},{"interface":"front","average":{"1min":0.64600000000000002,"5min":0.54600000000000004,"15min":0.52200000000000002},"min":{"1min":0.379,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":0.91100000000000003,"5min":0.91100000000000003,"15min":0.91100000000000003},"last":0.86499999999999999}]},{"osd":3,"last update":"Fri Jul 14 19:28:36 2023","interfaces":[{"interface":"back","average":{"1min":0.67900000000000005,"5min":0.57399999999999995,"15min":0.55400000000000005},"min":{"1min":0.48199999999999998,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":0.91700000000000004,"5min":0.91700000000000004,"15min":0.91700000000000004},"last":0.82999999999999996},{"interface":"front","average":{"1min":0.60299999999999998,"5min":0.54300000000000004,"15min":0.53300000000000003},"min":{"1min":0.44,"5min":0.34699999999999998,"15min":0.34699999999999998},"max":{"1min":0.76900000000000002,"5min":0.76900000000000002,"15min":0.76900000000000002},"last":0.67100000000000004}]},{"osd":4,"last update":"Fri Jul 14 19:27:59 2023","interfaces":[{"interface":"back","average":{"1min":0.53800000000000003,"5min":0.52200000000000002,"15min":0.52000000000000002},"min":{"1min":0.38800000000000001,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":0.68400000000000005,"5min":0.70399999999999996,"15min":0.70399999999999996},"last":0.68500000000000005},{"interface":"front","average":{"1min":0.63600000000000001,"5min":0.58399999999999996,"15min":0.57599999999999996},"min":{"1min":0.40100000000000002,"5min":0.40100000000000002,"15min":0.40100000000000002},"max":{"1min":0.81699999999999995,"5min":0.81699999999999995,"15min":0.81699999999999995},"last":0.63500000000000001}]},{"osd":5,"last update":"Fri Jul 14 19:28:16 2023","interfaces":[{"interface":"back","average":{"1min":0.47399999999999998,"5min":0.433,"15min":0.42599999999999999},"min":{"1min":0.308,"5min":0.26500000000000001,"15min":0.26500000000000001},"max":{"1min":0.66900000000000004,"5min":0.66900000000000004,"15min":0.66900000000000004},"last":0.39300000000000002},{"interface":"front","average":{"1min":0.54500000000000004,"5min":0.47699999999999998,"15min":0.46600000000000003},"min":{"1min":0.34999999999999998,"5min":0.313,"15min":0.313},"max":{"1min":0.77900000000000003,"5min":0.77900000000000003,"15min":0.77900000000000003},"last":0.72099999999999997}]},{"osd":6,"last update":"Fri Jul 14 19:28:43 2023","interfaces":[{"interface":"back","average":{"1min":0.59899999999999998,"5min":0.55500000000000005,"15min":0.54800000000000004},"min":{"1min":0.33500000000000002,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":0.93400000000000005,"5min":0.93400000000000005,"15min":0.93400000000000005},"last":0.65400000000000003},{"interface":"front","average":{"1min":0.69199999999999995,"5min":0.60899999999999999,"15min":0.59499999999999997},"min":{"1min":0.46999999999999997,"5min":0.371,"15min":0.371},"max":{"1min":0.92000000000000004,"5min":0.92000000000000004,"15min":0.92000000000000004},"last":0.61399999999999999}]},{"osd":7,"last update":"Fri Jul 14 19:27:59 2023","interfaces":[{"interface":"back","average":{"1min":0.622,"5min":0.622,"15min":0.622},"min":{"1min":0.44600000000000001,"5min":0.44600000000000001,"15min":0.44600000000000001},"max":{"1min":0.91900000000000004,"5min":0.91900000000000004,"15min":0.91900000000000004},"last":0.59899999999999998},{"interface":"front","average":{"1min":0.67500000000000004,"5min":0.67500000000000004,"15min":0.67500000000000004},"min":{"1min":0.433,"5min":0.433,"15min":0.433},"max":{"1min":0.96399999999999997,"5min":0.96399999999999997,"15min":0.96399999999999997},"last":0.66000000000000003}]},{"osd":8,"last update":"Fri Jul 14 19:28:16 2023","interfaces":[{"interface":"back","average":{"1min":0.627,"5min":0.627,"15min":0.627},"min":{"1min":0.35799999999999998,"5min":0.35799999999999998,"15min":0.35799999999999998},"max":{"1min":0.84899999999999998,"5min":0.84899999999999998,"15min":0.84899999999999998},"last":0.91600000000000004},{"interface":"front","average":{"1min":0.625,"5min":0.625,"15min":0.625},"min":{"1min":0.375,"5min":0.375,"15min":0.375},"max":{"1min":0.94199999999999995,"5min":0.94199999999999995,"15min":0.94199999999999995},"last":0.77600000000000002}]},{"osd":9,"last update":"Fri Jul 14 19:28:43 2023","interfaces":[{"interface":"back","average":{"1min":0.67800000000000005,"5min":0.67800000000000005,"15min":0.67800000000000005},"min":{"1min":0.45300000000000001,"5min":0.45300000000000001,"15min":0.45300000000000001},"max":{"1min":0.90700000000000003,"5min":0.90700000000000003,"15min":0.90700000000000003},"last":0.81499999999999995},{"interface":"front","average":{"1min":0.65300000000000002,"5min":0.65300000000000002,"15min":0.65300000000000002},"min":{"1min":0.55200000000000005,"5min":0.55200000000000005,"15min":0.55200000000000005},"max":{"1min":0.79700000000000004,"5min":0.79700000000000004,"15min":0.79700000000000004},"last":0.74199999999999999}]},{"osd":10,"last update":"Fri Jul 14 19:28:59 2023","interfaces":[{"interface":"back","average":{"1min":0.63200000000000001,"5min":0.63200000000000001,"15min":0.63200000000000001},"min":{"1min":0.46800000000000003,"5min":0.46800000000000003,"15min":0.46800000000000003},"max":{"1min":0.79700000000000004,"5min":0.79700000000000004,"15min":0.79700000000000004},"last":0.70599999999999996},{"interface":"front","average":{"1min":0.745,"5min":0.745,"15min":0.745},"min":{"1min":0.61899999999999999,"5min":0.61899999999999999,"15min":0.61899999999999999},"max":{"1min":0.88900000000000001,"5min":0.88900000000000001,"15min":0.88900000000000001},"last":0.96399999999999997}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.012}]}]},{"osd":3,"up_from":25,"seq":107374182445,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27904,"kb_used_data":876,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717248,"statfs":{"total":95995035648,"available":95966461952,"internally_reserved":0,"allocated":897024,"data_stored":428230,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":27654411},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jul 14 19:28:33 2023","interfaces":[{"interface":"back","average":{"1min":0.41099999999999998,"5min":0.34100000000000003,"15min":0.32300000000000001},"min":{"1min":0.27000000000000002,"5min":0.221,"15min":0.221},"max":{"1min":0.61299999999999999,"5min":0.61299999999999999,"15min":0.61299999999999999},"last":0.379},{"interface":"front","average":{"1min":0.435,"5min":0.375,"15min":0.35699999999999998},"min":{"1min":0.30299999999999999,"5min":0.23899999999999999,"15min":0.23899999999999999},"max":{"1min":0.63600000000000001,"5min":0.63600000000000001,"15min":0.63600000000000001},"last":0.40500000000000003}]},{"osd":1,"last update":"Fri Jul 14 19:28:33 2023","interfaces":[{"interface":"back","average":{"1min":0.54400000000000004,"5min":0.50800000000000001,"15min":0.49399999999999999},"min":{"1min":0.38500000000000001,"5min":0.36199999999999999,"15min":0.36199999999999999},"max":{"1min":0.84199999999999997,"5min":0.84199999999999997,"15min":0.84199999999999997},"last":0.52800000000000002},{"interface":"front","average":{"1min":0.56999999999999995,"5min":0.52000000000000002,"15min":0.502},"min":{"1min":0.40200000000000002,"5min":0.30399999999999999,"15min":0.30399999999999999},"max":{"1min":0.71099999999999997,"5min":0.71099999999999997,"15min":0.71099999999999997},"last":0.70399999999999996}]},{"osd":2,"last update":"Fri Jul 14 19:28:33 2023","interfaces":[{"interface":"back","average":{"1min":0.65600000000000003,"5min":0.56899999999999995,"15min":0.54100000000000004},"min":{"1min":0.48699999999999999,"5min":0.36199999999999999,"15min":0.36199999999999999},"max":{"1min":0.93899999999999995,"5min":0.93899999999999995,"15min":0.93899999999999995},"last":0.72299999999999998},{"interface":"front","average":{"1min":0.58899999999999997,"5min":0.54500000000000004,"15min":0.52700000000000002},"min":{"1min":0.441,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":0.74199999999999999,"5min":0.78300000000000003,"15min":0.78300000000000003},"last":0.76200000000000001}]},{"osd":4,"last update":"Fri Jul 14 19:28:51 2023","interfaces":[{"interface":"back","average":{"1min":0.66100000000000003,"5min":0.55600000000000005,"15min":0.52200000000000002},"min":{"1min":0.44800000000000001,"5min":0.32300000000000001,"15min":0.32300000000000001},"max":{"1min":0.91100000000000003,"5min":0.98499999999999999,"15min":0.98499999999999999},"last":0.68600000000000005},{"interface":"front","average":{"1min":0.60499999999999998,"5min":0.55000000000000004,"15min":0.52900000000000003},"min":{"1min":0.45500000000000002,"5min":0.39300000000000002,"15min":0.39300000000000002},"max":{"1min":0.90200000000000002,"5min":0.90200000000000002,"15min":0.90200000000000002},"last":0.63700000000000001}]},{"osd":5,"last update":"Fri Jul 14 19:28:11 2023","interfaces":[{"interface":"back","average":{"1min":0.63100000000000001,"5min":0.57899999999999996,"15min":0.57099999999999995},"min":{"1min":0.44900000000000001,"5min":0.44600000000000001,"15min":0.44600000000000001},"max":{"1min":0.81299999999999994,"5min":0.81299999999999994,"15min":0.81299999999999994},"last":0.60899999999999999},{"interface":"front","average":{"1min":0.69799999999999995,"5min":0.61099999999999999,"15min":0.59699999999999998},"min":{"1min":0.48199999999999998,"5min":0.39300000000000002,"15min":0.39300000000000002},"max":{"1min":0.98199999999999998,"5min":0.98199999999999998,"15min":0.98199999999999998},"last":0.81999999999999995}]},{"osd":6,"last update":"Fri Jul 14 19:28:38 2023","interfaces":[{"interface":"back","average":{"1min":0.61299999999999999,"5min":0.501,"15min":0.48199999999999998},"min":{"1min":0.39100000000000001,"5min":0.34699999999999998,"15min":0.34699999999999998},"max":{"1min":0.82599999999999996,"5min":0.82599999999999996,"15min":0.82599999999999996},"last":0.58299999999999996},{"interface":"front","average":{"1min":0.51500000000000001,"5min":0.52000000000000002,"15min":0.52100000000000002},"min":{"1min":0.36499999999999999,"5min":0.36299999999999999,"15min":0.36299999999999999},"max":{"1min":0.67300000000000004,"5min":0.67300000000000004,"15min":0.67300000000000004},"last":0.44800000000000001}]},{"osd":7,"last update":"Fri Jul 14 19:28:57 2023","interfaces":[{"interface":"back","average":{"1min":0.69699999999999995,"5min":0.68000000000000005,"15min":0.67700000000000005},"min":{"1min":0.48999999999999999,"5min":0.41699999999999998,"15min":0.41699999999999998},"max":{"1min":0.96399999999999997,"5min":1.196,"15min":1.196},"last":0.73699999999999999},{"interface":"front","average":{"1min":0.78700000000000003,"5min":0.69199999999999995,"15min":0.67600000000000005},"min":{"1min":0.58499999999999996,"5min":0.46100000000000002,"15min":0.46100000000000002},"max":{"1min":1.0289999999999999,"5min":1.0289999999999999,"15min":1.0289999999999999},"last":0.85699999999999998}]},{"osd":8,"last update":"Fri Jul 14 19:28:17 2023","interfaces":[{"interface":"back","average":{"1min":0.69899999999999995,"5min":0.69899999999999995,"15min":0.69899999999999995},"min":{"1min":0.48199999999999998,"5min":0.48199999999999998,"15min":0.48199999999999998},"max":{"1min":0.98799999999999999,"5min":0.98799999999999999,"15min":0.98799999999999999},"last":0.751},{"interface":"front","average":{"1min":0.749,"5min":0.749,"15min":0.749},"min":{"1min":0.51800000000000002,"5min":0.51800000000000002,"15min":0.51800000000000002},"max":{"1min":0.92500000000000004,"5min":0.92500000000000004,"15min":0.92500000000000004},"last":0.68100000000000005}]},{"osd":9,"last update":"Fri Jul 14 19:28:38 2023","interfaces":[{"interface":"back","average":{"1min":0.76600000000000001,"5min":0.76600000000000001,"15min":0.76600000000000001},"min":{"1min":0.42099999999999999,"5min":0.42099999999999999,"15min":0.42099999999999999},"max":{"1min":1.0549999999999999,"5min":1.0549999999999999,"15min":1.0549999999999999},"last":0.79100000000000004},{"interface":"front","average":{"1min":0.64900000000000002,"5min":0.64900000000000002,"15min":0.64900000000000002},"min":{"1min":0.40799999999999997,"5min":0.40799999999999997,"15min":0.40799999999999997},"max":{"1min":0.95099999999999996,"5min":0.95099999999999996,"15min":0.95099999999999996},"last":0.49099999999999999}]},{"osd":10,"last update":"Fri Jul 14 19:29:02 2023","interfaces":[{"interface":"back","average":{"1min":0.72699999999999998,"5min":0.72699999999999998,"15min":0.72699999999999998},"min":{"1min":0.56499999999999995,"5min":0.56499999999999995,"15min":0.56499999999999995},"max":{"1min":0.92700000000000005,"5min":0.92700000000000005,"15min":0.92700000000000005},"last":0.83399999999999996},{"interface":"front","average":{"1min":0.82099999999999995,"5min":0.82099999999999995,"15min":0.82099999999999995},"min":{"1min":0.59399999999999997,"5min":0.59399999999999997,"15min":0.59399999999999997},"max":{"1min":1.0880000000000001,"5min":1.0880000000000001,"15min":1.0880000000000001},"last":0.89700000000000002}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.80800000000000005}]}]},{"osd":4,"up_from":32,"seq":137438953513,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27892,"kb_used_data":876,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717260,"statfs":{"total":95995035648,"available":95966474240,"internally_reserved":0,"allocated":897024,"data_stored":428479,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":27654411},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jul 14 19:28:54 2023","interfaces":[{"interface":"back","average":{"1min":0.57599999999999996,"5min":0.52300000000000002,"15min":0.50900000000000001},"min":{"1min":0.39200000000000002,"5min":0.27400000000000002,"15min":0.27400000000000002},"max":{"1min":0.85299999999999998,"5min":0.85299999999999998,"15min":0.85299999999999998},"last":0.496},{"interface":"front","average":{"1min":0.56799999999999995,"5min":0.51700000000000002,"15min":0.497},"min":{"1min":0.35599999999999998,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":0.77600000000000002,"5min":0.90800000000000003,"15min":0.90800000000000003},"last":0.40699999999999997}]},{"osd":1,"last update":"Fri Jul 14 19:28:54 2023","interfaces":[{"interface":"back","average":{"1min":0.42799999999999999,"5min":0.378,"15min":0.35899999999999999},"min":{"1min":0.28699999999999998,"5min":0.224,"15min":0.224},"max":{"1min":0.77200000000000002,"5min":0.77200000000000002,"15min":0.77200000000000002},"last":0.40400000000000003},{"interface":"front","average":{"1min":0.48699999999999999,"5min":0.40500000000000003,"15min":0.38700000000000001},"min":{"1min":0.33800000000000002,"5min":0.247,"15min":0.247},"max":{"1min":0.72999999999999998,"5min":0.72999999999999998,"15min":0.72999999999999998},"last":0.29799999999999999}]},{"osd":2,"last update":"Fri Jul 14 19:28:54 2023","interfaces":[{"interface":"back","average":{"1min":0.61099999999999999,"5min":0.59799999999999998,"15min":0.59599999999999997},"min":{"1min":0.373,"5min":0.32800000000000001,"15min":0.32800000000000001},"max":{"1min":0.83799999999999997,"5min":0.86399999999999999,"15min":0.86399999999999999},"last":0.63800000000000001},{"interface":"front","average":{"1min":0.65200000000000002,"5min":0.59699999999999998,"15min":0.58299999999999996},"min":{"1min":0.43099999999999999,"5min":0.43099999999999999,"15min":0.43099999999999999},"max":{"1min":0.97099999999999997,"5min":0.97099999999999997,"15min":0.97099999999999997},"last":0.65700000000000003}]},{"osd":3,"last update":"Fri Jul 14 19:28:54 2023","interfaces":[{"interface":"back","average":{"1min":0.61299999999999999,"5min":0.55700000000000005,"15min":0.54400000000000004},"min":{"1min":0.442,"5min":0.36599999999999999,"15min":0.36599999999999999},"max":{"1min":0.98599999999999999,"5min":0.98599999999999999,"15min":0.98599999999999999},"last":0.52400000000000002},{"interface":"front","average":{"1min":0.59399999999999997,"5min":0.54600000000000004,"15min":0.53000000000000003},"min":{"1min":0.36399999999999999,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.876,"5min":0.876,"15min":0.876},"last":0.36699999999999999}]},{"osd":5,"last update":"Fri Jul 14 19:28:12 2023","interfaces":[{"interface":"back","average":{"1min":0.625,"5min":0.59699999999999998,"15min":0.59299999999999997},"min":{"1min":0.40000000000000002,"5min":0.34699999999999998,"15min":0.34699999999999998},"max":{"1min":0.81200000000000006,"5min":0.88200000000000001,"15min":0.88200000000000001},"last":0.73699999999999999},{"interface":"front","average":{"1min":0.61599999999999999,"5min":0.58899999999999997,"15min":0.58499999999999996},"min":{"1min":0.44500000000000001,"5min":0.314,"15min":0.314},"max":{"1min":0.749,"5min":0.97099999999999997,"15min":0.97099999999999997},"last":0.621}]},{"osd":6,"last update":"Fri Jul 14 19:28:36 2023","interfaces":[{"interface":"back","average":{"1min":0.61199999999999999,"5min":0.59199999999999997,"15min":0.58799999999999997},"min":{"1min":0.378,"5min":0.378,"15min":0.378},"max":{"1min":0.79900000000000004,"5min":0.93000000000000005,"15min":0.93000000000000005},"last":0.433},{"interface":"front","average":{"1min":0.66000000000000003,"5min":0.627,"15min":0.621},"min":{"1min":0.42699999999999999,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":0.89500000000000002,"5min":0.89500000000000002,"15min":0.89500000000000002},"last":0.60299999999999998}]},{"osd":7,"last update":"Fri Jul 14 19:28:59 2023","interfaces":[{"interface":"back","average":{"1min":0.61199999999999999,"5min":0.58299999999999996,"15min":0.57799999999999996},"min":{"1min":0.373,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":0.86499999999999999,"5min":0.89800000000000002,"15min":0.89800000000000002},"last":0.54200000000000004},{"interface":"front","average":{"1min":0.59199999999999997,"5min":0.55600000000000005,"15min":0.55000000000000004},"min":{"1min":0.38400000000000001,"5min":0.36199999999999999,"15min":0.36199999999999999},"max":{"1min":0.96899999999999997,"5min":0.96899999999999997,"15min":0.96899999999999997},"last":0.47199999999999998}]},{"osd":8,"last update":"Fri Jul 14 19:28:18 2023","interfaces":[{"interface":"back","average":{"1min":0.66500000000000004,"5min":0.66500000000000004,"15min":0.66500000000000004},"min":{"1min":0.52500000000000002,"5min":0.52500000000000002,"15min":0.52500000000000002},"max":{"1min":0.84499999999999997,"5min":0.84499999999999997,"15min":0.84499999999999997},"last":0.51000000000000001},{"interface":"front","average":{"1min":0.70899999999999996,"5min":0.70899999999999996,"15min":0.70899999999999996},"min":{"1min":0.49099999999999999,"5min":0.49099999999999999,"15min":0.49099999999999999},"max":{"1min":0.95899999999999996,"5min":0.95899999999999996,"15min":0.95899999999999996},"last":0.70499999999999996}]},{"osd":9,"last update":"Fri Jul 14 19:28:37 2023","interfaces":[{"interface":"back","average":{"1min":0.69799999999999995,"5min":0.69799999999999995,"15min":0.69799999999999995},"min":{"1min":0.49299999999999999,"5min":0.49299999999999999,"15min":0.49299999999999999},"max":{"1min":0.878,"5min":0.878,"15min":0.878},"last":0.45200000000000001},{"interface":"front","average":{"1min":0.74199999999999999,"5min":0.74199999999999999,"15min":0.74199999999999999},"min":{"1min":0.52000000000000002,"5min":0.52000000000000002,"15min":0.52000000000000002},"max":{"1min":1.04,"5min":1.04,"15min":1.04},"last":0.76200000000000001}]},{"osd":10,"last update":"Fri Jul 14 19:28:59 2023","interfaces":[{"interface":"back","average":{"1min":0.68300000000000005,"5min":0.68300000000000005,"15min":0.68300000000000005},"min":{"1min":0.47299999999999998,"5min":0.47299999999999998,"15min":0.47299999999999998},"max":{"1min":0.95299999999999996,"5min":0.95299999999999996,"15min":0.95299999999999996},"last":0.58399999999999996},{"interface":"front","average":{"1min":0.68000000000000005,"5min":0.68000000000000005,"15min":0.68000000000000005},"min":{"1min":0.443,"5min":0.443,"15min":0.443},"max":{"1min":0.995,"5min":0.995,"15min":0.995},"last":0.56399999999999995}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.67300000000000004}]}]},{"osd":5,"up_from":37,"seq":158913789989,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28496,"kb_used_data":1480,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716656,"statfs":{"total":95995035648,"available":95965855744,"internally_reserved":0,"allocated":1515520,"data_stored":1020643,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":27654411},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jul 14 19:28:13 2023","interfaces":[{"interface":"back","average":{"1min":0.54100000000000004,"5min":0.47499999999999998,"15min":0.46400000000000002},"min":{"1min":0.34899999999999998,"5min":0.28499999999999998,"15min":0.28499999999999998},"max":{"1min":0.79300000000000004,"5min":0.86399999999999999,"15min":0.86399999999999999},"last":0.66900000000000004},{"interface":"front","average":{"1min":0.55600000000000005,"5min":0.51100000000000001,"15min":0.503},"min":{"1min":0.35699999999999998,"5min":0.32800000000000001,"15min":0.32800000000000001},"max":{"1min":0.70299999999999996,"5min":0.84099999999999997,"15min":0.84099999999999997},"last":0.46899999999999997}]},{"osd":1,"last update":"Fri Jul 14 19:28:13 2023","interfaces":[{"interface":"back","average":{"1min":0.56799999999999995,"5min":0.54100000000000004,"15min":0.53700000000000003},"min":{"1min":0.38900000000000001,"5min":0.38500000000000001,"15min":0.38500000000000001},"max":{"1min":1.0620000000000001,"5min":1.0620000000000001,"15min":1.0620000000000001},"last":0.72599999999999998},{"interface":"front","average":{"1min":0.57099999999999995,"5min":0.55800000000000005,"15min":0.55600000000000005},"min":{"1min":0.40799999999999997,"5min":0.36099999999999999,"15min":0.36099999999999999},"max":{"1min":0.93899999999999995,"5min":0.93899999999999995,"15min":0.93899999999999995},"last":0.74299999999999999}]},{"osd":2,"last update":"Fri Jul 14 19:28:13 2023","interfaces":[{"interface":"back","average":{"1min":0.45400000000000001,"5min":0.442,"15min":0.44},"min":{"1min":0.307,"5min":0.217,"15min":0.217},"max":{"1min":0.67400000000000004,"5min":0.67400000000000004,"15min":0.67400000000000004},"last":0.50600000000000001},{"interface":"front","average":{"1min":0.48999999999999999,"5min":0.45500000000000002,"15min":0.44900000000000001},"min":{"1min":0.313,"5min":0.23699999999999999,"15min":0.23699999999999999},"max":{"1min":0.82499999999999996,"5min":0.82499999999999996,"15min":0.82499999999999996},"last":0.55200000000000005}]},{"osd":3,"last update":"Fri Jul 14 19:28:13 2023","interfaces":[{"interface":"back","average":{"1min":0.57499999999999996,"5min":0.57099999999999995,"15min":0.56999999999999995},"min":{"1min":0.36499999999999999,"5min":0.32200000000000001,"15min":0.32200000000000001},"max":{"1min":0.76400000000000001,"5min":0.90800000000000003,"15min":0.90800000000000003},"last":0.59299999999999997},{"interface":"front","average":{"1min":0.60899999999999999,"5min":0.57699999999999996,"15min":0.57199999999999995},"min":{"1min":0.41099999999999998,"5min":0.33900000000000002,"15min":0.33900000000000002},"max":{"1min":0.85499999999999998,"5min":0.94399999999999995,"15min":0.94399999999999995},"last":0.626}]},{"osd":4,"last update":"Fri Jul 14 19:28:13 2023","interfaces":[{"interface":"back","average":{"1min":0.627,"5min":0.60299999999999998,"15min":0.59899999999999998},"min":{"1min":0.45800000000000002,"5min":0.45800000000000002,"15min":0.45800000000000002},"max":{"1min":0.86899999999999999,"5min":0.86899999999999999,"15min":0.86899999999999999},"last":0.79700000000000004},{"interface":"front","average":{"1min":0.57699999999999996,"5min":0.59899999999999998,"15min":0.60299999999999998},"min":{"1min":0.38,"5min":0.38,"15min":0.38},"max":{"1min":0.745,"5min":0.79500000000000004,"15min":0.79500000000000004},"last":0.65200000000000002}]},{"osd":6,"last update":"Fri Jul 14 19:28:38 2023","interfaces":[{"interface":"back","average":{"1min":0.64900000000000002,"5min":0.60499999999999998,"15min":0.59699999999999998},"min":{"1min":0.443,"5min":0.38300000000000001,"15min":0.38300000000000001},"max":{"1min":0.85799999999999998,"5min":0.85799999999999998,"15min":0.85799999999999998},"last":0.77500000000000002},{"interface":"front","average":{"1min":0.70299999999999996,"5min":0.64500000000000002,"15min":0.63500000000000001},"min":{"1min":0.39900000000000002,"5min":0.36099999999999999,"15min":0.36099999999999999},"max":{"1min":1.0189999999999999,"5min":1.0189999999999999,"15min":1.0189999999999999},"last":0.81000000000000005}]},{"osd":7,"last update":"Fri Jul 14 19:28:59 2023","interfaces":[{"interface":"back","average":{"1min":0.65600000000000003,"5min":0.66400000000000003,"15min":0.66600000000000004},"min":{"1min":0.49099999999999999,"5min":0.49099999999999999,"15min":0.49099999999999999},"max":{"1min":0.88,"5min":0.95399999999999996,"15min":0.95399999999999996},"last":0.82599999999999996},{"interface":"front","average":{"1min":0.66800000000000004,"5min":0.68500000000000005,"15min":0.68799999999999994},"min":{"1min":0.46999999999999997,"5min":0.45800000000000002,"15min":0.45800000000000002},"max":{"1min":0.89500000000000002,"5min":0.96099999999999997,"15min":0.96099999999999997},"last":0.84099999999999997}]},{"osd":8,"last update":"Fri Jul 14 19:28:18 2023","interfaces":[{"interface":"back","average":{"1min":0.65200000000000002,"5min":0.65200000000000002,"15min":0.65200000000000002},"min":{"1min":0.41099999999999998,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":0.997,"5min":0.997,"15min":0.997},"last":0.85599999999999998},{"interface":"front","average":{"1min":0.60599999999999998,"5min":0.60599999999999998,"15min":0.60599999999999998},"min":{"1min":0.38600000000000001,"5min":0.38600000000000001,"15min":0.38600000000000001},"max":{"1min":0.80000000000000004,"5min":0.80000000000000004,"15min":0.80000000000000004},"last":0.68600000000000005}]},{"osd":9,"last update":"Fri Jul 14 19:28:38 2023","interfaces":[{"interface":"back","average":{"1min":0.70299999999999996,"5min":0.70299999999999996,"15min":0.70299999999999996},"min":{"1min":0.50600000000000001,"5min":0.50600000000000001,"15min":0.50600000000000001},"max":{"1min":0.88700000000000001,"5min":0.88700000000000001,"15min":0.88700000000000001},"last":0.76200000000000001},{"interface":"front","average":{"1min":0.77800000000000002,"5min":0.77800000000000002,"15min":0.77800000000000002},"min":{"1min":0.504,"5min":0.504,"15min":0.504},"max":{"1min":1.111,"5min":1.111,"15min":1.111},"last":0.877}]},{"osd":10,"last update":"Fri Jul 14 19:28:59 2023","interfaces":[{"interface":"back","average":{"1min":0.71699999999999997,"5min":0.71699999999999997,"15min":0.71699999999999997},"min":{"1min":0.46899999999999997,"5min":0.46899999999999997,"15min":0.46899999999999997},"max":{"1min":0.96599999999999997,"5min":0.96599999999999997,"15min":0.96599999999999997},"last":0.89000000000000001},{"interface":"front","average":{"1min":0.73199999999999998,"5min":0.73199999999999998,"15min":0.73199999999999998},"min":{"1min":0.54000000000000004,"5min":0.54000000000000004,"15min":0.54000000000000004},"max":{"1min":0.95099999999999996,"5min":0.95099999999999996,"15min":0.95099999999999996},"last":0.90100000000000002}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1478,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1564,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1598,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":596,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1818,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1156,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":738,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1758,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-07-14T19:29:08.936 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-07-14T19:29:08.941 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:29:08 smithi101 ceph-mon[128807]: from='client.14754 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "sub_name": "sv_0", "target": ["mon-mgr", ""]}]: dispatch 2023-07-14T19:29:08.942 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:29:08 smithi101 ceph-mon[128807]: from='client.14760 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_1", "group_name": "", "target": ["mon-mgr", ""]}]: dispatch 2023-07-14T19:29:08.942 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:29:08 smithi101 ceph-mon[128807]: pgmap v256: 97 pgs: 97 active+clean; 581 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 586 B/s rd, 5.0 KiB/s wr, 11 op/s 2023-07-14T19:29:08.942 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:29:08 smithi101 ceph-mon[128807]: mgrmap e17: x(active, since 6m), standbys: z, y 2023-07-14T19:29:08.942 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:29:08 smithi101 ceph-mon[128807]: from='client.14766 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "sub_name": "sv_1", "target": ["mon-mgr", ""]}]: dispatch 2023-07-14T19:29:08.943 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:29:08 smithi101 ceph-mon[128807]: from='client.? 172.21.15.98:0/4260159366' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-07-14T19:29:08.943 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:29:08 smithi101 ceph-mon[128807]: from='client.? 172.21.15.98:0/2261216108' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-07-14T19:29:08.970 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:29:08 smithi139 ceph-mon[129293]: from='client.14754 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "sub_name": "sv_0", "target": ["mon-mgr", ""]}]: dispatch 2023-07-14T19:29:08.971 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:29:08 smithi139 ceph-mon[129293]: from='client.14760 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_1", "group_name": "", "target": ["mon-mgr", ""]}]: dispatch 2023-07-14T19:29:08.971 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:29:08 smithi139 ceph-mon[129293]: pgmap v256: 97 pgs: 97 active+clean; 581 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 586 B/s rd, 5.0 KiB/s wr, 11 op/s 2023-07-14T19:29:08.971 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:29:08 smithi139 ceph-mon[129293]: mgrmap e17: x(active, since 6m), standbys: z, y 2023-07-14T19:29:08.972 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:29:08 smithi139 ceph-mon[129293]: from='client.14766 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "sub_name": "sv_1", "target": ["mon-mgr", ""]}]: dispatch 2023-07-14T19:29:08.972 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:29:08 smithi139 ceph-mon[129293]: from='client.? 172.21.15.98:0/4260159366' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-07-14T19:29:08.972 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:29:08 smithi139 ceph-mon[129293]: from='client.? 172.21.15.98:0/2261216108' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-07-14T19:29:09.088 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:09.084+0000 7f087c598700 1 -- 172.21.15.98:0/2741431869 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f08740ffcf0 msgr2=0x7f0874100140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:29:09.089 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:09.084+0000 7f087c598700 1 --2- 172.21.15.98:0/2741431869 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f08740ffcf0 0x7f0874100140 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f0864009990 tx=0x7f0864009ca0 comp rx=0 tx=0).stop 2023-07-14T19:29:09.089 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:09.084+0000 7f087c598700 1 -- 172.21.15.98:0/2741431869 shutdown_connections 2023-07-14T19:29:09.089 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:09.084+0000 7f087c598700 1 --2- 172.21.15.98:0/2741431869 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f08740feb00 0x7f08740fef10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:29:09.090 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:09.084+0000 7f087c598700 1 --2- 172.21.15.98:0/2741431869 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f0874100680 0x7f0874106f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:29:09.090 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:09.084+0000 7f087c598700 1 --2- 172.21.15.98:0/2741431869 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f08740ffcf0 0x7f0874100140 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:29:09.090 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:09.084+0000 7f087c598700 1 -- 172.21.15.98:0/2741431869 >> 172.21.15.98:0/2741431869 conn(0x7f08740fa100 msgr2=0x7f08740fc550 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:29:09.091 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:09.084+0000 7f087c598700 1 -- 172.21.15.98:0/2741431869 shutdown_connections 2023-07-14T19:29:09.091 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:09.084+0000 7f087c598700 1 -- 172.21.15.98:0/2741431869 wait complete. 2023-07-14T19:29:09.091 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:09.085+0000 7f087c598700 1 Processor -- start 2023-07-14T19:29:09.091 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:09.085+0000 7f087c598700 1 -- start start 2023-07-14T19:29:09.092 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:09.085+0000 7f087c598700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f08740feb00 0x7f0874068310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:29:09.092 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:09.085+0000 7f087c598700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f08740ffcf0 0x7f087419ca80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:29:09.092 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:09.085+0000 7f087c598700 1 --2- >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f0874100680 0x7f087419efd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:29:09.093 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:09.085+0000 7f087c598700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f087419f680 con 0x7f08740ffcf0 2023-07-14T19:29:09.093 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:09.085+0000 7f087c598700 1 -- --> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f087419f7f0 con 0x7f0874100680 2023-07-14T19:29:09.093 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:09.085+0000 7f087c598700 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f087419f9b0 con 0x7f08740feb00 2023-07-14T19:29:09.093 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:09.085+0000 7f0879b33700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f08740ffcf0 0x7f087419ca80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:29:09.094 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:09.085+0000 7f0879b33700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f08740ffcf0 0x7f087419ca80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:45866/0 (socket says 172.21.15.98:45866) 2023-07-14T19:29:09.094 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:09.085+0000 7f0879b33700 1 -- 172.21.15.98:0/2925387841 learned_addr learned my addr 172.21.15.98:0/2925387841 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-07-14T19:29:09.096 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:09.085+0000 7f087ab35700 1 --2- 172.21.15.98:0/2925387841 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f0874100680 0x7f087419efd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:29:09.096 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:09.085+0000 7f0879b33700 1 -- 172.21.15.98:0/2925387841 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f08740feb00 msgr2=0x7f0874068310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:29:09.096 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:09.085+0000 7f0879b33700 1 --2- 172.21.15.98:0/2925387841 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f08740feb00 0x7f0874068310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:29:09.097 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:09.085+0000 7f0879b33700 1 -- 172.21.15.98:0/2925387841 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f0874100680 msgr2=0x7f087419efd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:29:09.097 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:09.085+0000 7f0879b33700 1 --2- 172.21.15.98:0/2925387841 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f0874100680 0x7f087419efd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:29:09.097 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:09.085+0000 7f0879b33700 1 -- 172.21.15.98:0/2925387841 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f08640095f0 con 0x7f08740ffcf0 2023-07-14T19:29:09.098 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:09.085+0000 7f087ab35700 1 --2- 172.21.15.98:0/2925387841 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f0874100680 0x7f087419efd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-07-14T19:29:09.098 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:09.085+0000 7f0879b33700 1 --2- 172.21.15.98:0/2925387841 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f08740ffcf0 0x7f087419ca80 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f0864005910 tx=0x7f0864005e80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:29:09.098 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:09.085+0000 7f086b7fe700 1 -- 172.21.15.98:0/2925387841 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f086401d070 con 0x7f08740ffcf0 2023-07-14T19:29:09.099 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:09.085+0000 7f086b7fe700 1 -- 172.21.15.98:0/2925387841 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f0864004060 con 0x7f08740ffcf0 2023-07-14T19:29:09.099 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:09.085+0000 7f087c598700 1 -- 172.21.15.98:0/2925387841 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f087419fc30 con 0x7f08740ffcf0 2023-07-14T19:29:09.099 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:09.085+0000 7f086b7fe700 1 -- 172.21.15.98:0/2925387841 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0864022e10 con 0x7f08740ffcf0 2023-07-14T19:29:09.099 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:09.085+0000 7f087c598700 1 -- 172.21.15.98:0/2925387841 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f08741a0050 con 0x7f08740ffcf0 2023-07-14T19:29:09.100 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:09.086+0000 7f087c598700 1 -- 172.21.15.98:0/2925387841 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0874106070 con 0x7f08740ffcf0 2023-07-14T19:29:09.101 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:09.087+0000 7f086b7fe700 1 -- 172.21.15.98:0/2925387841 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 17) v1 ==== 135766+0+0 (secure 0 0 0) 0x7f0864022470 con 0x7f08740ffcf0 2023-07-14T19:29:09.101 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:09.088+0000 7f086b7fe700 1 --2- 172.21.15.98:0/2925387841 >> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] conn(0x7f08600a1220 0x7f08600a36d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:29:09.102 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:09.088+0000 7f087a334700 1 --2- 172.21.15.98:0/2925387841 >> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] conn(0x7f08600a1220 0x7f08600a36d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:29:09.102 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:09.088+0000 7f086b7fe700 1 -- 172.21.15.98:0/2925387841 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 7979+0+0 (secure 0 0 0) 0x7f08640ce180 con 0x7f08740ffcf0 2023-07-14T19:29:09.102 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:09.089+0000 7f087a334700 1 --2- 172.21.15.98:0/2925387841 >> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] conn(0x7f08600a1220 0x7f08600a36d0 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f08740ffb50 tx=0x7f0870005ca0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:29:09.103 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:09.092+0000 7f086b7fe700 1 -- 172.21.15.98:0/2925387841 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f0864097a40 con 0x7f08740ffcf0 2023-07-14T19:29:09.277 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:09.276+0000 7f087c598700 1 -- 172.21.15.98:0/2925387841 --> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f0874060a80 con 0x7f08600a1220 2023-07-14T19:29:09.288 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:09.286+0000 7f086b7fe700 1 -- 172.21.15.98:0/2925387841 <== mgr.14152 v2:172.21.15.98:6800/452042164 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+336085 (secure 0 0 0) 0x7f0874060a80 con 0x7f08600a1220 2023-07-14T19:29:09.291 INFO:teuthology.orchestra.run.smithi098.stdout: 2023-07-14T19:29:09.293 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:09.289+0000 7f087c598700 1 -- 172.21.15.98:0/2925387841 >> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] conn(0x7f08600a1220 msgr2=0x7f08600a36d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:29:09.293 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:09.289+0000 7f087c598700 1 --2- 172.21.15.98:0/2925387841 >> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] conn(0x7f08600a1220 0x7f08600a36d0 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f08740ffb50 tx=0x7f0870005ca0 comp rx=0 tx=0).stop 2023-07-14T19:29:09.294 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:09.289+0000 7f087c598700 1 -- 172.21.15.98:0/2925387841 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f08740ffcf0 msgr2=0x7f087419ca80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:29:09.294 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:09.289+0000 7f087c598700 1 --2- 172.21.15.98:0/2925387841 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f08740ffcf0 0x7f087419ca80 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f0864005910 tx=0x7f0864005e80 comp rx=0 tx=0).stop 2023-07-14T19:29:09.294 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:09.289+0000 7f087c598700 1 -- 172.21.15.98:0/2925387841 shutdown_connections 2023-07-14T19:29:09.294 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:09.289+0000 7f087c598700 1 --2- 172.21.15.98:0/2925387841 >> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] conn(0x7f08600a1220 0x7f08600a36d0 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:29:09.294 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:09.289+0000 7f087c598700 1 --2- 172.21.15.98:0/2925387841 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f08740feb00 0x7f0874068310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:29:09.294 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:09.289+0000 7f087c598700 1 --2- 172.21.15.98:0/2925387841 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f0874100680 0x7f087419efd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:29:09.294 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:09.289+0000 7f087c598700 1 --2- 172.21.15.98:0/2925387841 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f08740ffcf0 0x7f087419ca80 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:29:09.295 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:09.289+0000 7f087c598700 1 -- 172.21.15.98:0/2925387841 >> 172.21.15.98:0/2925387841 conn(0x7f08740fa100 msgr2=0x7f08741031b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:29:09.295 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:09.289+0000 7f087c598700 1 -- 172.21.15.98:0/2925387841 shutdown_connections 2023-07-14T19:29:09.295 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:09.289+0000 7f087c598700 1 -- 172.21.15.98:0/2925387841 wait complete. 2023-07-14T19:29:09.295 INFO:teuthology.orchestra.run.smithi098.stderr:dumped all 2023-07-14T19:29:09.303 INFO:teuthology.orchestra.run.smithi098.stdout:{"pg_ready":true,"pg_map":{"version":256,"stamp":"2023-07-14T19:29:07.592616+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":594984,"num_objects":41,"num_object_clones":0,"num_object_copies":123,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":41,"num_whiteouts":0,"num_read":141,"num_read_kb":123,"num_write":258,"num_write_kb":2973,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":208,"ondisk_log_size":208,"up":291,"acting":291,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":269,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":336572,"kb_used_data":12312,"kb_used_omap":20,"kb_used_meta":324075,"kb_avail":1124605252,"statfs":{"total":1151940427776,"available":1151595778048,"internally_reserved":0,"allocated":12607488,"data_stored":6912997,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":21372,"internal_metadata":331852932},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":14,"apply_latency_ms":14,"commit_latency_ns":14000000,"apply_latency_ns":14000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":4616,"num_objects":39,"num_object_clones":0,"num_object_copies":117,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":39,"num_whiteouts":0,"num_read":5,"num_read_kb":4,"num_write":77,"num_write_kb":35,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.985142"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":12,"reported_epoch":83,"state":"active+clean","last_fresh":"2023-07-14T19:29:02.573808+0000","last_change":"2023-07-14T19:29:01.313813+0000","last_active":"2023-07-14T19:29:02.573808+0000","last_peered":"2023-07-14T19:29:02.573808+0000","last_clean":"2023-07-14T19:29:02.573808+0000","last_became_active":"2023-07-14T19:29:01.313514+0000","last_became_peered":"2023-07-14T19:29:01.313514+0000","last_unstale":"2023-07-14T19:29:02.573808+0000","last_undegraded":"2023-07-14T19:29:02.573808+0000","last_fullsized":"2023-07-14T19:29:02.573808+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:29:00.604983+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:29:00.604983+0000","last_clean_scrub_stamp":"2023-07-14T19:29:00.604983+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T04:19:18.663334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":12,"reported_epoch":83,"state":"active+clean","last_fresh":"2023-07-14T19:29:02.603793+0000","last_change":"2023-07-14T19:29:01.316899+0000","last_active":"2023-07-14T19:29:02.603793+0000","last_peered":"2023-07-14T19:29:02.603793+0000","last_clean":"2023-07-14T19:29:02.603793+0000","last_became_active":"2023-07-14T19:29:01.316603+0000","last_became_peered":"2023-07-14T19:29:01.316603+0000","last_unstale":"2023-07-14T19:29:02.603793+0000","last_undegraded":"2023-07-14T19:29:02.603793+0000","last_fullsized":"2023-07-14T19:29:02.603793+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:29:00.604983+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:29:00.604983+0000","last_clean_scrub_stamp":"2023-07-14T19:29:00.604983+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T01:32:43.783686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":12,"reported_epoch":83,"state":"active+clean","last_fresh":"2023-07-14T19:29:02.603746+0000","last_change":"2023-07-14T19:29:01.316557+0000","last_active":"2023-07-14T19:29:02.603746+0000","last_peered":"2023-07-14T19:29:02.603746+0000","last_clean":"2023-07-14T19:29:02.603746+0000","last_became_active":"2023-07-14T19:29:01.316079+0000","last_became_peered":"2023-07-14T19:29:01.316079+0000","last_unstale":"2023-07-14T19:29:02.603746+0000","last_undegraded":"2023-07-14T19:29:02.603746+0000","last_fullsized":"2023-07-14T19:29:02.603746+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:29:00.604983+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:29:00.604983+0000","last_clean_scrub_stamp":"2023-07-14T19:29:00.604983+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T05:43:38.465282+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":12,"reported_epoch":83,"state":"active+clean","last_fresh":"2023-07-14T19:29:03.339617+0000","last_change":"2023-07-14T19:29:01.307992+0000","last_active":"2023-07-14T19:29:03.339617+0000","last_peered":"2023-07-14T19:29:03.339617+0000","last_clean":"2023-07-14T19:29:03.339617+0000","last_became_active":"2023-07-14T19:29:01.307687+0000","last_became_peered":"2023-07-14T19:29:01.307687+0000","last_unstale":"2023-07-14T19:29:03.339617+0000","last_undegraded":"2023-07-14T19:29:03.339617+0000","last_fullsized":"2023-07-14T19:29:03.339617+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:29:00.604983+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:29:00.604983+0000","last_clean_scrub_stamp":"2023-07-14T19:29:00.604983+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T03:04:56.480796+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":12,"reported_epoch":83,"state":"active+clean","last_fresh":"2023-07-14T19:29:02.660819+0000","last_change":"2023-07-14T19:29:01.311036+0000","last_active":"2023-07-14T19:29:02.660819+0000","last_peered":"2023-07-14T19:29:02.660819+0000","last_clean":"2023-07-14T19:29:02.660819+0000","last_became_active":"2023-07-14T19:29:01.310723+0000","last_became_peered":"2023-07-14T19:29:01.310723+0000","last_unstale":"2023-07-14T19:29:02.660819+0000","last_undegraded":"2023-07-14T19:29:02.660819+0000","last_fullsized":"2023-07-14T19:29:02.660819+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:29:00.604983+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:29:00.604983+0000","last_clean_scrub_stamp":"2023-07-14T19:29:00.604983+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T23:51:40.842901+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":12,"reported_epoch":83,"state":"active+clean","last_fresh":"2023-07-14T19:29:03.339712+0000","last_change":"2023-07-14T19:29:01.315246+0000","last_active":"2023-07-14T19:29:03.339712+0000","last_peered":"2023-07-14T19:29:03.339712+0000","last_clean":"2023-07-14T19:29:03.339712+0000","last_became_active":"2023-07-14T19:29:01.314885+0000","last_became_peered":"2023-07-14T19:29:01.314885+0000","last_unstale":"2023-07-14T19:29:03.339712+0000","last_undegraded":"2023-07-14T19:29:03.339712+0000","last_fullsized":"2023-07-14T19:29:03.339712+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:29:00.604983+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:29:00.604983+0000","last_clean_scrub_stamp":"2023-07-14T19:29:00.604983+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T04:22:25.924419+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":12,"reported_epoch":83,"state":"active+clean","last_fresh":"2023-07-14T19:29:03.338493+0000","last_change":"2023-07-14T19:29:01.306322+0000","last_active":"2023-07-14T19:29:03.338493+0000","last_peered":"2023-07-14T19:29:03.338493+0000","last_clean":"2023-07-14T19:29:03.338493+0000","last_became_active":"2023-07-14T19:29:01.306024+0000","last_became_peered":"2023-07-14T19:29:01.306024+0000","last_unstale":"2023-07-14T19:29:03.338493+0000","last_undegraded":"2023-07-14T19:29:03.338493+0000","last_fullsized":"2023-07-14T19:29:03.338493+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:29:00.604983+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:29:00.604983+0000","last_clean_scrub_stamp":"2023-07-14T19:29:00.604983+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T04:13:45.893986+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":12,"reported_epoch":83,"state":"active+clean","last_fresh":"2023-07-14T19:29:02.298956+0000","last_change":"2023-07-14T19:29:01.312514+0000","last_active":"2023-07-14T19:29:02.298956+0000","last_peered":"2023-07-14T19:29:02.298956+0000","last_clean":"2023-07-14T19:29:02.298956+0000","last_became_active":"2023-07-14T19:29:01.312260+0000","last_became_peered":"2023-07-14T19:29:01.312260+0000","last_unstale":"2023-07-14T19:29:02.298956+0000","last_undegraded":"2023-07-14T19:29:02.298956+0000","last_fullsized":"2023-07-14T19:29:02.298956+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:29:00.604983+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:29:00.604983+0000","last_clean_scrub_stamp":"2023-07-14T19:29:00.604983+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T21:31:03.757922+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":12,"reported_epoch":83,"state":"active+clean","last_fresh":"2023-07-14T19:29:02.300275+0000","last_change":"2023-07-14T19:29:01.312344+0000","last_active":"2023-07-14T19:29:02.300275+0000","last_peered":"2023-07-14T19:29:02.300275+0000","last_clean":"2023-07-14T19:29:02.300275+0000","last_became_active":"2023-07-14T19:29:01.311960+0000","last_became_peered":"2023-07-14T19:29:01.311960+0000","last_unstale":"2023-07-14T19:29:02.300275+0000","last_undegraded":"2023-07-14T19:29:02.300275+0000","last_fullsized":"2023-07-14T19:29:02.300275+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:29:00.604983+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:29:00.604983+0000","last_clean_scrub_stamp":"2023-07-14T19:29:00.604983+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T23:15:52.484949+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":12,"reported_epoch":83,"state":"active+clean","last_fresh":"2023-07-14T19:29:03.339513+0000","last_change":"2023-07-14T19:29:01.314868+0000","last_active":"2023-07-14T19:29:03.339513+0000","last_peered":"2023-07-14T19:29:03.339513+0000","last_clean":"2023-07-14T19:29:03.339513+0000","last_became_active":"2023-07-14T19:29:01.314617+0000","last_became_peered":"2023-07-14T19:29:01.314617+0000","last_unstale":"2023-07-14T19:29:03.339513+0000","last_undegraded":"2023-07-14T19:29:03.339513+0000","last_fullsized":"2023-07-14T19:29:03.339513+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:29:00.604983+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:29:00.604983+0000","last_clean_scrub_stamp":"2023-07-14T19:29:00.604983+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T20:33:35.726050+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":12,"reported_epoch":83,"state":"active+clean","last_fresh":"2023-07-14T19:29:03.339870+0000","last_change":"2023-07-14T19:29:01.319124+0000","last_active":"2023-07-14T19:29:03.339870+0000","last_peered":"2023-07-14T19:29:03.339870+0000","last_clean":"2023-07-14T19:29:03.339870+0000","last_became_active":"2023-07-14T19:29:01.318392+0000","last_became_peered":"2023-07-14T19:29:01.318392+0000","last_unstale":"2023-07-14T19:29:03.339870+0000","last_undegraded":"2023-07-14T19:29:03.339870+0000","last_fullsized":"2023-07-14T19:29:03.339870+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:29:00.604983+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:29:00.604983+0000","last_clean_scrub_stamp":"2023-07-14T19:29:00.604983+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T06:19:35.297680+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":12,"reported_epoch":83,"state":"active+clean","last_fresh":"2023-07-14T19:29:03.339129+0000","last_change":"2023-07-14T19:29:01.319003+0000","last_active":"2023-07-14T19:29:03.339129+0000","last_peered":"2023-07-14T19:29:03.339129+0000","last_clean":"2023-07-14T19:29:03.339129+0000","last_became_active":"2023-07-14T19:29:01.316550+0000","last_became_peered":"2023-07-14T19:29:01.316550+0000","last_unstale":"2023-07-14T19:29:03.339129+0000","last_undegraded":"2023-07-14T19:29:03.339129+0000","last_fullsized":"2023-07-14T19:29:03.339129+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:29:00.604983+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:29:00.604983+0000","last_clean_scrub_stamp":"2023-07-14T19:29:00.604983+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T22:06:18.517102+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":12,"reported_epoch":83,"state":"active+clean","last_fresh":"2023-07-14T19:29:02.444425+0000","last_change":"2023-07-14T19:29:01.318596+0000","last_active":"2023-07-14T19:29:02.444425+0000","last_peered":"2023-07-14T19:29:02.444425+0000","last_clean":"2023-07-14T19:29:02.444425+0000","last_became_active":"2023-07-14T19:29:01.318278+0000","last_became_peered":"2023-07-14T19:29:01.318278+0000","last_unstale":"2023-07-14T19:29:02.444425+0000","last_undegraded":"2023-07-14T19:29:02.444425+0000","last_fullsized":"2023-07-14T19:29:02.444425+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:29:00.604983+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:29:00.604983+0000","last_clean_scrub_stamp":"2023-07-14T19:29:00.604983+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T06:00:40.438416+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":12,"reported_epoch":83,"state":"active+clean","last_fresh":"2023-07-14T19:29:02.603553+0000","last_change":"2023-07-14T19:29:01.314344+0000","last_active":"2023-07-14T19:29:02.603553+0000","last_peered":"2023-07-14T19:29:02.603553+0000","last_clean":"2023-07-14T19:29:02.603553+0000","last_became_active":"2023-07-14T19:29:01.314054+0000","last_became_peered":"2023-07-14T19:29:01.314054+0000","last_unstale":"2023-07-14T19:29:02.603553+0000","last_undegraded":"2023-07-14T19:29:02.603553+0000","last_fullsized":"2023-07-14T19:29:02.603553+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:29:00.604983+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:29:00.604983+0000","last_clean_scrub_stamp":"2023-07-14T19:29:00.604983+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T23:03:20.122826+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":12,"reported_epoch":83,"state":"active+clean","last_fresh":"2023-07-14T19:29:02.603428+0000","last_change":"2023-07-14T19:29:01.314703+0000","last_active":"2023-07-14T19:29:02.603428+0000","last_peered":"2023-07-14T19:29:02.603428+0000","last_clean":"2023-07-14T19:29:02.603428+0000","last_became_active":"2023-07-14T19:29:01.314425+0000","last_became_peered":"2023-07-14T19:29:01.314425+0000","last_unstale":"2023-07-14T19:29:02.603428+0000","last_undegraded":"2023-07-14T19:29:02.603428+0000","last_fullsized":"2023-07-14T19:29:02.603428+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:29:00.604983+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:29:00.604983+0000","last_clean_scrub_stamp":"2023-07-14T19:29:00.604983+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T02:24:08.467559+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":12,"reported_epoch":83,"state":"active+clean","last_fresh":"2023-07-14T19:29:02.299763+0000","last_change":"2023-07-14T19:29:01.308769+0000","last_active":"2023-07-14T19:29:02.299763+0000","last_peered":"2023-07-14T19:29:02.299763+0000","last_clean":"2023-07-14T19:29:02.299763+0000","last_became_active":"2023-07-14T19:29:01.308450+0000","last_became_peered":"2023-07-14T19:29:01.308450+0000","last_unstale":"2023-07-14T19:29:02.299763+0000","last_undegraded":"2023-07-14T19:29:02.299763+0000","last_fullsized":"2023-07-14T19:29:02.299763+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:29:00.604983+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:29:00.604983+0000","last_clean_scrub_stamp":"2023-07-14T19:29:00.604983+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T06:36:10.772949+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":12,"reported_epoch":83,"state":"active+clean","last_fresh":"2023-07-14T19:29:02.573896+0000","last_change":"2023-07-14T19:29:01.313928+0000","last_active":"2023-07-14T19:29:02.573896+0000","last_peered":"2023-07-14T19:29:02.573896+0000","last_clean":"2023-07-14T19:29:02.573896+0000","last_became_active":"2023-07-14T19:29:01.313528+0000","last_became_peered":"2023-07-14T19:29:01.313528+0000","last_unstale":"2023-07-14T19:29:02.573896+0000","last_undegraded":"2023-07-14T19:29:02.573896+0000","last_fullsized":"2023-07-14T19:29:02.573896+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:29:00.604983+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:29:00.604983+0000","last_clean_scrub_stamp":"2023-07-14T19:29:00.604983+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T20:08:14.603380+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":12,"reported_epoch":83,"state":"active+clean","last_fresh":"2023-07-14T19:29:02.864370+0000","last_change":"2023-07-14T19:29:01.318903+0000","last_active":"2023-07-14T19:29:02.864370+0000","last_peered":"2023-07-14T19:29:02.864370+0000","last_clean":"2023-07-14T19:29:02.864370+0000","last_became_active":"2023-07-14T19:29:01.318610+0000","last_became_peered":"2023-07-14T19:29:01.318610+0000","last_unstale":"2023-07-14T19:29:02.864370+0000","last_undegraded":"2023-07-14T19:29:02.864370+0000","last_fullsized":"2023-07-14T19:29:02.864370+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:29:00.604983+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:29:00.604983+0000","last_clean_scrub_stamp":"2023-07-14T19:29:00.604983+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T20:01:45.469683+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":12,"reported_epoch":83,"state":"active+clean","last_fresh":"2023-07-14T19:29:03.222037+0000","last_change":"2023-07-14T19:29:01.315405+0000","last_active":"2023-07-14T19:29:03.222037+0000","last_peered":"2023-07-14T19:29:03.222037+0000","last_clean":"2023-07-14T19:29:03.222037+0000","last_became_active":"2023-07-14T19:29:01.315156+0000","last_became_peered":"2023-07-14T19:29:01.315156+0000","last_unstale":"2023-07-14T19:29:03.222037+0000","last_undegraded":"2023-07-14T19:29:03.222037+0000","last_fullsized":"2023-07-14T19:29:03.222037+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:29:00.604983+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:29:00.604983+0000","last_clean_scrub_stamp":"2023-07-14T19:29:00.604983+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T02:34:33.971469+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"82'4","reported_seq":18,"reported_epoch":83,"state":"active+clean","last_fresh":"2023-07-14T19:29:02.864366+0000","last_change":"2023-07-14T19:29:00.618385+0000","last_active":"2023-07-14T19:29:02.864366+0000","last_peered":"2023-07-14T19:29:02.864366+0000","last_clean":"2023-07-14T19:29:02.864366+0000","last_became_active":"2023-07-14T19:29:00.618081+0000","last_became_peered":"2023-07-14T19:29:00.618081+0000","last_unstale":"2023-07-14T19:29:02.864366+0000","last_undegraded":"2023-07-14T19:29:02.864366+0000","last_fullsized":"2023-07-14T19:29:02.864366+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:28:59.600128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:28:59.600128+0000","last_clean_scrub_stamp":"2023-07-14T19:28:59.600128+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T06:55:30.739187+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":12,"reported_epoch":83,"state":"active+clean","last_fresh":"2023-07-14T19:29:02.299334+0000","last_change":"2023-07-14T19:29:01.305197+0000","last_active":"2023-07-14T19:29:02.299334+0000","last_peered":"2023-07-14T19:29:02.299334+0000","last_clean":"2023-07-14T19:29:02.299334+0000","last_became_active":"2023-07-14T19:29:01.304902+0000","last_became_peered":"2023-07-14T19:29:01.304902+0000","last_unstale":"2023-07-14T19:29:02.299334+0000","last_undegraded":"2023-07-14T19:29:02.299334+0000","last_fullsized":"2023-07-14T19:29:02.299334+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:29:00.604983+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:29:00.604983+0000","last_clean_scrub_stamp":"2023-07-14T19:29:00.604983+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T20:54:29.407321+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1e","version":"82'1","reported_seq":15,"reported_epoch":83,"state":"active+clean","last_fresh":"2023-07-14T19:29:02.299472+0000","last_change":"2023-07-14T19:29:00.626316+0000","last_active":"2023-07-14T19:29:02.299472+0000","last_peered":"2023-07-14T19:29:02.299472+0000","last_clean":"2023-07-14T19:29:02.299472+0000","last_became_active":"2023-07-14T19:29:00.626035+0000","last_became_peered":"2023-07-14T19:29:00.626035+0000","last_unstale":"2023-07-14T19:29:02.299472+0000","last_undegraded":"2023-07-14T19:29:02.299472+0000","last_fullsized":"2023-07-14T19:29:02.299472+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:28:59.600128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:28:59.600128+0000","last_clean_scrub_stamp":"2023-07-14T19:28:59.600128+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T23:59:08.083433+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":12,"reported_epoch":83,"state":"active+clean","last_fresh":"2023-07-14T19:29:02.603883+0000","last_change":"2023-07-14T19:29:01.317118+0000","last_active":"2023-07-14T19:29:02.603883+0000","last_peered":"2023-07-14T19:29:02.603883+0000","last_clean":"2023-07-14T19:29:02.603883+0000","last_became_active":"2023-07-14T19:29:01.316805+0000","last_became_peered":"2023-07-14T19:29:01.316805+0000","last_unstale":"2023-07-14T19:29:02.603883+0000","last_undegraded":"2023-07-14T19:29:02.603883+0000","last_fullsized":"2023-07-14T19:29:02.603883+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:29:00.604983+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:29:00.604983+0000","last_clean_scrub_stamp":"2023-07-14T19:29:00.604983+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T04:47:32.634271+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1d","version":"83'4","reported_seq":18,"reported_epoch":83,"state":"active+clean","last_fresh":"2023-07-14T19:29:05.679368+0000","last_change":"2023-07-14T19:29:00.619435+0000","last_active":"2023-07-14T19:29:05.679368+0000","last_peered":"2023-07-14T19:29:05.679368+0000","last_clean":"2023-07-14T19:29:05.679368+0000","last_became_active":"2023-07-14T19:29:00.619131+0000","last_became_peered":"2023-07-14T19:29:00.619131+0000","last_unstale":"2023-07-14T19:29:05.679368+0000","last_undegraded":"2023-07-14T19:29:05.679368+0000","last_fullsized":"2023-07-14T19:29:05.679368+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:28:59.600128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:28:59.600128+0000","last_clean_scrub_stamp":"2023-07-14T19:28:59.600128+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T04:51:10.432113+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":12,"reported_epoch":83,"state":"active+clean","last_fresh":"2023-07-14T19:29:02.299884+0000","last_change":"2023-07-14T19:29:01.317947+0000","last_active":"2023-07-14T19:29:02.299884+0000","last_peered":"2023-07-14T19:29:02.299884+0000","last_clean":"2023-07-14T19:29:02.299884+0000","last_became_active":"2023-07-14T19:29:01.317633+0000","last_became_peered":"2023-07-14T19:29:01.317633+0000","last_unstale":"2023-07-14T19:29:02.299884+0000","last_undegraded":"2023-07-14T19:29:02.299884+0000","last_fullsized":"2023-07-14T19:29:02.299884+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:29:00.604983+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:29:00.604983+0000","last_clean_scrub_stamp":"2023-07-14T19:29:00.604983+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T00:25:43.200161+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1c","version":"82'2","reported_seq":16,"reported_epoch":83,"state":"active+clean","last_fresh":"2023-07-14T19:29:02.661063+0000","last_change":"2023-07-14T19:29:00.620401+0000","last_active":"2023-07-14T19:29:02.661063+0000","last_peered":"2023-07-14T19:29:02.661063+0000","last_clean":"2023-07-14T19:29:02.661063+0000","last_became_active":"2023-07-14T19:29:00.620134+0000","last_became_peered":"2023-07-14T19:29:00.620134+0000","last_unstale":"2023-07-14T19:29:02.661063+0000","last_undegraded":"2023-07-14T19:29:02.661063+0000","last_fullsized":"2023-07-14T19:29:02.661063+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:28:59.600128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:28:59.600128+0000","last_clean_scrub_stamp":"2023-07-14T19:28:59.600128+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T06:39:10.382383+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":12,"reported_epoch":83,"state":"active+clean","last_fresh":"2023-07-14T19:29:02.299819+0000","last_change":"2023-07-14T19:29:01.312174+0000","last_active":"2023-07-14T19:29:02.299819+0000","last_peered":"2023-07-14T19:29:02.299819+0000","last_clean":"2023-07-14T19:29:02.299819+0000","last_became_active":"2023-07-14T19:29:01.311891+0000","last_became_peered":"2023-07-14T19:29:01.311891+0000","last_unstale":"2023-07-14T19:29:02.299819+0000","last_undegraded":"2023-07-14T19:29:02.299819+0000","last_fullsized":"2023-07-14T19:29:02.299819+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:29:00.604983+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:29:00.604983+0000","last_clean_scrub_stamp":"2023-07-14T19:29:00.604983+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T19:35:58.889437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1b","version":"83'6","reported_seq":23,"reported_epoch":83,"state":"active+clean","last_fresh":"2023-07-14T19:29:06.303726+0000","last_change":"2023-07-14T19:29:00.622257+0000","last_active":"2023-07-14T19:29:06.303726+0000","last_peered":"2023-07-14T19:29:06.303726+0000","last_clean":"2023-07-14T19:29:06.303726+0000","last_became_active":"2023-07-14T19:29:00.621984+0000","last_became_peered":"2023-07-14T19:29:00.621984+0000","last_unstale":"2023-07-14T19:29:06.303726+0000","last_undegraded":"2023-07-14T19:29:06.303726+0000","last_fullsized":"2023-07-14T19:29:06.303726+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:28:59.600128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:28:59.600128+0000","last_clean_scrub_stamp":"2023-07-14T19:28:59.600128+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T01:01:52.888462+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":964,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":3,"num_read_kb":2,"num_write":3,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":12,"reported_epoch":83,"state":"active+clean","last_fresh":"2023-07-14T19:29:02.660873+0000","last_change":"2023-07-14T19:29:01.306274+0000","last_active":"2023-07-14T19:29:02.660873+0000","last_peered":"2023-07-14T19:29:02.660873+0000","last_clean":"2023-07-14T19:29:02.660873+0000","last_became_active":"2023-07-14T19:29:01.305863+0000","last_became_peered":"2023-07-14T19:29:01.305863+0000","last_unstale":"2023-07-14T19:29:02.660873+0000","last_undegraded":"2023-07-14T19:29:02.660873+0000","last_fullsized":"2023-07-14T19:29:02.660873+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:29:00.604983+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:29:00.604983+0000","last_clean_scrub_stamp":"2023-07-14T19:29:00.604983+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T00:42:02.531260+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.1a","version":"82'1","reported_seq":15,"reported_epoch":83,"state":"active+clean","last_fresh":"2023-07-14T19:29:03.339637+0000","last_change":"2023-07-14T19:29:00.619509+0000","last_active":"2023-07-14T19:29:03.339637+0000","last_peered":"2023-07-14T19:29:03.339637+0000","last_clean":"2023-07-14T19:29:03.339637+0000","last_became_active":"2023-07-14T19:29:00.619193+0000","last_became_peered":"2023-07-14T19:29:00.619193+0000","last_unstale":"2023-07-14T19:29:03.339637+0000","last_undegraded":"2023-07-14T19:29:03.339637+0000","last_fullsized":"2023-07-14T19:29:03.339637+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:28:59.600128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:28:59.600128+0000","last_clean_scrub_stamp":"2023-07-14T19:28:59.600128+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T01:07:45.846978+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":12,"reported_epoch":83,"state":"active+clean","last_fresh":"2023-07-14T19:29:02.299714+0000","last_change":"2023-07-14T19:29:01.307535+0000","last_active":"2023-07-14T19:29:02.299714+0000","last_peered":"2023-07-14T19:29:02.299714+0000","last_clean":"2023-07-14T19:29:02.299714+0000","last_became_active":"2023-07-14T19:29:01.307209+0000","last_became_peered":"2023-07-14T19:29:01.307209+0000","last_unstale":"2023-07-14T19:29:02.299714+0000","last_undegraded":"2023-07-14T19:29:02.299714+0000","last_fullsized":"2023-07-14T19:29:02.299714+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:29:00.604983+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:29:00.604983+0000","last_clean_scrub_stamp":"2023-07-14T19:29:00.604983+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T03:49:13.610038+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":14,"reported_epoch":83,"state":"active+clean","last_fresh":"2023-07-14T19:29:02.444474+0000","last_change":"2023-07-14T19:29:00.620671+0000","last_active":"2023-07-14T19:29:02.444474+0000","last_peered":"2023-07-14T19:29:02.444474+0000","last_clean":"2023-07-14T19:29:02.444474+0000","last_became_active":"2023-07-14T19:29:00.619786+0000","last_became_peered":"2023-07-14T19:29:00.619786+0000","last_unstale":"2023-07-14T19:29:02.444474+0000","last_undegraded":"2023-07-14T19:29:02.444474+0000","last_fullsized":"2023-07-14T19:29:02.444474+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:28:59.600128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:28:59.600128+0000","last_clean_scrub_stamp":"2023-07-14T19:28:59.600128+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T06:56:01.025242+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":12,"reported_epoch":83,"state":"active+clean","last_fresh":"2023-07-14T19:29:02.298842+0000","last_change":"2023-07-14T19:29:01.316248+0000","last_active":"2023-07-14T19:29:02.298842+0000","last_peered":"2023-07-14T19:29:02.298842+0000","last_clean":"2023-07-14T19:29:02.298842+0000","last_became_active":"2023-07-14T19:29:01.315974+0000","last_became_peered":"2023-07-14T19:29:01.315974+0000","last_unstale":"2023-07-14T19:29:02.298842+0000","last_undegraded":"2023-07-14T19:29:02.298842+0000","last_fullsized":"2023-07-14T19:29:02.298842+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:29:00.604983+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:29:00.604983+0000","last_clean_scrub_stamp":"2023-07-14T19:29:00.604983+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T01:48:48.147093+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"83'1","reported_seq":15,"reported_epoch":83,"state":"active+clean","last_fresh":"2023-07-14T19:29:02.299408+0000","last_change":"2023-07-14T19:29:00.624210+0000","last_active":"2023-07-14T19:29:02.299408+0000","last_peered":"2023-07-14T19:29:02.299408+0000","last_clean":"2023-07-14T19:29:02.299408+0000","last_became_active":"2023-07-14T19:29:00.623918+0000","last_became_peered":"2023-07-14T19:29:00.623918+0000","last_unstale":"2023-07-14T19:29:02.299408+0000","last_undegraded":"2023-07-14T19:29:02.299408+0000","last_fullsized":"2023-07-14T19:29:02.299408+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:28:59.600128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:28:59.600128+0000","last_clean_scrub_stamp":"2023-07-14T19:28:59.600128+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T01:38:53.165557+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":12,"reported_epoch":83,"state":"active+clean","last_fresh":"2023-07-14T19:29:02.864241+0000","last_change":"2023-07-14T19:29:01.313787+0000","last_active":"2023-07-14T19:29:02.864241+0000","last_peered":"2023-07-14T19:29:02.864241+0000","last_clean":"2023-07-14T19:29:02.864241+0000","last_became_active":"2023-07-14T19:29:01.313479+0000","last_became_peered":"2023-07-14T19:29:01.313479+0000","last_unstale":"2023-07-14T19:29:02.864241+0000","last_undegraded":"2023-07-14T19:29:02.864241+0000","last_fullsized":"2023-07-14T19:29:02.864241+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:29:00.604983+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:29:00.604983+0000","last_clean_scrub_stamp":"2023-07-14T19:29:00.604983+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T22:33:21.348977+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":14,"reported_epoch":83,"state":"active+clean","last_fresh":"2023-07-14T19:29:03.339281+0000","last_change":"2023-07-14T19:29:00.623053+0000","last_active":"2023-07-14T19:29:03.339281+0000","last_peered":"2023-07-14T19:29:03.339281+0000","last_clean":"2023-07-14T19:29:03.339281+0000","last_became_active":"2023-07-14T19:29:00.622760+0000","last_became_peered":"2023-07-14T19:29:00.622760+0000","last_unstale":"2023-07-14T19:29:03.339281+0000","last_undegraded":"2023-07-14T19:29:03.339281+0000","last_fullsized":"2023-07-14T19:29:03.339281+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:28:59.600128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:28:59.600128+0000","last_clean_scrub_stamp":"2023-07-14T19:28:59.600128+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T21:15:50.464654+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":12,"reported_epoch":83,"state":"active+clean","last_fresh":"2023-07-14T19:29:02.299982+0000","last_change":"2023-07-14T19:29:01.317509+0000","last_active":"2023-07-14T19:29:02.299982+0000","last_peered":"2023-07-14T19:29:02.299982+0000","last_clean":"2023-07-14T19:29:02.299982+0000","last_became_active":"2023-07-14T19:29:01.317238+0000","last_became_peered":"2023-07-14T19:29:01.317238+0000","last_unstale":"2023-07-14T19:29:02.299982+0000","last_undegraded":"2023-07-14T19:29:02.299982+0000","last_fullsized":"2023-07-14T19:29:02.299982+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:29:00.604983+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:29:00.604983+0000","last_clean_scrub_stamp":"2023-07-14T19:29:00.604983+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T04:59:01.463446+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.16","version":"83'2","reported_seq":16,"reported_epoch":83,"state":"active+clean","last_fresh":"2023-07-14T19:29:04.671803+0000","last_change":"2023-07-14T19:29:00.624024+0000","last_active":"2023-07-14T19:29:04.671803+0000","last_peered":"2023-07-14T19:29:04.671803+0000","last_clean":"2023-07-14T19:29:04.671803+0000","last_became_active":"2023-07-14T19:29:00.623638+0000","last_became_peered":"2023-07-14T19:29:00.623638+0000","last_unstale":"2023-07-14T19:29:04.671803+0000","last_undegraded":"2023-07-14T19:29:04.671803+0000","last_fullsized":"2023-07-14T19:29:04.671803+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:28:59.600128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:28:59.600128+0000","last_clean_scrub_stamp":"2023-07-14T19:28:59.600128+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T03:50:19.311702+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":12,"reported_epoch":83,"state":"active+clean","last_fresh":"2023-07-14T19:29:02.603804+0000","last_change":"2023-07-14T19:29:01.309164+0000","last_active":"2023-07-14T19:29:02.603804+0000","last_peered":"2023-07-14T19:29:02.603804+0000","last_clean":"2023-07-14T19:29:02.603804+0000","last_became_active":"2023-07-14T19:29:01.308797+0000","last_became_peered":"2023-07-14T19:29:01.308797+0000","last_unstale":"2023-07-14T19:29:02.603804+0000","last_undegraded":"2023-07-14T19:29:02.603804+0000","last_fullsized":"2023-07-14T19:29:02.603804+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:29:00.604983+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:29:00.604983+0000","last_clean_scrub_stamp":"2023-07-14T19:29:00.604983+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T23:09:54.074320+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"83'4","reported_seq":19,"reported_epoch":83,"state":"active+clean","last_fresh":"2023-07-14T19:29:04.671757+0000","last_change":"2023-07-14T19:29:00.625435+0000","last_active":"2023-07-14T19:29:04.671757+0000","last_peered":"2023-07-14T19:29:04.671757+0000","last_clean":"2023-07-14T19:29:04.671757+0000","last_became_active":"2023-07-14T19:29:00.625092+0000","last_became_peered":"2023-07-14T19:29:00.625092+0000","last_unstale":"2023-07-14T19:29:04.671757+0000","last_undegraded":"2023-07-14T19:29:04.671757+0000","last_fullsized":"2023-07-14T19:29:04.671757+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:28:59.600128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:28:59.600128+0000","last_clean_scrub_stamp":"2023-07-14T19:28:59.600128+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T02:44:25.187321+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":12,"reported_epoch":83,"state":"active+clean","last_fresh":"2023-07-14T19:29:02.300025+0000","last_change":"2023-07-14T19:29:01.315672+0000","last_active":"2023-07-14T19:29:02.300025+0000","last_peered":"2023-07-14T19:29:02.300025+0000","last_clean":"2023-07-14T19:29:02.300025+0000","last_became_active":"2023-07-14T19:29:01.315331+0000","last_became_peered":"2023-07-14T19:29:01.315331+0000","last_unstale":"2023-07-14T19:29:02.300025+0000","last_undegraded":"2023-07-14T19:29:02.300025+0000","last_fullsized":"2023-07-14T19:29:02.300025+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:29:00.604983+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:29:00.604983+0000","last_clean_scrub_stamp":"2023-07-14T19:29:00.604983+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T02:46:22.959698+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.14","version":"82'6","reported_seq":20,"reported_epoch":83,"state":"active+clean","last_fresh":"2023-07-14T19:29:03.338983+0000","last_change":"2023-07-14T19:29:00.622910+0000","last_active":"2023-07-14T19:29:03.338983+0000","last_peered":"2023-07-14T19:29:03.338983+0000","last_clean":"2023-07-14T19:29:03.338983+0000","last_became_active":"2023-07-14T19:29:00.622582+0000","last_became_peered":"2023-07-14T19:29:00.622582+0000","last_unstale":"2023-07-14T19:29:03.338983+0000","last_undegraded":"2023-07-14T19:29:03.338983+0000","last_fullsized":"2023-07-14T19:29:03.338983+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:28:59.600128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:28:59.600128+0000","last_clean_scrub_stamp":"2023-07-14T19:28:59.600128+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T20:35:52.757984+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1564,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":12,"reported_epoch":83,"state":"active+clean","last_fresh":"2023-07-14T19:29:02.660955+0000","last_change":"2023-07-14T19:29:01.306169+0000","last_active":"2023-07-14T19:29:02.660955+0000","last_peered":"2023-07-14T19:29:02.660955+0000","last_clean":"2023-07-14T19:29:02.660955+0000","last_became_active":"2023-07-14T19:29:01.305830+0000","last_became_peered":"2023-07-14T19:29:01.305830+0000","last_unstale":"2023-07-14T19:29:02.660955+0000","last_undegraded":"2023-07-14T19:29:02.660955+0000","last_fullsized":"2023-07-14T19:29:02.660955+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:29:00.604983+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:29:00.604983+0000","last_clean_scrub_stamp":"2023-07-14T19:29:00.604983+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T00:12:18.670125+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.13","version":"82'2","reported_seq":16,"reported_epoch":83,"state":"active+clean","last_fresh":"2023-07-14T19:29:03.221973+0000","last_change":"2023-07-14T19:29:00.626500+0000","last_active":"2023-07-14T19:29:03.221973+0000","last_peered":"2023-07-14T19:29:03.221973+0000","last_clean":"2023-07-14T19:29:03.221973+0000","last_became_active":"2023-07-14T19:29:00.626130+0000","last_became_peered":"2023-07-14T19:29:00.626130+0000","last_unstale":"2023-07-14T19:29:03.221973+0000","last_undegraded":"2023-07-14T19:29:03.221973+0000","last_fullsized":"2023-07-14T19:29:03.221973+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:28:59.600128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:28:59.600128+0000","last_clean_scrub_stamp":"2023-07-14T19:28:59.600128+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T01:08:40.766374+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":12,"reported_epoch":83,"state":"active+clean","last_fresh":"2023-07-14T19:29:02.444400+0000","last_change":"2023-07-14T19:29:01.317902+0000","last_active":"2023-07-14T19:29:02.444400+0000","last_peered":"2023-07-14T19:29:02.444400+0000","last_clean":"2023-07-14T19:29:02.444400+0000","last_became_active":"2023-07-14T19:29:01.317587+0000","last_became_peered":"2023-07-14T19:29:01.317587+0000","last_unstale":"2023-07-14T19:29:02.444400+0000","last_undegraded":"2023-07-14T19:29:02.444400+0000","last_fullsized":"2023-07-14T19:29:02.444400+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:29:00.604983+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:29:00.604983+0000","last_clean_scrub_stamp":"2023-07-14T19:29:00.604983+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T03:39:39.441094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.12","version":"83'3","reported_seq":17,"reported_epoch":83,"state":"active+clean","last_fresh":"2023-07-14T19:29:04.671869+0000","last_change":"2023-07-14T19:29:00.626345+0000","last_active":"2023-07-14T19:29:04.671869+0000","last_peered":"2023-07-14T19:29:04.671869+0000","last_clean":"2023-07-14T19:29:04.671869+0000","last_became_active":"2023-07-14T19:29:00.626059+0000","last_became_peered":"2023-07-14T19:29:00.626059+0000","last_unstale":"2023-07-14T19:29:04.671869+0000","last_undegraded":"2023-07-14T19:29:04.671869+0000","last_fullsized":"2023-07-14T19:29:04.671869+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:28:59.600128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:28:59.600128+0000","last_clean_scrub_stamp":"2023-07-14T19:28:59.600128+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T21:18:49.525099+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":12,"reported_epoch":83,"state":"active+clean","last_fresh":"2023-07-14T19:29:03.339214+0000","last_change":"2023-07-14T19:29:01.310518+0000","last_active":"2023-07-14T19:29:03.339214+0000","last_peered":"2023-07-14T19:29:03.339214+0000","last_clean":"2023-07-14T19:29:03.339214+0000","last_became_active":"2023-07-14T19:29:01.310213+0000","last_became_peered":"2023-07-14T19:29:01.310213+0000","last_unstale":"2023-07-14T19:29:03.339214+0000","last_undegraded":"2023-07-14T19:29:03.339214+0000","last_fullsized":"2023-07-14T19:29:03.339214+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:29:00.604983+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:29:00.604983+0000","last_clean_scrub_stamp":"2023-07-14T19:29:00.604983+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T01:11:48.079392+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"83'3","reported_seq":17,"reported_epoch":83,"state":"active+clean","last_fresh":"2023-07-14T19:29:05.678131+0000","last_change":"2023-07-14T19:29:00.620450+0000","last_active":"2023-07-14T19:29:05.678131+0000","last_peered":"2023-07-14T19:29:05.678131+0000","last_clean":"2023-07-14T19:29:05.678131+0000","last_became_active":"2023-07-14T19:29:00.620100+0000","last_became_peered":"2023-07-14T19:29:00.620100+0000","last_unstale":"2023-07-14T19:29:05.678131+0000","last_undegraded":"2023-07-14T19:29:05.678131+0000","last_fullsized":"2023-07-14T19:29:05.678131+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:28:59.600128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:28:59.600128+0000","last_clean_scrub_stamp":"2023-07-14T19:28:59.600128+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T23:01:06.885919+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":540,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":12,"reported_epoch":83,"state":"active+clean","last_fresh":"2023-07-14T19:29:03.338482+0000","last_change":"2023-07-14T19:29:01.320612+0000","last_active":"2023-07-14T19:29:03.338482+0000","last_peered":"2023-07-14T19:29:03.338482+0000","last_clean":"2023-07-14T19:29:03.338482+0000","last_became_active":"2023-07-14T19:29:01.319569+0000","last_became_peered":"2023-07-14T19:29:01.319569+0000","last_unstale":"2023-07-14T19:29:03.338482+0000","last_undegraded":"2023-07-14T19:29:03.338482+0000","last_fullsized":"2023-07-14T19:29:03.338482+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:29:00.604983+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:29:00.604983+0000","last_clean_scrub_stamp":"2023-07-14T19:29:00.604983+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T05:46:45.094405+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.10","version":"83'15","reported_seq":29,"reported_epoch":83,"state":"active+clean","last_fresh":"2023-07-14T19:29:05.679927+0000","last_change":"2023-07-14T19:29:00.618619+0000","last_active":"2023-07-14T19:29:05.679927+0000","last_peered":"2023-07-14T19:29:05.679927+0000","last_clean":"2023-07-14T19:29:05.679927+0000","last_became_active":"2023-07-14T19:29:00.617735+0000","last_became_peered":"2023-07-14T19:29:00.617735+0000","last_unstale":"2023-07-14T19:29:05.679927+0000","last_undegraded":"2023-07-14T19:29:05.679927+0000","last_fullsized":"2023-07-14T19:29:05.679927+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:28:59.600128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:28:59.600128+0000","last_clean_scrub_stamp":"2023-07-14T19:28:59.600128+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T22:03:33.860671+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":12,"reported_epoch":83,"state":"active+clean","last_fresh":"2023-07-14T19:29:02.299414+0000","last_change":"2023-07-14T19:29:01.303053+0000","last_active":"2023-07-14T19:29:02.299414+0000","last_peered":"2023-07-14T19:29:02.299414+0000","last_clean":"2023-07-14T19:29:02.299414+0000","last_became_active":"2023-07-14T19:29:01.302679+0000","last_became_peered":"2023-07-14T19:29:01.302679+0000","last_unstale":"2023-07-14T19:29:02.299414+0000","last_undegraded":"2023-07-14T19:29:02.299414+0000","last_fullsized":"2023-07-14T19:29:02.299414+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:29:00.604983+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:29:00.604983+0000","last_clean_scrub_stamp":"2023-07-14T19:29:00.604983+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T03:30:02.623040+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"82'1","reported_seq":15,"reported_epoch":83,"state":"active+clean","last_fresh":"2023-07-14T19:29:02.661220+0000","last_change":"2023-07-14T19:29:00.622494+0000","last_active":"2023-07-14T19:29:02.661220+0000","last_peered":"2023-07-14T19:29:02.661220+0000","last_clean":"2023-07-14T19:29:02.661220+0000","last_became_active":"2023-07-14T19:29:00.622195+0000","last_became_peered":"2023-07-14T19:29:00.622195+0000","last_unstale":"2023-07-14T19:29:02.661220+0000","last_undegraded":"2023-07-14T19:29:02.661220+0000","last_fullsized":"2023-07-14T19:29:02.661220+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:28:59.600128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:28:59.600128+0000","last_clean_scrub_stamp":"2023-07-14T19:28:59.600128+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T01:34:47.084212+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":12,"reported_epoch":83,"state":"active+clean","last_fresh":"2023-07-14T19:29:02.573720+0000","last_change":"2023-07-14T19:29:01.319375+0000","last_active":"2023-07-14T19:29:02.573720+0000","last_peered":"2023-07-14T19:29:02.573720+0000","last_clean":"2023-07-14T19:29:02.573720+0000","last_became_active":"2023-07-14T19:29:01.319068+0000","last_became_peered":"2023-07-14T19:29:01.319068+0000","last_unstale":"2023-07-14T19:29:02.573720+0000","last_undegraded":"2023-07-14T19:29:02.573720+0000","last_fullsized":"2023-07-14T19:29:02.573720+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:29:00.604983+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:29:00.604983+0000","last_clean_scrub_stamp":"2023-07-14T19:29:00.604983+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T06:31:40.249974+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":14,"reported_epoch":83,"state":"active+clean","last_fresh":"2023-07-14T19:29:02.864257+0000","last_change":"2023-07-14T19:29:00.622364+0000","last_active":"2023-07-14T19:29:02.864257+0000","last_peered":"2023-07-14T19:29:02.864257+0000","last_clean":"2023-07-14T19:29:02.864257+0000","last_became_active":"2023-07-14T19:29:00.622073+0000","last_became_peered":"2023-07-14T19:29:00.622073+0000","last_unstale":"2023-07-14T19:29:02.864257+0000","last_undegraded":"2023-07-14T19:29:02.864257+0000","last_fullsized":"2023-07-14T19:29:02.864257+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:28:59.600128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:28:59.600128+0000","last_clean_scrub_stamp":"2023-07-14T19:28:59.600128+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T20:41:09.339550+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":12,"reported_epoch":83,"state":"active+clean","last_fresh":"2023-07-14T19:29:03.339325+0000","last_change":"2023-07-14T19:29:01.314076+0000","last_active":"2023-07-14T19:29:03.339325+0000","last_peered":"2023-07-14T19:29:03.339325+0000","last_clean":"2023-07-14T19:29:03.339325+0000","last_became_active":"2023-07-14T19:29:01.313769+0000","last_became_peered":"2023-07-14T19:29:01.313769+0000","last_unstale":"2023-07-14T19:29:03.339325+0000","last_undegraded":"2023-07-14T19:29:03.339325+0000","last_fullsized":"2023-07-14T19:29:03.339325+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:29:00.604983+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:29:00.604983+0000","last_clean_scrub_stamp":"2023-07-14T19:29:00.604983+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T02:26:15.354399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":14,"reported_epoch":83,"state":"active+clean","last_fresh":"2023-07-14T19:29:03.338900+0000","last_change":"2023-07-14T19:29:00.622928+0000","last_active":"2023-07-14T19:29:03.338900+0000","last_peered":"2023-07-14T19:29:03.338900+0000","last_clean":"2023-07-14T19:29:03.338900+0000","last_became_active":"2023-07-14T19:29:00.622628+0000","last_became_peered":"2023-07-14T19:29:00.622628+0000","last_unstale":"2023-07-14T19:29:03.338900+0000","last_undegraded":"2023-07-14T19:29:03.338900+0000","last_fullsized":"2023-07-14T19:29:03.338900+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:28:59.600128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:28:59.600128+0000","last_clean_scrub_stamp":"2023-07-14T19:28:59.600128+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T21:41:30.895447+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":12,"reported_epoch":83,"state":"active+clean","last_fresh":"2023-07-14T19:29:03.339904+0000","last_change":"2023-07-14T19:29:01.322519+0000","last_active":"2023-07-14T19:29:03.339904+0000","last_peered":"2023-07-14T19:29:03.339904+0000","last_clean":"2023-07-14T19:29:03.339904+0000","last_became_active":"2023-07-14T19:29:01.322233+0000","last_became_peered":"2023-07-14T19:29:01.322233+0000","last_unstale":"2023-07-14T19:29:03.339904+0000","last_undegraded":"2023-07-14T19:29:03.339904+0000","last_fullsized":"2023-07-14T19:29:03.339904+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:29:00.604983+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:29:00.604983+0000","last_clean_scrub_stamp":"2023-07-14T19:29:00.604983+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T23:32:02.997469+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":12,"reported_epoch":83,"state":"active+clean","last_fresh":"2023-07-14T19:29:02.299512+0000","last_change":"2023-07-14T19:29:01.305439+0000","last_active":"2023-07-14T19:29:02.299512+0000","last_peered":"2023-07-14T19:29:02.299512+0000","last_clean":"2023-07-14T19:29:02.299512+0000","last_became_active":"2023-07-14T19:29:01.305060+0000","last_became_peered":"2023-07-14T19:29:01.305060+0000","last_unstale":"2023-07-14T19:29:02.299512+0000","last_undegraded":"2023-07-14T19:29:02.299512+0000","last_fullsized":"2023-07-14T19:29:02.299512+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:29:00.604983+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:29:00.604983+0000","last_clean_scrub_stamp":"2023-07-14T19:29:00.604983+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T06:07:47.229804+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":14,"reported_epoch":83,"state":"active+clean","last_fresh":"2023-07-14T19:29:02.444211+0000","last_change":"2023-07-14T19:29:00.620867+0000","last_active":"2023-07-14T19:29:02.444211+0000","last_peered":"2023-07-14T19:29:02.444211+0000","last_clean":"2023-07-14T19:29:02.444211+0000","last_became_active":"2023-07-14T19:29:00.620370+0000","last_became_peered":"2023-07-14T19:29:00.620370+0000","last_unstale":"2023-07-14T19:29:02.444211+0000","last_undegraded":"2023-07-14T19:29:02.444211+0000","last_fullsized":"2023-07-14T19:29:02.444211+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:28:59.600128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:28:59.600128+0000","last_clean_scrub_stamp":"2023-07-14T19:28:59.600128+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T05:53:10.376311+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":12,"reported_epoch":83,"state":"active+clean","last_fresh":"2023-07-14T19:29:02.299185+0000","last_change":"2023-07-14T19:29:01.306289+0000","last_active":"2023-07-14T19:29:02.299185+0000","last_peered":"2023-07-14T19:29:02.299185+0000","last_clean":"2023-07-14T19:29:02.299185+0000","last_became_active":"2023-07-14T19:29:01.305987+0000","last_became_peered":"2023-07-14T19:29:01.305987+0000","last_unstale":"2023-07-14T19:29:02.299185+0000","last_undegraded":"2023-07-14T19:29:02.299185+0000","last_fullsized":"2023-07-14T19:29:02.299185+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:29:00.604983+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:29:00.604983+0000","last_clean_scrub_stamp":"2023-07-14T19:29:00.604983+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T07:21:20.067549+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"83'3","reported_seq":17,"reported_epoch":83,"state":"active+clean","last_fresh":"2023-07-14T19:29:04.670894+0000","last_change":"2023-07-14T19:29:00.620937+0000","last_active":"2023-07-14T19:29:04.670894+0000","last_peered":"2023-07-14T19:29:04.670894+0000","last_clean":"2023-07-14T19:29:04.670894+0000","last_became_active":"2023-07-14T19:29:00.620616+0000","last_became_peered":"2023-07-14T19:29:00.620616+0000","last_unstale":"2023-07-14T19:29:04.670894+0000","last_undegraded":"2023-07-14T19:29:04.670894+0000","last_fullsized":"2023-07-14T19:29:04.670894+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:28:59.600128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:28:59.600128+0000","last_clean_scrub_stamp":"2023-07-14T19:28:59.600128+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T20:32:12.990940+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"22'137","reported_seq":341,"reported_epoch":83,"state":"active+clean","last_fresh":"2023-07-14T19:29:03.338586+0000","last_change":"2023-07-14T19:28:01.948805+0000","last_active":"2023-07-14T19:29:03.338586+0000","last_peered":"2023-07-14T19:29:03.338586+0000","last_clean":"2023-07-14T19:29:03.338586+0000","last_became_active":"2023-07-14T19:28:01.948295+0000","last_became_peered":"2023-07-14T19:28:01.948295+0000","last_unstale":"2023-07-14T19:29:03.338586+0000","last_undegraded":"2023-07-14T19:29:03.338586+0000","last_fullsized":"2023-07-14T19:29:03.338586+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:25:10.240820+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:25:10.240820+0000","last_clean_scrub_stamp":"2023-07-14T19:25:10.240820+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T23:39:19.337174+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":12,"reported_epoch":83,"state":"active+clean","last_fresh":"2023-07-14T19:29:02.444463+0000","last_change":"2023-07-14T19:29:01.321025+0000","last_active":"2023-07-14T19:29:02.444463+0000","last_peered":"2023-07-14T19:29:02.444463+0000","last_clean":"2023-07-14T19:29:02.444463+0000","last_became_active":"2023-07-14T19:29:01.320787+0000","last_became_peered":"2023-07-14T19:29:01.320787+0000","last_unstale":"2023-07-14T19:29:02.444463+0000","last_undegraded":"2023-07-14T19:29:02.444463+0000","last_fullsized":"2023-07-14T19:29:02.444463+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:29:00.604983+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:29:00.604983+0000","last_clean_scrub_stamp":"2023-07-14T19:29:00.604983+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T02:26:20.883377+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"83'1","reported_seq":15,"reported_epoch":83,"state":"active+clean","last_fresh":"2023-07-14T19:29:03.338874+0000","last_change":"2023-07-14T19:29:00.623891+0000","last_active":"2023-07-14T19:29:03.338874+0000","last_peered":"2023-07-14T19:29:03.338874+0000","last_clean":"2023-07-14T19:29:03.338874+0000","last_became_active":"2023-07-14T19:29:00.623562+0000","last_became_peered":"2023-07-14T19:29:00.623562+0000","last_unstale":"2023-07-14T19:29:03.338874+0000","last_undegraded":"2023-07-14T19:29:03.338874+0000","last_fullsized":"2023-07-14T19:29:03.338874+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:28:59.600128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:28:59.600128+0000","last_clean_scrub_stamp":"2023-07-14T19:28:59.600128+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T19:35:24.864363+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.5","version":"83'1","reported_seq":15,"reported_epoch":83,"state":"active+clean","last_fresh":"2023-07-14T19:29:02.661060+0000","last_change":"2023-07-14T19:29:00.620282+0000","last_active":"2023-07-14T19:29:02.661060+0000","last_peered":"2023-07-14T19:29:02.661060+0000","last_clean":"2023-07-14T19:29:02.661060+0000","last_became_active":"2023-07-14T19:29:00.620005+0000","last_became_peered":"2023-07-14T19:29:00.620005+0000","last_unstale":"2023-07-14T19:29:02.661060+0000","last_undegraded":"2023-07-14T19:29:02.661060+0000","last_fullsized":"2023-07-14T19:29:02.661060+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:28:59.600128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:28:59.600128+0000","last_clean_scrub_stamp":"2023-07-14T19:28:59.600128+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T03:20:25.333743+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":12,"reported_epoch":83,"state":"active+clean","last_fresh":"2023-07-14T19:29:02.444350+0000","last_change":"2023-07-14T19:29:01.311708+0000","last_active":"2023-07-14T19:29:02.444350+0000","last_peered":"2023-07-14T19:29:02.444350+0000","last_clean":"2023-07-14T19:29:02.444350+0000","last_became_active":"2023-07-14T19:29:01.311400+0000","last_became_peered":"2023-07-14T19:29:01.311400+0000","last_unstale":"2023-07-14T19:29:02.444350+0000","last_undegraded":"2023-07-14T19:29:02.444350+0000","last_fullsized":"2023-07-14T19:29:02.444350+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:29:00.604983+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:29:00.604983+0000","last_clean_scrub_stamp":"2023-07-14T19:29:00.604983+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T23:01:26.499242+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":14,"reported_epoch":83,"state":"active+clean","last_fresh":"2023-07-14T19:29:02.864292+0000","last_change":"2023-07-14T19:29:00.617411+0000","last_active":"2023-07-14T19:29:02.864292+0000","last_peered":"2023-07-14T19:29:02.864292+0000","last_clean":"2023-07-14T19:29:02.864292+0000","last_became_active":"2023-07-14T19:29:00.617078+0000","last_became_peered":"2023-07-14T19:29:00.617078+0000","last_unstale":"2023-07-14T19:29:02.864292+0000","last_undegraded":"2023-07-14T19:29:02.864292+0000","last_fullsized":"2023-07-14T19:29:02.864292+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:28:59.600128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:28:59.600128+0000","last_clean_scrub_stamp":"2023-07-14T19:28:59.600128+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T21:38:43.985609+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":12,"reported_epoch":83,"state":"active+clean","last_fresh":"2023-07-14T19:29:02.299351+0000","last_change":"2023-07-14T19:29:01.307282+0000","last_active":"2023-07-14T19:29:02.299351+0000","last_peered":"2023-07-14T19:29:02.299351+0000","last_clean":"2023-07-14T19:29:02.299351+0000","last_became_active":"2023-07-14T19:29:01.306966+0000","last_became_peered":"2023-07-14T19:29:01.306966+0000","last_unstale":"2023-07-14T19:29:02.299351+0000","last_undegraded":"2023-07-14T19:29:02.299351+0000","last_fullsized":"2023-07-14T19:29:02.299351+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:29:00.604983+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:29:00.604983+0000","last_clean_scrub_stamp":"2023-07-14T19:29:00.604983+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T23:06:00.918796+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"83'4","reported_seq":18,"reported_epoch":83,"state":"active+clean","last_fresh":"2023-07-14T19:29:04.671123+0000","last_change":"2023-07-14T19:29:00.624599+0000","last_active":"2023-07-14T19:29:04.671123+0000","last_peered":"2023-07-14T19:29:04.671123+0000","last_clean":"2023-07-14T19:29:04.671123+0000","last_became_active":"2023-07-14T19:29:00.624272+0000","last_became_peered":"2023-07-14T19:29:00.624272+0000","last_unstale":"2023-07-14T19:29:04.671123+0000","last_undegraded":"2023-07-14T19:29:04.671123+0000","last_fullsized":"2023-07-14T19:29:04.671123+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:28:59.600128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:28:59.600128+0000","last_clean_scrub_stamp":"2023-07-14T19:28:59.600128+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T20:00:31.406685+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":12,"reported_epoch":83,"state":"active+clean","last_fresh":"2023-07-14T19:29:02.603663+0000","last_change":"2023-07-14T19:29:01.309411+0000","last_active":"2023-07-14T19:29:02.603663+0000","last_peered":"2023-07-14T19:29:02.603663+0000","last_clean":"2023-07-14T19:29:02.603663+0000","last_became_active":"2023-07-14T19:29:01.309113+0000","last_became_peered":"2023-07-14T19:29:01.309113+0000","last_unstale":"2023-07-14T19:29:02.603663+0000","last_undegraded":"2023-07-14T19:29:02.603663+0000","last_fullsized":"2023-07-14T19:29:02.603663+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:29:00.604983+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:29:00.604983+0000","last_clean_scrub_stamp":"2023-07-14T19:29:00.604983+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T07:09:16.642605+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":14,"reported_epoch":83,"state":"active+clean","last_fresh":"2023-07-14T19:29:02.299764+0000","last_change":"2023-07-14T19:29:00.622758+0000","last_active":"2023-07-14T19:29:02.299764+0000","last_peered":"2023-07-14T19:29:02.299764+0000","last_clean":"2023-07-14T19:29:02.299764+0000","last_became_active":"2023-07-14T19:29:00.622460+0000","last_became_peered":"2023-07-14T19:29:00.622460+0000","last_unstale":"2023-07-14T19:29:02.299764+0000","last_undegraded":"2023-07-14T19:29:02.299764+0000","last_fullsized":"2023-07-14T19:29:02.299764+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:28:59.600128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:28:59.600128+0000","last_clean_scrub_stamp":"2023-07-14T19:28:59.600128+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T23:30:15.407153+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":12,"reported_epoch":83,"state":"active+clean","last_fresh":"2023-07-14T19:29:02.660717+0000","last_change":"2023-07-14T19:29:01.317046+0000","last_active":"2023-07-14T19:29:02.660717+0000","last_peered":"2023-07-14T19:29:02.660717+0000","last_clean":"2023-07-14T19:29:02.660717+0000","last_became_active":"2023-07-14T19:29:01.316653+0000","last_became_peered":"2023-07-14T19:29:01.316653+0000","last_unstale":"2023-07-14T19:29:02.660717+0000","last_undegraded":"2023-07-14T19:29:02.660717+0000","last_fullsized":"2023-07-14T19:29:02.660717+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:29:00.604983+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:29:00.604983+0000","last_clean_scrub_stamp":"2023-07-14T19:29:00.604983+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T04:50:19.726545+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":14,"reported_epoch":83,"state":"active+clean","last_fresh":"2023-07-14T19:29:02.299046+0000","last_change":"2023-07-14T19:29:00.619474+0000","last_active":"2023-07-14T19:29:02.299046+0000","last_peered":"2023-07-14T19:29:02.299046+0000","last_clean":"2023-07-14T19:29:02.299046+0000","last_became_active":"2023-07-14T19:29:00.619240+0000","last_became_peered":"2023-07-14T19:29:00.619240+0000","last_unstale":"2023-07-14T19:29:02.299046+0000","last_undegraded":"2023-07-14T19:29:02.299046+0000","last_fullsized":"2023-07-14T19:29:02.299046+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:28:59.600128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:28:59.600128+0000","last_clean_scrub_stamp":"2023-07-14T19:28:59.600128+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T02:51:09.423733+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":12,"reported_epoch":83,"state":"active+clean","last_fresh":"2023-07-14T19:29:02.573459+0000","last_change":"2023-07-14T19:29:01.308964+0000","last_active":"2023-07-14T19:29:02.573459+0000","last_peered":"2023-07-14T19:29:02.573459+0000","last_clean":"2023-07-14T19:29:02.573459+0000","last_became_active":"2023-07-14T19:29:01.308638+0000","last_became_peered":"2023-07-14T19:29:01.308638+0000","last_unstale":"2023-07-14T19:29:02.573459+0000","last_undegraded":"2023-07-14T19:29:02.573459+0000","last_fullsized":"2023-07-14T19:29:02.573459+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:29:00.604983+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:29:00.604983+0000","last_clean_scrub_stamp":"2023-07-14T19:29:00.604983+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T03:51:09.875511+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":14,"reported_epoch":83,"state":"active+clean","last_fresh":"2023-07-14T19:29:03.221784+0000","last_change":"2023-07-14T19:29:00.617894+0000","last_active":"2023-07-14T19:29:03.221784+0000","last_peered":"2023-07-14T19:29:03.221784+0000","last_clean":"2023-07-14T19:29:03.221784+0000","last_became_active":"2023-07-14T19:29:00.617584+0000","last_became_peered":"2023-07-14T19:29:00.617584+0000","last_unstale":"2023-07-14T19:29:03.221784+0000","last_undegraded":"2023-07-14T19:29:03.221784+0000","last_fullsized":"2023-07-14T19:29:03.221784+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:28:59.600128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:28:59.600128+0000","last_clean_scrub_stamp":"2023-07-14T19:28:59.600128+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T23:24:30.467604+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":12,"reported_epoch":83,"state":"active+clean","last_fresh":"2023-07-14T19:29:02.299239+0000","last_change":"2023-07-14T19:29:01.317153+0000","last_active":"2023-07-14T19:29:02.299239+0000","last_peered":"2023-07-14T19:29:02.299239+0000","last_clean":"2023-07-14T19:29:02.299239+0000","last_became_active":"2023-07-14T19:29:01.316800+0000","last_became_peered":"2023-07-14T19:29:01.316800+0000","last_unstale":"2023-07-14T19:29:02.299239+0000","last_undegraded":"2023-07-14T19:29:02.299239+0000","last_fullsized":"2023-07-14T19:29:02.299239+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:29:00.604983+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:29:00.604983+0000","last_clean_scrub_stamp":"2023-07-14T19:29:00.604983+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T04:58:24.726933+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"83'4","reported_seq":18,"reported_epoch":83,"state":"active+clean","last_fresh":"2023-07-14T19:29:04.671987+0000","last_change":"2023-07-14T19:29:00.618469+0000","last_active":"2023-07-14T19:29:04.671987+0000","last_peered":"2023-07-14T19:29:04.671987+0000","last_clean":"2023-07-14T19:29:04.671987+0000","last_became_active":"2023-07-14T19:29:00.618218+0000","last_became_peered":"2023-07-14T19:29:00.618218+0000","last_unstale":"2023-07-14T19:29:04.671987+0000","last_undegraded":"2023-07-14T19:29:04.671987+0000","last_fullsized":"2023-07-14T19:29:04.671987+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:28:59.600128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:28:59.600128+0000","last_clean_scrub_stamp":"2023-07-14T19:28:59.600128+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T22:41:36.925396+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":12,"reported_epoch":83,"state":"active+clean","last_fresh":"2023-07-14T19:29:03.339251+0000","last_change":"2023-07-14T19:29:01.319023+0000","last_active":"2023-07-14T19:29:03.339251+0000","last_peered":"2023-07-14T19:29:03.339251+0000","last_clean":"2023-07-14T19:29:03.339251+0000","last_became_active":"2023-07-14T19:29:01.311530+0000","last_became_peered":"2023-07-14T19:29:01.311530+0000","last_unstale":"2023-07-14T19:29:03.339251+0000","last_undegraded":"2023-07-14T19:29:03.339251+0000","last_fullsized":"2023-07-14T19:29:03.339251+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:29:00.604983+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:29:00.604983+0000","last_clean_scrub_stamp":"2023-07-14T19:29:00.604983+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T01:46:30.175019+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":14,"reported_epoch":83,"state":"active+clean","last_fresh":"2023-07-14T19:29:03.222071+0000","last_change":"2023-07-14T19:29:00.624227+0000","last_active":"2023-07-14T19:29:03.222071+0000","last_peered":"2023-07-14T19:29:03.222071+0000","last_clean":"2023-07-14T19:29:03.222071+0000","last_became_active":"2023-07-14T19:29:00.623953+0000","last_became_peered":"2023-07-14T19:29:00.623953+0000","last_unstale":"2023-07-14T19:29:03.222071+0000","last_undegraded":"2023-07-14T19:29:03.222071+0000","last_fullsized":"2023-07-14T19:29:03.222071+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:28:59.600128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:28:59.600128+0000","last_clean_scrub_stamp":"2023-07-14T19:28:59.600128+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T04:31:39.691751+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":12,"reported_epoch":83,"state":"active+clean","last_fresh":"2023-07-14T19:29:03.340024+0000","last_change":"2023-07-14T19:29:01.319936+0000","last_active":"2023-07-14T19:29:03.340024+0000","last_peered":"2023-07-14T19:29:03.340024+0000","last_clean":"2023-07-14T19:29:03.340024+0000","last_became_active":"2023-07-14T19:29:01.319437+0000","last_became_peered":"2023-07-14T19:29:01.319437+0000","last_unstale":"2023-07-14T19:29:03.340024+0000","last_undegraded":"2023-07-14T19:29:03.340024+0000","last_fullsized":"2023-07-14T19:29:03.340024+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:29:00.604983+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:29:00.604983+0000","last_clean_scrub_stamp":"2023-07-14T19:29:00.604983+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T07:14:58.343588+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"82'1","reported_seq":16,"reported_epoch":83,"state":"active+clean","last_fresh":"2023-07-14T19:29:02.299764+0000","last_change":"2023-07-14T19:29:00.628103+0000","last_active":"2023-07-14T19:29:02.299764+0000","last_peered":"2023-07-14T19:29:02.299764+0000","last_clean":"2023-07-14T19:29:02.299764+0000","last_became_active":"2023-07-14T19:29:00.627689+0000","last_became_peered":"2023-07-14T19:29:00.627689+0000","last_unstale":"2023-07-14T19:29:02.299764+0000","last_undegraded":"2023-07-14T19:29:02.299764+0000","last_fullsized":"2023-07-14T19:29:02.299764+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:28:59.600128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:28:59.600128+0000","last_clean_scrub_stamp":"2023-07-14T19:28:59.600128+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T23:24:22.985508+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":12,"reported_epoch":83,"state":"active+clean","last_fresh":"2023-07-14T19:29:03.338902+0000","last_change":"2023-07-14T19:29:01.319519+0000","last_active":"2023-07-14T19:29:03.338902+0000","last_peered":"2023-07-14T19:29:03.338902+0000","last_clean":"2023-07-14T19:29:03.338902+0000","last_became_active":"2023-07-14T19:29:01.318900+0000","last_became_peered":"2023-07-14T19:29:01.318900+0000","last_unstale":"2023-07-14T19:29:03.338902+0000","last_undegraded":"2023-07-14T19:29:03.338902+0000","last_fullsized":"2023-07-14T19:29:03.338902+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:29:00.604983+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:29:00.604983+0000","last_clean_scrub_stamp":"2023-07-14T19:29:00.604983+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T20:31:55.947593+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"82'2","reported_seq":16,"reported_epoch":83,"state":"active+clean","last_fresh":"2023-07-14T19:29:02.444432+0000","last_change":"2023-07-14T19:29:00.619280+0000","last_active":"2023-07-14T19:29:02.444432+0000","last_peered":"2023-07-14T19:29:02.444432+0000","last_clean":"2023-07-14T19:29:02.444432+0000","last_became_active":"2023-07-14T19:29:00.618980+0000","last_became_peered":"2023-07-14T19:29:00.618980+0000","last_unstale":"2023-07-14T19:29:02.444432+0000","last_undegraded":"2023-07-14T19:29:02.444432+0000","last_fullsized":"2023-07-14T19:29:02.444432+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:28:59.600128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:28:59.600128+0000","last_clean_scrub_stamp":"2023-07-14T19:28:59.600128+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T22:09:48.530415+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":12,"reported_epoch":83,"state":"active+clean","last_fresh":"2023-07-14T19:29:03.339356+0000","last_change":"2023-07-14T19:29:01.320160+0000","last_active":"2023-07-14T19:29:03.339356+0000","last_peered":"2023-07-14T19:29:03.339356+0000","last_clean":"2023-07-14T19:29:03.339356+0000","last_became_active":"2023-07-14T19:29:01.319850+0000","last_became_peered":"2023-07-14T19:29:01.319850+0000","last_unstale":"2023-07-14T19:29:03.339356+0000","last_undegraded":"2023-07-14T19:29:03.339356+0000","last_fullsized":"2023-07-14T19:29:03.339356+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:29:00.604983+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:29:00.604983+0000","last_clean_scrub_stamp":"2023-07-14T19:29:00.604983+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T04:25:40.648850+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":12,"reported_epoch":83,"state":"active+clean","last_fresh":"2023-07-14T19:29:02.573560+0000","last_change":"2023-07-14T19:29:01.309770+0000","last_active":"2023-07-14T19:29:02.573560+0000","last_peered":"2023-07-14T19:29:02.573560+0000","last_clean":"2023-07-14T19:29:02.573560+0000","last_became_active":"2023-07-14T19:29:01.308677+0000","last_became_peered":"2023-07-14T19:29:01.308677+0000","last_unstale":"2023-07-14T19:29:02.573560+0000","last_undegraded":"2023-07-14T19:29:02.573560+0000","last_fullsized":"2023-07-14T19:29:02.573560+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:29:00.604983+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:29:00.604983+0000","last_clean_scrub_stamp":"2023-07-14T19:29:00.604983+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T21:27:03.743070+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":12,"reported_epoch":83,"state":"active+clean","last_fresh":"2023-07-14T19:29:03.221821+0000","last_change":"2023-07-14T19:29:01.319447+0000","last_active":"2023-07-14T19:29:03.221821+0000","last_peered":"2023-07-14T19:29:03.221821+0000","last_clean":"2023-07-14T19:29:03.221821+0000","last_became_active":"2023-07-14T19:29:01.319120+0000","last_became_peered":"2023-07-14T19:29:01.319120+0000","last_unstale":"2023-07-14T19:29:03.221821+0000","last_undegraded":"2023-07-14T19:29:03.221821+0000","last_fullsized":"2023-07-14T19:29:03.221821+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:29:00.604983+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:29:00.604983+0000","last_clean_scrub_stamp":"2023-07-14T19:29:00.604983+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T07:03:28.055942+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":12,"reported_epoch":83,"state":"active+clean","last_fresh":"2023-07-14T19:29:03.221929+0000","last_change":"2023-07-14T19:29:01.310554+0000","last_active":"2023-07-14T19:29:03.221929+0000","last_peered":"2023-07-14T19:29:03.221929+0000","last_clean":"2023-07-14T19:29:03.221929+0000","last_became_active":"2023-07-14T19:29:01.310302+0000","last_became_peered":"2023-07-14T19:29:01.310302+0000","last_unstale":"2023-07-14T19:29:03.221929+0000","last_undegraded":"2023-07-14T19:29:03.221929+0000","last_fullsized":"2023-07-14T19:29:03.221929+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:29:00.604983+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:29:00.604983+0000","last_clean_scrub_stamp":"2023-07-14T19:29:00.604983+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T04:50:18.928722+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":12,"reported_epoch":83,"state":"active+clean","last_fresh":"2023-07-14T19:29:02.603764+0000","last_change":"2023-07-14T19:29:01.318753+0000","last_active":"2023-07-14T19:29:02.603764+0000","last_peered":"2023-07-14T19:29:02.603764+0000","last_clean":"2023-07-14T19:29:02.603764+0000","last_became_active":"2023-07-14T19:29:01.318476+0000","last_became_peered":"2023-07-14T19:29:01.318476+0000","last_unstale":"2023-07-14T19:29:02.603764+0000","last_undegraded":"2023-07-14T19:29:02.603764+0000","last_fullsized":"2023-07-14T19:29:02.603764+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:29:00.604983+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:29:00.604983+0000","last_clean_scrub_stamp":"2023-07-14T19:29:00.604983+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T07:27:40.189788+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":12,"reported_epoch":83,"state":"active+clean","last_fresh":"2023-07-14T19:29:02.444422+0000","last_change":"2023-07-14T19:29:01.317843+0000","last_active":"2023-07-14T19:29:02.444422+0000","last_peered":"2023-07-14T19:29:02.444422+0000","last_clean":"2023-07-14T19:29:02.444422+0000","last_became_active":"2023-07-14T19:29:01.317550+0000","last_became_peered":"2023-07-14T19:29:01.317550+0000","last_unstale":"2023-07-14T19:29:02.444422+0000","last_undegraded":"2023-07-14T19:29:02.444422+0000","last_fullsized":"2023-07-14T19:29:02.444422+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:29:00.604983+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:29:00.604983+0000","last_clean_scrub_stamp":"2023-07-14T19:29:00.604983+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T22:59:15.139142+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":12,"reported_epoch":83,"state":"active+clean","last_fresh":"2023-07-14T19:29:02.573731+0000","last_change":"2023-07-14T19:29:01.309103+0000","last_active":"2023-07-14T19:29:02.573731+0000","last_peered":"2023-07-14T19:29:02.573731+0000","last_clean":"2023-07-14T19:29:02.573731+0000","last_became_active":"2023-07-14T19:29:01.308786+0000","last_became_peered":"2023-07-14T19:29:01.308786+0000","last_unstale":"2023-07-14T19:29:02.573731+0000","last_undegraded":"2023-07-14T19:29:02.573731+0000","last_fullsized":"2023-07-14T19:29:02.573731+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:29:00.604983+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:29:00.604983+0000","last_clean_scrub_stamp":"2023-07-14T19:29:00.604983+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T02:55:34.529448+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":12,"reported_epoch":83,"state":"active+clean","last_fresh":"2023-07-14T19:29:03.222213+0000","last_change":"2023-07-14T19:29:01.309940+0000","last_active":"2023-07-14T19:29:03.222213+0000","last_peered":"2023-07-14T19:29:03.222213+0000","last_clean":"2023-07-14T19:29:03.222213+0000","last_became_active":"2023-07-14T19:29:01.309638+0000","last_became_peered":"2023-07-14T19:29:01.309638+0000","last_unstale":"2023-07-14T19:29:03.222213+0000","last_undegraded":"2023-07-14T19:29:03.222213+0000","last_fullsized":"2023-07-14T19:29:03.222213+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:29:00.604983+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:29:00.604983+0000","last_clean_scrub_stamp":"2023-07-14T19:29:00.604983+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T03:28:26.563593+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":12,"reported_epoch":83,"state":"active+clean","last_fresh":"2023-07-14T19:29:02.299283+0000","last_change":"2023-07-14T19:29:01.305957+0000","last_active":"2023-07-14T19:29:02.299283+0000","last_peered":"2023-07-14T19:29:02.299283+0000","last_clean":"2023-07-14T19:29:02.299283+0000","last_became_active":"2023-07-14T19:29:01.305564+0000","last_became_peered":"2023-07-14T19:29:01.305564+0000","last_unstale":"2023-07-14T19:29:02.299283+0000","last_undegraded":"2023-07-14T19:29:02.299283+0000","last_fullsized":"2023-07-14T19:29:02.299283+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:29:00.604983+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:29:00.604983+0000","last_clean_scrub_stamp":"2023-07-14T19:29:00.604983+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T21:47:06.297622+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":12,"reported_epoch":83,"state":"active+clean","last_fresh":"2023-07-14T19:29:02.299375+0000","last_change":"2023-07-14T19:29:01.310521+0000","last_active":"2023-07-14T19:29:02.299375+0000","last_peered":"2023-07-14T19:29:02.299375+0000","last_clean":"2023-07-14T19:29:02.299375+0000","last_became_active":"2023-07-14T19:29:01.310211+0000","last_became_peered":"2023-07-14T19:29:01.310211+0000","last_unstale":"2023-07-14T19:29:02.299375+0000","last_undegraded":"2023-07-14T19:29:02.299375+0000","last_fullsized":"2023-07-14T19:29:02.299375+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:29:00.604983+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:29:00.604983+0000","last_clean_scrub_stamp":"2023-07-14T19:29:00.604983+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T19:41:09.317667+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":12,"reported_epoch":83,"state":"active+clean","last_fresh":"2023-07-14T19:29:02.298983+0000","last_change":"2023-07-14T19:29:01.308033+0000","last_active":"2023-07-14T19:29:02.298983+0000","last_peered":"2023-07-14T19:29:02.298983+0000","last_clean":"2023-07-14T19:29:02.298983+0000","last_became_active":"2023-07-14T19:29:01.307166+0000","last_became_peered":"2023-07-14T19:29:01.307166+0000","last_unstale":"2023-07-14T19:29:02.298983+0000","last_undegraded":"2023-07-14T19:29:02.298983+0000","last_fullsized":"2023-07-14T19:29:02.298983+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:29:00.604983+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:29:00.604983+0000","last_clean_scrub_stamp":"2023-07-14T19:29:00.604983+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T06:29:57.949181+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":12,"reported_epoch":83,"state":"active+clean","last_fresh":"2023-07-14T19:29:02.299844+0000","last_change":"2023-07-14T19:29:01.308122+0000","last_active":"2023-07-14T19:29:02.299844+0000","last_peered":"2023-07-14T19:29:02.299844+0000","last_clean":"2023-07-14T19:29:02.299844+0000","last_became_active":"2023-07-14T19:29:01.307568+0000","last_became_peered":"2023-07-14T19:29:01.307568+0000","last_unstale":"2023-07-14T19:29:02.299844+0000","last_undegraded":"2023-07-14T19:29:02.299844+0000","last_fullsized":"2023-07-14T19:29:02.299844+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:29:00.604983+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:29:00.604983+0000","last_clean_scrub_stamp":"2023-07-14T19:29:00.604983+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T04:08:56.378075+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":12,"reported_epoch":83,"state":"active+clean","last_fresh":"2023-07-14T19:29:02.444527+0000","last_change":"2023-07-14T19:29:01.314261+0000","last_active":"2023-07-14T19:29:02.444527+0000","last_peered":"2023-07-14T19:29:02.444527+0000","last_clean":"2023-07-14T19:29:02.444527+0000","last_became_active":"2023-07-14T19:29:01.313967+0000","last_became_peered":"2023-07-14T19:29:01.313967+0000","last_unstale":"2023-07-14T19:29:02.444527+0000","last_undegraded":"2023-07-14T19:29:02.444527+0000","last_fullsized":"2023-07-14T19:29:02.444527+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:29:00.604983+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:29:00.604983+0000","last_clean_scrub_stamp":"2023-07-14T19:29:00.604983+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T04:19:07.923742+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":12,"reported_epoch":83,"state":"active+clean","last_fresh":"2023-07-14T19:29:02.573578+0000","last_change":"2023-07-14T19:29:01.319605+0000","last_active":"2023-07-14T19:29:02.573578+0000","last_peered":"2023-07-14T19:29:02.573578+0000","last_clean":"2023-07-14T19:29:02.573578+0000","last_became_active":"2023-07-14T19:29:01.319310+0000","last_became_peered":"2023-07-14T19:29:01.319310+0000","last_unstale":"2023-07-14T19:29:02.573578+0000","last_undegraded":"2023-07-14T19:29:02.573578+0000","last_fullsized":"2023-07-14T19:29:02.573578+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:29:00.604983+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:29:00.604983+0000","last_clean_scrub_stamp":"2023-07-14T19:29:00.604983+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T20:14:29.714925+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":4751360,"data_stored":4722944,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":4616,"num_objects":39,"num_object_clones":0,"num_object_copies":117,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":39,"num_whiteouts":0,"num_read":5,"num_read_kb":4,"num_write":77,"num_write_kb":35,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":167936,"data_stored":11392,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":71,"ondisk_log_size":71,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":78,"seq":335007449099,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27892,"kb_used_data":872,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717260,"statfs":{"total":95995035648,"available":95966474240,"internally_reserved":0,"allocated":892928,"data_stored":428457,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":27654411},"hb_peers":[0,1,2,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.5}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.77200000000000002}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.749}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.37}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.68700000000000006}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.66700000000000004}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.72599999999999998}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.92300000000000004}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.89500000000000002}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.872}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.84199999999999997}]}]},{"osd":10,"up_from":70,"seq":300647710736,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28488,"kb_used_data":1460,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716664,"statfs":{"total":95995035648,"available":95965863936,"internally_reserved":0,"allocated":1495040,"data_stored":1011682,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":27654411},"hb_peers":[1,2,3,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.52600000000000002}]},{"osd":1,"last update":"Fri Jul 14 19:29:00 2023","interfaces":[{"interface":"back","average":{"1min":0.44800000000000001,"5min":0.44800000000000001,"15min":0.44800000000000001},"min":{"1min":0.27000000000000002,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":0.73299999999999998,"5min":0.73299999999999998,"15min":0.73299999999999998},"last":0.33900000000000002},{"interface":"front","average":{"1min":0.46800000000000003,"5min":0.46800000000000003,"15min":0.46800000000000003},"min":{"1min":0.28599999999999998,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":0.80800000000000005,"5min":0.80800000000000005,"15min":0.80800000000000005},"last":0.40100000000000002}]},{"osd":2,"last update":"Fri Jul 14 19:29:00 2023","interfaces":[{"interface":"back","average":{"1min":0.59899999999999998,"5min":0.59899999999999998,"15min":0.59899999999999998},"min":{"1min":0.40200000000000002,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":0.83699999999999997,"5min":0.83699999999999997,"15min":0.83699999999999997},"last":0.68700000000000006},{"interface":"front","average":{"1min":0.64300000000000002,"5min":0.64300000000000002,"15min":0.64300000000000002},"min":{"1min":0.46800000000000003,"5min":0.46800000000000003,"15min":0.46800000000000003},"max":{"1min":0.89100000000000001,"5min":0.89100000000000001,"15min":0.89100000000000001},"last":0.75600000000000001}]},{"osd":3,"last update":"Fri Jul 14 19:29:00 2023","interfaces":[{"interface":"back","average":{"1min":0.58199999999999996,"5min":0.58199999999999996,"15min":0.58199999999999996},"min":{"1min":0.35999999999999999,"5min":0.35999999999999999,"15min":0.35999999999999999},"max":{"1min":0.77800000000000002,"5min":0.77800000000000002,"15min":0.77800000000000002},"last":0.56999999999999995},{"interface":"front","average":{"1min":0.63200000000000001,"5min":0.63200000000000001,"15min":0.63200000000000001},"min":{"1min":0.45200000000000001,"5min":0.45200000000000001,"15min":0.45200000000000001},"max":{"1min":0.89700000000000002,"5min":0.89700000000000002,"15min":0.89700000000000002},"last":0.47899999999999998}]},{"osd":4,"last update":"Fri Jul 14 19:29:00 2023","interfaces":[{"interface":"back","average":{"1min":0.55600000000000005,"5min":0.55600000000000005,"15min":0.55600000000000005},"min":{"1min":0.316,"5min":0.316,"15min":0.316},"max":{"1min":0.99399999999999999,"5min":0.99399999999999999,"15min":0.99399999999999999},"last":0.37},{"interface":"front","average":{"1min":0.54800000000000004,"5min":0.54800000000000004,"15min":0.54800000000000004},"min":{"1min":0.29299999999999998,"5min":0.29299999999999998,"15min":0.29299999999999998},"max":{"1min":0.94999999999999996,"5min":0.94999999999999996,"15min":0.94999999999999996},"last":0.42399999999999999}]},{"osd":5,"last update":"Fri Jul 14 19:29:00 2023","interfaces":[{"interface":"back","average":{"1min":0.68600000000000005,"5min":0.68600000000000005,"15min":0.68600000000000005},"min":{"1min":0.45300000000000001,"5min":0.45300000000000001,"15min":0.45300000000000001},"max":{"1min":1.0129999999999999,"5min":1.0129999999999999,"15min":1.0129999999999999},"last":0.77500000000000002},{"interface":"front","average":{"1min":0.67100000000000004,"5min":0.67100000000000004,"15min":0.67100000000000004},"min":{"1min":0.374,"5min":0.374,"15min":0.374},"max":{"1min":0.96499999999999997,"5min":0.96499999999999997,"15min":0.96499999999999997},"last":0.60699999999999998}]},{"osd":6,"last update":"Fri Jul 14 19:29:00 2023","interfaces":[{"interface":"back","average":{"1min":0.65700000000000003,"5min":0.65700000000000003,"15min":0.65700000000000003},"min":{"1min":0.40400000000000003,"5min":0.40400000000000003,"15min":0.40400000000000003},"max":{"1min":1.032,"5min":1.032,"15min":1.032},"last":0.51800000000000002},{"interface":"front","average":{"1min":0.68799999999999994,"5min":0.68799999999999994,"15min":0.68799999999999994},"min":{"1min":0.45700000000000002,"5min":0.45700000000000002,"15min":0.45700000000000002},"max":{"1min":1.0649999999999999,"5min":1.0649999999999999,"15min":1.0649999999999999},"last":0.50800000000000001}]},{"osd":7,"last update":"Fri Jul 14 19:29:00 2023","interfaces":[{"interface":"back","average":{"1min":0.66000000000000003,"5min":0.66000000000000003,"15min":0.66000000000000003},"min":{"1min":0.34200000000000003,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":1.0960000000000001,"5min":1.0960000000000001,"15min":1.0960000000000001},"last":0.46000000000000002},{"interface":"front","average":{"1min":0.69399999999999995,"5min":0.69399999999999995,"15min":0.69399999999999995},"min":{"1min":0.44600000000000001,"5min":0.44600000000000001,"15min":0.44600000000000001},"max":{"1min":0.97899999999999998,"5min":0.97899999999999998,"15min":0.97899999999999998},"last":0.44600000000000001}]},{"osd":8,"last update":"Fri Jul 14 19:29:00 2023","interfaces":[{"interface":"back","average":{"1min":0.75900000000000001,"5min":0.75900000000000001,"15min":0.75900000000000001},"min":{"1min":0.52800000000000002,"5min":0.52800000000000002,"15min":0.52800000000000002},"max":{"1min":1.143,"5min":1.143,"15min":1.143},"last":0.65200000000000002},{"interface":"front","average":{"1min":0.77600000000000002,"5min":0.77600000000000002,"15min":0.77600000000000002},"min":{"1min":0.47599999999999998,"5min":0.47599999999999998,"15min":0.47599999999999998},"max":{"1min":1.1060000000000001,"5min":1.1060000000000001,"15min":1.1060000000000001},"last":0.71199999999999997}]},{"osd":9,"last update":"Fri Jul 14 19:29:00 2023","interfaces":[{"interface":"back","average":{"1min":0.754,"5min":0.754,"15min":0.754},"min":{"1min":0.54500000000000004,"5min":0.54500000000000004,"15min":0.54500000000000004},"max":{"1min":1.115,"5min":1.115,"15min":1.115},"last":0.55200000000000005},{"interface":"front","average":{"1min":0.77000000000000002,"5min":0.77000000000000002,"15min":0.77000000000000002},"min":{"1min":0.49299999999999999,"5min":0.49299999999999999,"15min":0.49299999999999999},"max":{"1min":1.127,"5min":1.127,"15min":1.127},"last":0.49299999999999999}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.69699999999999995}]}]},{"osd":9,"up_from":65,"seq":279172874260,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27896,"kb_used_data":876,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717256,"statfs":{"total":95995035648,"available":95966470144,"internally_reserved":0,"allocated":897024,"data_stored":429039,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":27654411},"hb_peers":[0,1,2,3,4,5,6,7,8,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jul 14 19:28:37 2023","interfaces":[{"interface":"back","average":{"1min":0.434,"5min":0.434,"15min":0.434},"min":{"1min":0.28000000000000003,"5min":0.28000000000000003,"15min":0.28000000000000003},"max":{"1min":0.54800000000000004,"5min":0.54800000000000004,"15min":0.54800000000000004},"last":0.35399999999999998},{"interface":"front","average":{"1min":0.47199999999999998,"5min":0.47199999999999998,"15min":0.47199999999999998},"min":{"1min":0.26100000000000001,"5min":0.26100000000000001,"15min":0.26100000000000001},"max":{"1min":0.76400000000000001,"5min":0.76400000000000001,"15min":0.76400000000000001},"last":0.39500000000000002}]},{"osd":1,"last update":"Fri Jul 14 19:28:37 2023","interfaces":[{"interface":"back","average":{"1min":0.63200000000000001,"5min":0.63200000000000001,"15min":0.63200000000000001},"min":{"1min":0.45000000000000001,"5min":0.45000000000000001,"15min":0.45000000000000001},"max":{"1min":0.82199999999999995,"5min":0.82199999999999995,"15min":0.82199999999999995},"last":0.55000000000000004},{"interface":"front","average":{"1min":0.54800000000000004,"5min":0.54800000000000004,"15min":0.54800000000000004},"min":{"1min":0.437,"5min":0.437,"15min":0.437},"max":{"1min":0.70899999999999996,"5min":0.70899999999999996,"15min":0.70899999999999996},"last":0.57899999999999996}]},{"osd":2,"last update":"Fri Jul 14 19:28:37 2023","interfaces":[{"interface":"back","average":{"1min":0.63,"5min":0.63,"15min":0.63},"min":{"1min":0.49099999999999999,"5min":0.49099999999999999,"15min":0.49099999999999999},"max":{"1min":0.75800000000000001,"5min":0.75800000000000001,"15min":0.75800000000000001},"last":0.49399999999999999},{"interface":"front","average":{"1min":0.58899999999999997,"5min":0.58899999999999997,"15min":0.58899999999999997},"min":{"1min":0.373,"5min":0.373,"15min":0.373},"max":{"1min":0.72399999999999998,"5min":0.72399999999999998,"15min":0.72399999999999998},"last":0.378}]},{"osd":3,"last update":"Fri Jul 14 19:28:37 2023","interfaces":[{"interface":"back","average":{"1min":0.51900000000000002,"5min":0.51900000000000002,"15min":0.51900000000000002},"min":{"1min":0.35099999999999998,"5min":0.35099999999999998,"15min":0.35099999999999998},"max":{"1min":0.71399999999999997,"5min":0.71399999999999997,"15min":0.71399999999999997},"last":0.51100000000000001},{"interface":"front","average":{"1min":0.58399999999999996,"5min":0.58399999999999996,"15min":0.58399999999999996},"min":{"1min":0.33900000000000002,"5min":0.33900000000000002,"15min":0.33900000000000002},"max":{"1min":0.85699999999999998,"5min":0.85699999999999998,"15min":0.85699999999999998},"last":0.318}]},{"osd":4,"last update":"Fri Jul 14 19:28:37 2023","interfaces":[{"interface":"back","average":{"1min":0.59599999999999997,"5min":0.59599999999999997,"15min":0.59599999999999997},"min":{"1min":0.499,"5min":0.499,"15min":0.499},"max":{"1min":0.77200000000000002,"5min":0.77200000000000002,"15min":0.77200000000000002},"last":0.45300000000000001},{"interface":"front","average":{"1min":0.66000000000000003,"5min":0.66000000000000003,"15min":0.66000000000000003},"min":{"1min":0.47699999999999998,"5min":0.47699999999999998,"15min":0.47699999999999998},"max":{"1min":0.84499999999999997,"5min":0.84499999999999997,"15min":0.84499999999999997},"last":0.628}]},{"osd":5,"last update":"Fri Jul 14 19:28:37 2023","interfaces":[{"interface":"back","average":{"1min":0.67200000000000004,"5min":0.67200000000000004,"15min":0.67200000000000004},"min":{"1min":0.54000000000000004,"5min":0.54000000000000004,"15min":0.54000000000000004},"max":{"1min":0.872,"5min":0.872,"15min":0.872},"last":0.56399999999999995},{"interface":"front","average":{"1min":0.73899999999999999,"5min":0.73899999999999999,"15min":0.73899999999999999},"min":{"1min":0.52500000000000002,"5min":0.52500000000000002,"15min":0.52500000000000002},"max":{"1min":0.90600000000000003,"5min":0.90600000000000003,"15min":0.90600000000000003},"last":0.46500000000000002}]},{"osd":6,"last update":"Fri Jul 14 19:28:37 2023","interfaces":[{"interface":"back","average":{"1min":0.69399999999999995,"5min":0.69399999999999995,"15min":0.69399999999999995},"min":{"1min":0.40699999999999997,"5min":0.40699999999999997,"15min":0.40699999999999997},"max":{"1min":0.89900000000000002,"5min":0.89900000000000002,"15min":0.89900000000000002},"last":0.42599999999999999},{"interface":"front","average":{"1min":0.60399999999999998,"5min":0.60399999999999998,"15min":0.60399999999999998},"min":{"1min":0.38800000000000001,"5min":0.38800000000000001,"15min":0.38800000000000001},"max":{"1min":0.86399999999999999,"5min":0.86399999999999999,"15min":0.86399999999999999},"last":0.41499999999999998}]},{"osd":7,"last update":"Fri Jul 14 19:28:37 2023","interfaces":[{"interface":"back","average":{"1min":0.75800000000000001,"5min":0.75800000000000001,"15min":0.75800000000000001},"min":{"1min":0.58199999999999996,"5min":0.58199999999999996,"15min":0.58199999999999996},"max":{"1min":0.96799999999999997,"5min":0.96799999999999997,"15min":0.96799999999999997},"last":0.65500000000000003},{"interface":"front","average":{"1min":0.77500000000000002,"5min":0.77500000000000002,"15min":0.77500000000000002},"min":{"1min":0.52700000000000002,"5min":0.52700000000000002,"15min":0.52700000000000002},"max":{"1min":0.95299999999999996,"5min":0.95299999999999996,"15min":0.95299999999999996},"last":0.61699999999999999}]},{"osd":8,"last update":"Fri Jul 14 19:28:37 2023","interfaces":[{"interface":"back","average":{"1min":0.71699999999999997,"5min":0.71699999999999997,"15min":0.71699999999999997},"min":{"1min":0.53700000000000003,"5min":0.53700000000000003,"15min":0.53700000000000003},"max":{"1min":0.91300000000000003,"5min":0.91300000000000003,"15min":0.91300000000000003},"last":0.53800000000000003},{"interface":"front","average":{"1min":0.74199999999999999,"5min":0.74199999999999999,"15min":0.74199999999999999},"min":{"1min":0.54500000000000004,"5min":0.54500000000000004,"15min":0.54500000000000004},"max":{"1min":0.89500000000000002,"5min":0.89500000000000002,"15min":0.89500000000000002},"last":0.68400000000000005}]},{"osd":10,"last update":"Fri Jul 14 19:29:00 2023","interfaces":[{"interface":"back","average":{"1min":0.70299999999999996,"5min":0.70299999999999996,"15min":0.70299999999999996},"min":{"1min":0.439,"5min":0.439,"15min":0.439},"max":{"1min":0.95199999999999996,"5min":0.95199999999999996,"15min":0.95199999999999996},"last":0.60499999999999998},{"interface":"front","average":{"1min":0.73799999999999999,"5min":0.73799999999999999,"15min":0.73799999999999999},"min":{"1min":0.47799999999999998,"5min":0.47799999999999998,"15min":0.47799999999999998},"max":{"1min":1.016,"5min":1.016,"15min":1.016},"last":0.72699999999999998}]}]},{"osd":8,"up_from":57,"seq":244813135896,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27896,"kb_used_data":880,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717256,"statfs":{"total":95995035648,"available":95966470144,"internally_reserved":0,"allocated":901120,"data_stored":428802,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":27654411},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jul 14 19:28:18 2023","interfaces":[{"interface":"back","average":{"1min":0.55500000000000005,"5min":0.55500000000000005,"15min":0.55500000000000005},"min":{"1min":0.30099999999999999,"5min":0.30099999999999999,"15min":0.30099999999999999},"max":{"1min":0.73699999999999999,"5min":0.73699999999999999,"15min":0.73699999999999999},"last":0.39300000000000002},{"interface":"front","average":{"1min":0.54500000000000004,"5min":0.54500000000000004,"15min":0.54500000000000004},"min":{"1min":0.32100000000000001,"5min":0.32100000000000001,"15min":0.32100000000000001},"max":{"1min":0.755,"5min":0.755,"15min":0.755},"last":0.46100000000000002}]},{"osd":1,"last update":"Fri Jul 14 19:28:18 2023","interfaces":[{"interface":"back","average":{"1min":0.58199999999999996,"5min":0.58199999999999996,"15min":0.58199999999999996},"min":{"1min":0.35199999999999998,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":0.85499999999999998,"5min":0.85499999999999998,"15min":0.85499999999999998},"last":0.66100000000000003},{"interface":"front","average":{"1min":0.60399999999999998,"5min":0.60399999999999998,"15min":0.60399999999999998},"min":{"1min":0.34499999999999997,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":0.93400000000000005,"5min":0.93400000000000005,"15min":0.93400000000000005},"last":0.55900000000000005}]},{"osd":2,"last update":"Fri Jul 14 19:28:18 2023","interfaces":[{"interface":"back","average":{"1min":0.47199999999999998,"5min":0.47199999999999998,"15min":0.47199999999999998},"min":{"1min":0.27700000000000002,"5min":0.27700000000000002,"15min":0.27700000000000002},"max":{"1min":0.67200000000000004,"5min":0.67200000000000004,"15min":0.67200000000000004},"last":0.42999999999999999},{"interface":"front","average":{"1min":0.45500000000000002,"5min":0.45500000000000002,"15min":0.45500000000000002},"min":{"1min":0.25600000000000001,"5min":0.25600000000000001,"15min":0.25600000000000001},"max":{"1min":0.61399999999999999,"5min":0.61399999999999999,"15min":0.61399999999999999},"last":0.48699999999999999}]},{"osd":3,"last update":"Fri Jul 14 19:28:18 2023","interfaces":[{"interface":"back","average":{"1min":0.57199999999999995,"5min":0.57199999999999995,"15min":0.57199999999999995},"min":{"1min":0.36199999999999999,"5min":0.36199999999999999,"15min":0.36199999999999999},"max":{"1min":0.77000000000000002,"5min":0.77000000000000002,"15min":0.77000000000000002},"last":0.68799999999999994},{"interface":"front","average":{"1min":0.58399999999999996,"5min":0.58399999999999996,"15min":0.58399999999999996},"min":{"1min":0.41499999999999998,"5min":0.41499999999999998,"15min":0.41499999999999998},"max":{"1min":0.77700000000000002,"5min":0.77700000000000002,"15min":0.77700000000000002},"last":0.54300000000000004}]},{"osd":4,"last update":"Fri Jul 14 19:28:18 2023","interfaces":[{"interface":"back","average":{"1min":0.60399999999999998,"5min":0.60399999999999998,"15min":0.60399999999999998},"min":{"1min":0.41699999999999998,"5min":0.41699999999999998,"15min":0.41699999999999998},"max":{"1min":0.86399999999999999,"5min":0.86399999999999999,"15min":0.86399999999999999},"last":0.78800000000000003},{"interface":"front","average":{"1min":0.63700000000000001,"5min":0.63700000000000001,"15min":0.63700000000000001},"min":{"1min":0.39500000000000002,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":0.88,"5min":0.88,"15min":0.88},"last":0.56899999999999995}]},{"osd":5,"last update":"Fri Jul 14 19:28:18 2023","interfaces":[{"interface":"back","average":{"1min":0.55100000000000005,"5min":0.55100000000000005,"15min":0.55100000000000005},"min":{"1min":0.311,"5min":0.311,"15min":0.311},"max":{"1min":0.81000000000000005,"5min":0.81000000000000005,"15min":0.81000000000000005},"last":0.52800000000000002},{"interface":"front","average":{"1min":0.56599999999999995,"5min":0.56599999999999995,"15min":0.56599999999999995},"min":{"1min":0.34799999999999998,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":0.90900000000000003,"5min":0.90900000000000003,"15min":0.90900000000000003},"last":0.752}]},{"osd":6,"last update":"Fri Jul 14 19:28:18 2023","interfaces":[{"interface":"back","average":{"1min":0.65400000000000003,"5min":0.65400000000000003,"15min":0.65400000000000003},"min":{"1min":0.436,"5min":0.436,"15min":0.436},"max":{"1min":0.873,"5min":0.873,"15min":0.873},"last":0.72699999999999998},{"interface":"front","average":{"1min":0.68000000000000005,"5min":0.68000000000000005,"15min":0.68000000000000005},"min":{"1min":0.45600000000000002,"5min":0.45600000000000002,"15min":0.45600000000000002},"max":{"1min":1.1479999999999999,"5min":1.1479999999999999,"15min":1.1479999999999999},"last":0.59499999999999997}]},{"osd":7,"last update":"Fri Jul 14 19:28:18 2023","interfaces":[{"interface":"back","average":{"1min":0.72599999999999998,"5min":0.72599999999999998,"15min":0.72599999999999998},"min":{"1min":0.53400000000000003,"5min":0.53400000000000003,"15min":0.53400000000000003},"max":{"1min":1.2070000000000001,"5min":1.2070000000000001,"15min":1.2070000000000001},"last":0.63700000000000001},{"interface":"front","average":{"1min":0.71199999999999997,"5min":0.71199999999999997,"15min":0.71199999999999997},"min":{"1min":0.47499999999999998,"5min":0.47499999999999998,"15min":0.47499999999999998},"max":{"1min":1.127,"5min":1.127,"15min":1.127},"last":0.83799999999999997}]},{"osd":9,"last update":"Fri Jul 14 19:28:39 2023","interfaces":[{"interface":"back","average":{"1min":0.68000000000000005,"5min":0.68000000000000005,"15min":0.68000000000000005},"min":{"1min":0.497,"5min":0.497,"15min":0.497},"max":{"1min":0.997,"5min":0.997,"15min":0.997},"last":0.81499999999999995},{"interface":"front","average":{"1min":0.68400000000000005,"5min":0.68400000000000005,"15min":0.68400000000000005},"min":{"1min":0.48299999999999998,"5min":0.48299999999999998,"15min":0.48299999999999998},"max":{"1min":0.86299999999999999,"5min":0.86299999999999999,"15min":0.86299999999999999},"last":0.77000000000000002}]},{"osd":10,"last update":"Fri Jul 14 19:28:58 2023","interfaces":[{"interface":"back","average":{"1min":0.69599999999999995,"5min":0.69599999999999995,"15min":0.69599999999999995},"min":{"1min":0.55800000000000005,"5min":0.55800000000000005,"15min":0.55800000000000005},"max":{"1min":0.94799999999999995,"5min":0.94799999999999995,"15min":0.94799999999999995},"last":0.88900000000000001},{"interface":"front","average":{"1min":0.72299999999999998,"5min":0.72299999999999998,"15min":0.72299999999999998},"min":{"1min":0.55200000000000005,"5min":0.55200000000000005,"15min":0.55200000000000005},"max":{"1min":0.91100000000000003,"5min":0.91100000000000003,"15min":0.91100000000000003},"last":0.82799999999999996}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.86399999999999999}]}]},{"osd":7,"up_from":50,"seq":214748364828,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28484,"kb_used_data":1464,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716668,"statfs":{"total":95995035648,"available":95965868032,"internally_reserved":0,"allocated":1499136,"data_stored":1019981,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":27654411},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jul 14 19:28:59 2023","interfaces":[{"interface":"back","average":{"1min":0.60799999999999998,"5min":0.57299999999999995,"15min":0.56699999999999995},"min":{"1min":0.41599999999999998,"5min":0.40400000000000003,"15min":0.40400000000000003},"max":{"1min":0.95899999999999996,"5min":1.0349999999999999,"15min":1.0349999999999999},"last":0.45100000000000001},{"interface":"front","average":{"1min":0.56499999999999995,"5min":0.54200000000000004,"15min":0.53800000000000003},"min":{"1min":0.34899999999999998,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":1.0089999999999999,"5min":1.0089999999999999,"15min":1.0089999999999999},"last":0.40999999999999998}]},{"osd":1,"last update":"Fri Jul 14 19:28:59 2023","interfaces":[{"interface":"back","average":{"1min":0.47299999999999998,"5min":0.45600000000000002,"15min":0.45300000000000001},"min":{"1min":0.27600000000000002,"5min":0.27600000000000002,"15min":0.27600000000000002},"max":{"1min":0.76000000000000001,"5min":0.76000000000000001,"15min":0.76000000000000001},"last":0.35599999999999998},{"interface":"front","average":{"1min":0.48899999999999999,"5min":0.47699999999999998,"15min":0.47499999999999998},"min":{"1min":0.251,"5min":0.251,"15min":0.251},"max":{"1min":0.81999999999999995,"5min":0.81999999999999995,"15min":0.81999999999999995},"last":0.48199999999999998}]},{"osd":2,"last update":"Fri Jul 14 19:28:59 2023","interfaces":[{"interface":"back","average":{"1min":0.64700000000000002,"5min":0.621,"15min":0.61699999999999999},"min":{"1min":0.41399999999999998,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":1.091,"5min":1.091,"15min":1.091},"last":0.64400000000000002},{"interface":"front","average":{"1min":0.63800000000000001,"5min":0.61799999999999999,"15min":0.61499999999999999},"min":{"1min":0.35199999999999998,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":1.1040000000000001,"5min":1.1040000000000001,"15min":1.1040000000000001},"last":0.67700000000000005}]},{"osd":3,"last update":"Fri Jul 14 19:28:59 2023","interfaces":[{"interface":"back","average":{"1min":0.624,"5min":0.57999999999999996,"15min":0.57299999999999995},"min":{"1min":0.45100000000000001,"5min":0.38800000000000001,"15min":0.38800000000000001},"max":{"1min":0.90200000000000002,"5min":1.0109999999999999,"15min":1.0109999999999999},"last":0.52900000000000003},{"interface":"front","average":{"1min":0.64900000000000002,"5min":0.621,"15min":0.61599999999999999},"min":{"1min":0.495,"5min":0.47299999999999998,"15min":0.47299999999999998},"max":{"1min":0.88500000000000001,"5min":0.88600000000000001,"15min":0.88600000000000001},"last":0.56200000000000006}]},{"osd":4,"last update":"Fri Jul 14 19:28:59 2023","interfaces":[{"interface":"back","average":{"1min":0.60799999999999998,"5min":0.58899999999999997,"15min":0.58599999999999997},"min":{"1min":0.36099999999999999,"5min":0.36099999999999999,"15min":0.36099999999999999},"max":{"1min":1.125,"5min":1.125,"15min":1.125},"last":0.41399999999999998},{"interface":"front","average":{"1min":0.58799999999999997,"5min":0.54200000000000004,"15min":0.53400000000000003},"min":{"1min":0.33600000000000002,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":0.879,"5min":0.879,"15min":0.879},"last":0.39500000000000002}]},{"osd":5,"last update":"Fri Jul 14 19:28:59 2023","interfaces":[{"interface":"back","average":{"1min":0.69899999999999995,"5min":0.68300000000000005,"15min":0.68000000000000005},"min":{"1min":0.38300000000000001,"5min":0.38300000000000001,"15min":0.38300000000000001},"max":{"1min":1.173,"5min":1.173,"15min":1.173},"last":0.66500000000000004},{"interface":"front","average":{"1min":0.69299999999999995,"5min":0.69299999999999995,"15min":0.69299999999999995},"min":{"1min":0.40200000000000002,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":1.0329999999999999,"5min":1.0329999999999999,"15min":1.0329999999999999},"last":0.628}]},{"osd":6,"last update":"Fri Jul 14 19:28:59 2023","interfaces":[{"interface":"back","average":{"1min":0.72699999999999998,"5min":0.68799999999999994,"15min":0.68200000000000005},"min":{"1min":0.48499999999999999,"5min":0.42799999999999999,"15min":0.42799999999999999},"max":{"1min":1.0309999999999999,"5min":1.0309999999999999,"15min":1.0309999999999999},"last":0.60199999999999998},{"interface":"front","average":{"1min":0.71899999999999997,"5min":0.69299999999999995,"15min":0.68899999999999995},"min":{"1min":0.50800000000000001,"5min":0.443,"15min":0.443},"max":{"1min":0.93999999999999995,"5min":1.0209999999999999,"15min":1.0209999999999999},"last":0.45300000000000001}]},{"osd":8,"last update":"Fri Jul 14 19:28:18 2023","interfaces":[{"interface":"back","average":{"1min":0.754,"5min":0.754,"15min":0.754},"min":{"1min":0.56499999999999995,"5min":0.56499999999999995,"15min":0.56499999999999995},"max":{"1min":1.0720000000000001,"5min":1.0720000000000001,"15min":1.0720000000000001},"last":0.73599999999999999},{"interface":"front","average":{"1min":0.753,"5min":0.753,"15min":0.753},"min":{"1min":0.55400000000000005,"5min":0.55400000000000005,"15min":0.55400000000000005},"max":{"1min":1.141,"5min":1.141,"15min":1.141},"last":0.61699999999999999}]},{"osd":9,"last update":"Fri Jul 14 19:28:39 2023","interfaces":[{"interface":"back","average":{"1min":0.74099999999999999,"5min":0.74099999999999999,"15min":0.74099999999999999},"min":{"1min":0.53400000000000003,"5min":0.53400000000000003,"15min":0.53400000000000003},"max":{"1min":1.0489999999999999,"5min":1.0489999999999999,"15min":1.0489999999999999},"last":0.505},{"interface":"front","average":{"1min":0.75900000000000001,"5min":0.75900000000000001,"15min":0.75900000000000001},"min":{"1min":0.53300000000000003,"5min":0.53300000000000003,"15min":0.53300000000000003},"max":{"1min":1.1519999999999999,"5min":1.1519999999999999,"15min":1.1519999999999999},"last":0.58299999999999996}]},{"osd":10,"last update":"Fri Jul 14 19:28:59 2023","interfaces":[{"interface":"back","average":{"1min":0.72999999999999998,"5min":0.72999999999999998,"15min":0.72999999999999998},"min":{"1min":0.38700000000000001,"5min":0.38700000000000001,"15min":0.38700000000000001},"max":{"1min":1.077,"5min":1.077,"15min":1.077},"last":0.69199999999999995},{"interface":"front","average":{"1min":0.74199999999999999,"5min":0.74199999999999999,"15min":0.74199999999999999},"min":{"1min":0.38500000000000001,"5min":0.38500000000000001,"15min":0.38500000000000001},"max":{"1min":1.117,"5min":1.117,"15min":1.117},"last":0.65200000000000002}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.76600000000000001}]}]},{"osd":6,"up_from":45,"seq":193273528353,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27892,"kb_used_data":872,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717260,"statfs":{"total":95995035648,"available":95966474240,"internally_reserved":0,"allocated":892928,"data_stored":428457,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":27654411},"hb_peers":[0,1,2,3,4,5,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jul 14 19:28:36 2023","interfaces":[{"interface":"back","average":{"1min":0.42099999999999999,"5min":0.38200000000000001,"15min":0.376},"min":{"1min":0.29099999999999998,"5min":0.23400000000000001,"15min":0.23400000000000001},"max":{"1min":0.63,"5min":0.63,"15min":0.63},"last":0.34399999999999997},{"interface":"front","average":{"1min":0.42699999999999999,"5min":0.38900000000000001,"15min":0.38300000000000001},"min":{"1min":0.32000000000000001,"5min":0.249,"15min":0.249},"max":{"1min":0.63900000000000001,"5min":0.63900000000000001,"15min":0.63900000000000001},"last":0.307}]},{"osd":1,"last update":"Fri Jul 14 19:28:36 2023","interfaces":[{"interface":"back","average":{"1min":0.66400000000000003,"5min":0.53600000000000003,"15min":0.51400000000000001},"min":{"1min":0.41699999999999998,"5min":0.29899999999999999,"15min":0.29899999999999999},"max":{"1min":0.83999999999999997,"5min":0.83999999999999997,"15min":0.83999999999999997},"last":0.51500000000000001},{"interface":"front","average":{"1min":0.58199999999999996,"5min":0.505,"15min":0.49199999999999999},"min":{"1min":0.36699999999999999,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":0.872,"5min":0.872,"15min":0.872},"last":0.435}]},{"osd":2,"last update":"Fri Jul 14 19:28:36 2023","interfaces":[{"interface":"back","average":{"1min":0.58399999999999996,"5min":0.58799999999999997,"15min":0.58899999999999997},"min":{"1min":0.40600000000000003,"5min":0.27600000000000002,"15min":0.27600000000000002},"max":{"1min":0.68799999999999994,"5min":0.76800000000000002,"15min":0.76800000000000002},"last":0.46899999999999997},{"interface":"front","average":{"1min":0.628,"5min":0.61199999999999999,"15min":0.60899999999999999},"min":{"1min":0.46500000000000002,"5min":0.378,"15min":0.378},"max":{"1min":0.83299999999999996,"5min":0.83299999999999996,"15min":0.83299999999999996},"last":0.621}]},{"osd":3,"last update":"Fri Jul 14 19:28:36 2023","interfaces":[{"interface":"back","average":{"1min":0.47399999999999998,"5min":0.47599999999999998,"15min":0.47599999999999998},"min":{"1min":0.30099999999999999,"5min":0.30099999999999999,"15min":0.30099999999999999},"max":{"1min":0.64200000000000002,"5min":0.78500000000000003,"15min":0.78500000000000003},"last":0.55700000000000005},{"interface":"front","average":{"1min":0.45100000000000001,"5min":0.441,"15min":0.439},"min":{"1min":0.315,"5min":0.28999999999999998,"15min":0.28999999999999998},"max":{"1min":0.75700000000000001,"5min":0.75700000000000001,"15min":0.75700000000000001},"last":0.35799999999999998}]},{"osd":4,"last update":"Fri Jul 14 19:28:36 2023","interfaces":[{"interface":"back","average":{"1min":0.61099999999999999,"5min":0.53500000000000003,"15min":0.52200000000000002},"min":{"1min":0.39500000000000002,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":0.82899999999999996,"5min":0.82899999999999996,"15min":0.82899999999999996},"last":0.40300000000000002},{"interface":"front","average":{"1min":0.67300000000000004,"5min":0.62,"15min":0.61099999999999999},"min":{"1min":0.38700000000000001,"5min":0.38700000000000001,"15min":0.38700000000000001},"max":{"1min":0.96899999999999997,"5min":0.96899999999999997,"15min":0.96899999999999997},"last":0.45300000000000001}]},{"osd":5,"last update":"Fri Jul 14 19:28:36 2023","interfaces":[{"interface":"back","average":{"1min":0.56799999999999995,"5min":0.57399999999999995,"15min":0.57499999999999996},"min":{"1min":0.38300000000000001,"5min":0.38300000000000001,"15min":0.38300000000000001},"max":{"1min":0.752,"5min":0.81799999999999995,"15min":0.81799999999999995},"last":0.56899999999999995},{"interface":"front","average":{"1min":0.64400000000000002,"5min":0.63500000000000001,"15min":0.63300000000000001},"min":{"1min":0.47399999999999998,"5min":0.46000000000000002,"15min":0.46000000000000002},"max":{"1min":0.81399999999999995,"5min":0.88500000000000001,"15min":0.88500000000000001},"last":0.54000000000000004}]},{"osd":7,"last update":"Fri Jul 14 19:28:59 2023","interfaces":[{"interface":"back","average":{"1min":0.68600000000000005,"5min":0.68100000000000005,"15min":0.68000000000000005},"min":{"1min":0.47599999999999998,"5min":0.44500000000000001,"15min":0.44500000000000001},"max":{"1min":1.0029999999999999,"5min":1.0029999999999999,"15min":1.0029999999999999},"last":0.48699999999999999},{"interface":"front","average":{"1min":0.69699999999999995,"5min":0.69699999999999995,"15min":0.69699999999999995},"min":{"1min":0.50900000000000001,"5min":0.46700000000000003,"15min":0.46700000000000003},"max":{"1min":0.86599999999999999,"5min":1.0189999999999999,"15min":1.0189999999999999},"last":0.41899999999999998}]},{"osd":8,"last update":"Fri Jul 14 19:28:17 2023","interfaces":[{"interface":"back","average":{"1min":0.70999999999999996,"5min":0.70999999999999996,"15min":0.70999999999999996},"min":{"1min":0.50900000000000001,"5min":0.50900000000000001,"15min":0.50900000000000001},"max":{"1min":1.1000000000000001,"5min":1.1000000000000001,"15min":1.1000000000000001},"last":0.60299999999999998},{"interface":"front","average":{"1min":0.59799999999999998,"5min":0.59799999999999998,"15min":0.59799999999999998},"min":{"1min":0.434,"5min":0.434,"15min":0.434},"max":{"1min":0.754,"5min":0.754,"15min":0.754},"last":0.53300000000000003}]},{"osd":9,"last update":"Fri Jul 14 19:28:41 2023","interfaces":[{"interface":"back","average":{"1min":0.65100000000000002,"5min":0.65100000000000002,"15min":0.65100000000000002},"min":{"1min":0.434,"5min":0.434,"15min":0.434},"max":{"1min":0.90600000000000003,"5min":0.90600000000000003,"15min":0.90600000000000003},"last":0.58699999999999997},{"interface":"front","average":{"1min":0.76900000000000002,"5min":0.76900000000000002,"15min":0.76900000000000002},"min":{"1min":0.51100000000000001,"5min":0.51100000000000001,"15min":0.51100000000000001},"max":{"1min":1.1379999999999999,"5min":1.1379999999999999,"15min":1.1379999999999999},"last":0.67100000000000004}]},{"osd":10,"last update":"Fri Jul 14 19:28:59 2023","interfaces":[{"interface":"back","average":{"1min":0.63900000000000001,"5min":0.63900000000000001,"15min":0.63900000000000001},"min":{"1min":0.505,"5min":0.505,"15min":0.505},"max":{"1min":0.82999999999999996,"5min":0.82999999999999996,"15min":0.82999999999999996},"last":0.498},{"interface":"front","average":{"1min":0.76800000000000002,"5min":0.76800000000000002,"15min":0.76800000000000002},"min":{"1min":0.58999999999999997,"5min":0.58999999999999997,"15min":0.58999999999999997},"max":{"1min":1.0329999999999999,"5min":1.0329999999999999,"15min":1.0329999999999999},"last":0.64100000000000001}]}]},{"osd":1,"up_from":13,"seq":55834574902,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27912,"kb_used_data":884,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717240,"statfs":{"total":95995035648,"available":95966453760,"internally_reserved":0,"allocated":905216,"data_stored":430021,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":27654411},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jul 14 19:28:58 2023","interfaces":[{"interface":"back","average":{"1min":0.53800000000000003,"5min":0.498,"15min":0.47099999999999997},"min":{"1min":0.43099999999999999,"5min":0.215,"15min":0.215},"max":{"1min":0.82399999999999995,"5min":1.6970000000000001,"15min":1.6970000000000001},"last":0.42699999999999999},{"interface":"front","average":{"1min":0.58599999999999997,"5min":0.48699999999999999,"15min":0.46100000000000002},"min":{"1min":0.39100000000000001,"5min":0.23699999999999999,"15min":0.23699999999999999},"max":{"1min":0.82899999999999996,"5min":0.82899999999999996,"15min":0.82899999999999996},"last":0.51000000000000001}]},{"osd":2,"last update":"Fri Jul 14 19:28:13 2023","interfaces":[{"interface":"back","average":{"1min":0.53900000000000003,"5min":0.496,"15min":0.48699999999999999},"min":{"1min":0.34799999999999998,"5min":0.31,"15min":0.31},"max":{"1min":0.77500000000000002,"5min":0.77500000000000002,"15min":0.77500000000000002},"last":0.40600000000000003},{"interface":"front","average":{"1min":0.55900000000000005,"5min":0.52400000000000002,"15min":0.498},"min":{"1min":0.317,"5min":0.317,"15min":0.317},"max":{"1min":0.76700000000000002,"5min":1.74,"15min":1.74},"last":0.55300000000000005}]},{"osd":3,"last update":"Fri Jul 14 19:28:33 2023","interfaces":[{"interface":"back","average":{"1min":0.59099999999999997,"5min":0.55400000000000005,"15min":0.53600000000000003},"min":{"1min":0.35499999999999998,"5min":0.32200000000000001,"15min":0.32200000000000001},"max":{"1min":0.73899999999999999,"5min":1.8460000000000001,"15min":1.8460000000000001},"last":0.53300000000000003},{"interface":"front","average":{"1min":0.62,"5min":0.55900000000000005,"15min":0.54800000000000004},"min":{"1min":0.47499999999999998,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.81499999999999995,"5min":0.81499999999999995,"15min":0.81499999999999995},"last":0.626}]},{"osd":4,"last update":"Fri Jul 14 19:28:58 2023","interfaces":[{"interface":"back","average":{"1min":0.51400000000000001,"5min":0.49199999999999999,"15min":0.502},"min":{"1min":0.314,"5min":0.29899999999999999,"15min":0.29899999999999999},"max":{"1min":0.78100000000000003,"5min":2.2029999999999998,"15min":2.2029999999999998},"last":0.379},{"interface":"front","average":{"1min":0.53700000000000003,"5min":0.495,"15min":0.49199999999999999},"min":{"1min":0.309,"5min":0.26400000000000001,"15min":0.26400000000000001},"max":{"1min":0.90400000000000003,"5min":1.901,"15min":1.901},"last":0.65400000000000003}]},{"osd":5,"last update":"Fri Jul 14 19:28:13 2023","interfaces":[{"interface":"back","average":{"1min":0.626,"5min":0.57399999999999995,"15min":0.56499999999999995},"min":{"1min":0.308,"5min":0.308,"15min":0.308},"max":{"1min":0.78500000000000003,"5min":0.78500000000000003,"15min":0.78500000000000003},"last":0.57199999999999995},{"interface":"front","average":{"1min":0.67700000000000005,"5min":0.65600000000000003,"15min":0.65200000000000002},"min":{"1min":0.442,"5min":0.42699999999999999,"15min":0.42699999999999999},"max":{"1min":0.84299999999999997,"5min":1.8,"15min":1.8},"last":0.64000000000000001}]},{"osd":6,"last update":"Fri Jul 14 19:28:39 2023","interfaces":[{"interface":"back","average":{"1min":0.64700000000000002,"5min":0.63500000000000001,"15min":0.63300000000000001},"min":{"1min":0.45000000000000001,"5min":0.40999999999999998,"15min":0.40999999999999998},"max":{"1min":0.91500000000000004,"5min":1.8300000000000001,"15min":1.8300000000000001},"last":0.59299999999999997},{"interface":"front","average":{"1min":0.68500000000000005,"5min":0.66100000000000003,"15min":0.65700000000000003},"min":{"1min":0.46600000000000003,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":1.026,"5min":1.875,"15min":1.875},"last":0.70599999999999996}]},{"osd":7,"last update":"Fri Jul 14 19:28:58 2023","interfaces":[{"interface":"back","average":{"1min":0.67400000000000004,"5min":0.56599999999999995,"15min":0.54800000000000004},"min":{"1min":0.36199999999999999,"5min":0.36199999999999999,"15min":0.36199999999999999},"max":{"1min":0.95399999999999996,"5min":0.95399999999999996,"15min":0.95399999999999996},"last":0.49099999999999999},{"interface":"front","average":{"1min":0.70199999999999996,"5min":0.60799999999999998,"15min":0.59199999999999997},"min":{"1min":0.39200000000000002,"5min":0.38400000000000001,"15min":0.38400000000000001},"max":{"1min":1.24,"5min":1.24,"15min":1.24},"last":0.60899999999999999}]},{"osd":8,"last update":"Fri Jul 14 19:28:16 2023","interfaces":[{"interface":"back","average":{"1min":0.69199999999999995,"5min":0.69199999999999995,"15min":0.69199999999999995},"min":{"1min":0.42199999999999999,"5min":0.42199999999999999,"15min":0.42199999999999999},"max":{"1min":0.93100000000000005,"5min":0.93100000000000005,"15min":0.93100000000000005},"last":0.66200000000000003},{"interface":"front","average":{"1min":0.73999999999999999,"5min":0.73999999999999999,"15min":0.73999999999999999},"min":{"1min":0.505,"5min":0.505,"15min":0.505},"max":{"1min":1.0449999999999999,"5min":1.0449999999999999,"15min":1.0449999999999999},"last":0.79800000000000004}]},{"osd":9,"last update":"Fri Jul 14 19:28:39 2023","interfaces":[{"interface":"back","average":{"1min":0.77600000000000002,"5min":0.77600000000000002,"15min":0.77600000000000002},"min":{"1min":0.52100000000000002,"5min":0.52100000000000002,"15min":0.52100000000000002},"max":{"1min":1.042,"5min":1.042,"15min":1.042},"last":0.72299999999999998},{"interface":"front","average":{"1min":0.76100000000000001,"5min":0.76100000000000001,"15min":0.76100000000000001},"min":{"1min":0.51200000000000001,"5min":0.51200000000000001,"15min":0.51200000000000001},"max":{"1min":1.175,"5min":1.175,"15min":1.175},"last":0.751}]},{"osd":10,"last update":"Fri Jul 14 19:28:58 2023","interfaces":[{"interface":"back","average":{"1min":0.73699999999999999,"5min":0.73699999999999999,"15min":0.73699999999999999},"min":{"1min":0.44800000000000001,"5min":0.44800000000000001,"15min":0.44800000000000001},"max":{"1min":1.079,"5min":1.079,"15min":1.079},"last":0.69299999999999995},{"interface":"front","average":{"1min":0.81799999999999995,"5min":0.81799999999999995,"15min":0.81799999999999995},"min":{"1min":0.58799999999999997,"5min":0.58799999999999997,"15min":0.58799999999999997},"max":{"1min":1.127,"5min":1.127,"15min":1.127},"last":0.67700000000000005}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.73599999999999999}]}]},{"osd":0,"up_from":8,"seq":34359738426,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27904,"kb_used_data":884,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717248,"statfs":{"total":95995035648,"available":95966461952,"internally_reserved":0,"allocated":905216,"data_stored":429151,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":27654411},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Jul 14 19:28:52 2023","interfaces":[{"interface":"back","average":{"1min":0.629,"5min":0.52600000000000002,"15min":0.503},"min":{"1min":0.35999999999999999,"5min":0.28000000000000003,"15min":0.28000000000000003},"max":{"1min":1.79,"5min":1.79,"15min":1.79},"last":0.41199999999999998},{"interface":"front","average":{"1min":0.57199999999999995,"5min":0.51700000000000002,"15min":0.505},"min":{"1min":0.32000000000000001,"5min":0.313,"15min":0.313},"max":{"1min":1.605,"5min":1.605,"15min":1.605},"last":0.40200000000000002}]},{"osd":2,"last update":"Fri Jul 14 19:28:17 2023","interfaces":[{"interface":"back","average":{"1min":0.64700000000000002,"5min":0.54500000000000004,"15min":0.51700000000000002},"min":{"1min":0.40699999999999997,"5min":0.34599999999999997,"15min":0.34599999999999997},"max":{"1min":0.77700000000000002,"5min":0.77700000000000002,"15min":0.77700000000000002},"last":0.47699999999999998},{"interface":"front","average":{"1min":0.60799999999999998,"5min":0.54100000000000004,"15min":0.52100000000000002},"min":{"1min":0.35899999999999999,"5min":0.29999999999999999,"15min":0.29999999999999999},"max":{"1min":0.81399999999999995,"5min":0.81399999999999995,"15min":0.81399999999999995},"last":0.38600000000000001}]},{"osd":3,"last update":"Fri Jul 14 19:27:35 2023","interfaces":[{"interface":"back","average":{"1min":0.439,"5min":0.41099999999999998,"15min":0.40600000000000003},"min":{"1min":0.29399999999999998,"5min":0.26400000000000001,"15min":0.26400000000000001},"max":{"1min":0.72299999999999998,"5min":0.72299999999999998,"15min":0.72299999999999998},"last":0.36299999999999999},{"interface":"front","average":{"1min":0.44900000000000001,"5min":0.437,"15min":0.435},"min":{"1min":0.245,"5min":0.24099999999999999,"15min":0.24099999999999999},"max":{"1min":0.61499999999999999,"5min":0.61499999999999999,"15min":0.61499999999999999},"last":0.32900000000000001}]},{"osd":4,"last update":"Fri Jul 14 19:29:02 2023","interfaces":[{"interface":"back","average":{"1min":0.57099999999999995,"5min":0.53900000000000003,"15min":0.52500000000000002},"min":{"1min":0.38800000000000001,"5min":0.377,"15min":0.377},"max":{"1min":1.139,"5min":1.139,"15min":1.139},"last":0.44800000000000001},{"interface":"front","average":{"1min":0.58499999999999996,"5min":0.55000000000000004,"15min":0.53700000000000003},"min":{"1min":0.34799999999999998,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":1.2190000000000001,"5min":1.2190000000000001,"15min":1.2190000000000001},"last":0.46000000000000002}]},{"osd":5,"last update":"Fri Jul 14 19:28:17 2023","interfaces":[{"interface":"back","average":{"1min":0.626,"5min":0.60499999999999998,"15min":0.60099999999999998},"min":{"1min":0.47999999999999998,"5min":0.45900000000000002,"15min":0.45900000000000002},"max":{"1min":0.81799999999999995,"5min":0.81799999999999995,"15min":0.81799999999999995},"last":0.57099999999999995},{"interface":"front","average":{"1min":0.64400000000000002,"5min":0.61599999999999999,"15min":0.61099999999999999},"min":{"1min":0.45600000000000002,"5min":0.436,"15min":0.436},"max":{"1min":0.85499999999999998,"5min":0.85499999999999998,"15min":0.85499999999999998},"last":0.624}]},{"osd":6,"last update":"Fri Jul 14 19:28:36 2023","interfaces":[{"interface":"back","average":{"1min":0.48899999999999999,"5min":0.495,"15min":0.496},"min":{"1min":0.25900000000000001,"5min":0.25900000000000001,"15min":0.25900000000000001},"max":{"1min":0.80000000000000004,"5min":0.80000000000000004,"15min":0.80000000000000004},"last":0.50700000000000001},{"interface":"front","average":{"1min":0.55300000000000005,"5min":0.51400000000000001,"15min":0.50800000000000001},"min":{"1min":0.32500000000000001,"5min":0.32500000000000001,"15min":0.32500000000000001},"max":{"1min":0.74199999999999999,"5min":0.76300000000000001,"15min":0.76300000000000001},"last":0.42799999999999999}]},{"osd":7,"last update":"Fri Jul 14 19:27:57 2023","interfaces":[{"interface":"back","average":{"1min":0.60299999999999998,"5min":0.60299999999999998,"15min":0.60299999999999998},"min":{"1min":0.41999999999999998,"5min":0.41999999999999998,"15min":0.41999999999999998},"max":{"1min":0.83599999999999997,"5min":0.83599999999999997,"15min":0.83599999999999997},"last":0.52100000000000002},{"interface":"front","average":{"1min":0.65200000000000002,"5min":0.65200000000000002,"15min":0.65200000000000002},"min":{"1min":0.47299999999999998,"5min":0.47299999999999998,"15min":0.47299999999999998},"max":{"1min":0.89500000000000002,"5min":0.89500000000000002,"15min":0.89500000000000002},"last":0.40799999999999997}]},{"osd":8,"last update":"Fri Jul 14 19:28:17 2023","interfaces":[{"interface":"back","average":{"1min":0.69299999999999995,"5min":0.69299999999999995,"15min":0.69299999999999995},"min":{"1min":0.57599999999999996,"5min":0.57599999999999996,"15min":0.57599999999999996},"max":{"1min":0.80700000000000005,"5min":0.80700000000000005,"15min":0.80700000000000005},"last":0.53500000000000003},{"interface":"front","average":{"1min":0.71399999999999997,"5min":0.71399999999999997,"15min":0.71399999999999997},"min":{"1min":0.53400000000000003,"5min":0.53400000000000003,"15min":0.53400000000000003},"max":{"1min":0.86499999999999999,"5min":0.86499999999999999,"15min":0.86499999999999999},"last":0.55500000000000005}]},{"osd":9,"last update":"Fri Jul 14 19:28:39 2023","interfaces":[{"interface":"back","average":{"1min":0.63500000000000001,"5min":0.63500000000000001,"15min":0.63500000000000001},"min":{"1min":0.36499999999999999,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":0.85299999999999998,"5min":0.85299999999999998,"15min":0.85299999999999998},"last":0.54500000000000004},{"interface":"front","average":{"1min":0.64200000000000002,"5min":0.64200000000000002,"15min":0.64200000000000002},"min":{"1min":0.32300000000000001,"5min":0.32300000000000001,"15min":0.32300000000000001},"max":{"1min":0.96099999999999997,"5min":0.96099999999999997,"15min":0.96099999999999997},"last":0.48899999999999999}]},{"osd":10,"last update":"Fri Jul 14 19:29:02 2023","interfaces":[{"interface":"back","average":{"1min":0.754,"5min":0.754,"15min":0.754},"min":{"1min":0.45900000000000002,"5min":0.45900000000000002,"15min":0.45900000000000002},"max":{"1min":1.776,"5min":1.776,"15min":1.776},"last":0.52400000000000002},{"interface":"front","average":{"1min":0.72499999999999998,"5min":0.72499999999999998,"15min":0.72499999999999998},"min":{"1min":0.502,"5min":0.502,"15min":0.502},"max":{"1min":1.522,"5min":1.522,"15min":1.522},"last":0.58099999999999996}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.60399999999999998}]}]},{"osd":2,"up_from":18,"seq":77309411377,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27916,"kb_used_data":888,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717236,"statfs":{"total":95995035648,"available":95966449664,"internally_reserved":0,"allocated":909312,"data_stored":430055,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":27654411},"hb_peers":[0,1,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jul 14 19:28:16 2023","interfaces":[{"interface":"back","average":{"1min":0.53600000000000003,"5min":0.503,"15min":0.49199999999999999},"min":{"1min":0.34399999999999997,"5min":0.312,"15min":0.312},"max":{"1min":0.89000000000000001,"5min":0.89000000000000001,"15min":0.89000000000000001},"last":0.75600000000000001},{"interface":"front","average":{"1min":0.51300000000000001,"5min":0.49099999999999999,"15min":0.48699999999999999},"min":{"1min":0.31900000000000001,"5min":0.31900000000000001,"15min":0.31900000000000001},"max":{"1min":0.65100000000000002,"5min":0.69699999999999995,"15min":0.69699999999999995},"last":0.47699999999999998}]},{"osd":1,"last update":"Fri Jul 14 19:28:16 2023","interfaces":[{"interface":"back","average":{"1min":0.628,"5min":0.53300000000000003,"15min":0.51500000000000001},"min":{"1min":0.40000000000000002,"5min":0.316,"15min":0.316},"max":{"1min":0.89300000000000002,"5min":0.89300000000000002,"15min":0.89300000000000002},"last":0.54000000000000004},{"interface":"front","average":{"1min":0.64600000000000002,"5min":0.54600000000000004,"15min":0.52200000000000002},"min":{"1min":0.379,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":0.91100000000000003,"5min":0.91100000000000003,"15min":0.91100000000000003},"last":0.86499999999999999}]},{"osd":3,"last update":"Fri Jul 14 19:28:36 2023","interfaces":[{"interface":"back","average":{"1min":0.67900000000000005,"5min":0.57399999999999995,"15min":0.55400000000000005},"min":{"1min":0.48199999999999998,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":0.91700000000000004,"5min":0.91700000000000004,"15min":0.91700000000000004},"last":0.82999999999999996},{"interface":"front","average":{"1min":0.60299999999999998,"5min":0.54300000000000004,"15min":0.53300000000000003},"min":{"1min":0.44,"5min":0.34699999999999998,"15min":0.34699999999999998},"max":{"1min":0.76900000000000002,"5min":0.76900000000000002,"15min":0.76900000000000002},"last":0.67100000000000004}]},{"osd":4,"last update":"Fri Jul 14 19:27:59 2023","interfaces":[{"interface":"back","average":{"1min":0.53800000000000003,"5min":0.52200000000000002,"15min":0.52000000000000002},"min":{"1min":0.38800000000000001,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":0.68400000000000005,"5min":0.70399999999999996,"15min":0.70399999999999996},"last":0.68500000000000005},{"interface":"front","average":{"1min":0.63600000000000001,"5min":0.58399999999999996,"15min":0.57599999999999996},"min":{"1min":0.40100000000000002,"5min":0.40100000000000002,"15min":0.40100000000000002},"max":{"1min":0.81699999999999995,"5min":0.81699999999999995,"15min":0.81699999999999995},"last":0.63500000000000001}]},{"osd":5,"last update":"Fri Jul 14 19:28:16 2023","interfaces":[{"interface":"back","average":{"1min":0.47399999999999998,"5min":0.433,"15min":0.42599999999999999},"min":{"1min":0.308,"5min":0.26500000000000001,"15min":0.26500000000000001},"max":{"1min":0.66900000000000004,"5min":0.66900000000000004,"15min":0.66900000000000004},"last":0.39300000000000002},{"interface":"front","average":{"1min":0.54500000000000004,"5min":0.47699999999999998,"15min":0.46600000000000003},"min":{"1min":0.34999999999999998,"5min":0.313,"15min":0.313},"max":{"1min":0.77900000000000003,"5min":0.77900000000000003,"15min":0.77900000000000003},"last":0.72099999999999997}]},{"osd":6,"last update":"Fri Jul 14 19:28:43 2023","interfaces":[{"interface":"back","average":{"1min":0.59899999999999998,"5min":0.55500000000000005,"15min":0.54800000000000004},"min":{"1min":0.33500000000000002,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":0.93400000000000005,"5min":0.93400000000000005,"15min":0.93400000000000005},"last":0.65400000000000003},{"interface":"front","average":{"1min":0.69199999999999995,"5min":0.60899999999999999,"15min":0.59499999999999997},"min":{"1min":0.46999999999999997,"5min":0.371,"15min":0.371},"max":{"1min":0.92000000000000004,"5min":0.92000000000000004,"15min":0.92000000000000004},"last":0.61399999999999999}]},{"osd":7,"last update":"Fri Jul 14 19:27:59 2023","interfaces":[{"interface":"back","average":{"1min":0.622,"5min":0.622,"15min":0.622},"min":{"1min":0.44600000000000001,"5min":0.44600000000000001,"15min":0.44600000000000001},"max":{"1min":0.91900000000000004,"5min":0.91900000000000004,"15min":0.91900000000000004},"last":0.59899999999999998},{"interface":"front","average":{"1min":0.67500000000000004,"5min":0.67500000000000004,"15min":0.67500000000000004},"min":{"1min":0.433,"5min":0.433,"15min":0.433},"max":{"1min":0.96399999999999997,"5min":0.96399999999999997,"15min":0.96399999999999997},"last":0.66000000000000003}]},{"osd":8,"last update":"Fri Jul 14 19:28:16 2023","interfaces":[{"interface":"back","average":{"1min":0.627,"5min":0.627,"15min":0.627},"min":{"1min":0.35799999999999998,"5min":0.35799999999999998,"15min":0.35799999999999998},"max":{"1min":0.84899999999999998,"5min":0.84899999999999998,"15min":0.84899999999999998},"last":0.91600000000000004},{"interface":"front","average":{"1min":0.625,"5min":0.625,"15min":0.625},"min":{"1min":0.375,"5min":0.375,"15min":0.375},"max":{"1min":0.94199999999999995,"5min":0.94199999999999995,"15min":0.94199999999999995},"last":0.77600000000000002}]},{"osd":9,"last update":"Fri Jul 14 19:28:43 2023","interfaces":[{"interface":"back","average":{"1min":0.67800000000000005,"5min":0.67800000000000005,"15min":0.67800000000000005},"min":{"1min":0.45300000000000001,"5min":0.45300000000000001,"15min":0.45300000000000001},"max":{"1min":0.90700000000000003,"5min":0.90700000000000003,"15min":0.90700000000000003},"last":0.81499999999999995},{"interface":"front","average":{"1min":0.65300000000000002,"5min":0.65300000000000002,"15min":0.65300000000000002},"min":{"1min":0.55200000000000005,"5min":0.55200000000000005,"15min":0.55200000000000005},"max":{"1min":0.79700000000000004,"5min":0.79700000000000004,"15min":0.79700000000000004},"last":0.74199999999999999}]},{"osd":10,"last update":"Fri Jul 14 19:28:59 2023","interfaces":[{"interface":"back","average":{"1min":0.63200000000000001,"5min":0.63200000000000001,"15min":0.63200000000000001},"min":{"1min":0.46800000000000003,"5min":0.46800000000000003,"15min":0.46800000000000003},"max":{"1min":0.79700000000000004,"5min":0.79700000000000004,"15min":0.79700000000000004},"last":0.70599999999999996},{"interface":"front","average":{"1min":0.745,"5min":0.745,"15min":0.745},"min":{"1min":0.61899999999999999,"5min":0.61899999999999999,"15min":0.61899999999999999},"max":{"1min":0.88900000000000001,"5min":0.88900000000000001,"15min":0.88900000000000001},"last":0.96399999999999997}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.012}]}]},{"osd":3,"up_from":25,"seq":107374182445,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27904,"kb_used_data":876,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717248,"statfs":{"total":95995035648,"available":95966461952,"internally_reserved":0,"allocated":897024,"data_stored":428230,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":27654411},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jul 14 19:28:33 2023","interfaces":[{"interface":"back","average":{"1min":0.41099999999999998,"5min":0.34100000000000003,"15min":0.32300000000000001},"min":{"1min":0.27000000000000002,"5min":0.221,"15min":0.221},"max":{"1min":0.61299999999999999,"5min":0.61299999999999999,"15min":0.61299999999999999},"last":0.379},{"interface":"front","average":{"1min":0.435,"5min":0.375,"15min":0.35699999999999998},"min":{"1min":0.30299999999999999,"5min":0.23899999999999999,"15min":0.23899999999999999},"max":{"1min":0.63600000000000001,"5min":0.63600000000000001,"15min":0.63600000000000001},"last":0.40500000000000003}]},{"osd":1,"last update":"Fri Jul 14 19:28:33 2023","interfaces":[{"interface":"back","average":{"1min":0.54400000000000004,"5min":0.50800000000000001,"15min":0.49399999999999999},"min":{"1min":0.38500000000000001,"5min":0.36199999999999999,"15min":0.36199999999999999},"max":{"1min":0.84199999999999997,"5min":0.84199999999999997,"15min":0.84199999999999997},"last":0.52800000000000002},{"interface":"front","average":{"1min":0.56999999999999995,"5min":0.52000000000000002,"15min":0.502},"min":{"1min":0.40200000000000002,"5min":0.30399999999999999,"15min":0.30399999999999999},"max":{"1min":0.71099999999999997,"5min":0.71099999999999997,"15min":0.71099999999999997},"last":0.70399999999999996}]},{"osd":2,"last update":"Fri Jul 14 19:28:33 2023","interfaces":[{"interface":"back","average":{"1min":0.65600000000000003,"5min":0.56899999999999995,"15min":0.54100000000000004},"min":{"1min":0.48699999999999999,"5min":0.36199999999999999,"15min":0.36199999999999999},"max":{"1min":0.93899999999999995,"5min":0.93899999999999995,"15min":0.93899999999999995},"last":0.72299999999999998},{"interface":"front","average":{"1min":0.58899999999999997,"5min":0.54500000000000004,"15min":0.52700000000000002},"min":{"1min":0.441,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":0.74199999999999999,"5min":0.78300000000000003,"15min":0.78300000000000003},"last":0.76200000000000001}]},{"osd":4,"last update":"Fri Jul 14 19:28:51 2023","interfaces":[{"interface":"back","average":{"1min":0.66100000000000003,"5min":0.55600000000000005,"15min":0.52200000000000002},"min":{"1min":0.44800000000000001,"5min":0.32300000000000001,"15min":0.32300000000000001},"max":{"1min":0.91100000000000003,"5min":0.98499999999999999,"15min":0.98499999999999999},"last":0.68600000000000005},{"interface":"front","average":{"1min":0.60499999999999998,"5min":0.55000000000000004,"15min":0.52900000000000003},"min":{"1min":0.45500000000000002,"5min":0.39300000000000002,"15min":0.39300000000000002},"max":{"1min":0.90200000000000002,"5min":0.90200000000000002,"15min":0.90200000000000002},"last":0.63700000000000001}]},{"osd":5,"last update":"Fri Jul 14 19:28:11 2023","interfaces":[{"interface":"back","average":{"1min":0.63100000000000001,"5min":0.57899999999999996,"15min":0.57099999999999995},"min":{"1min":0.44900000000000001,"5min":0.44600000000000001,"15min":0.44600000000000001},"max":{"1min":0.81299999999999994,"5min":0.81299999999999994,"15min":0.81299999999999994},"last":0.60899999999999999},{"interface":"front","average":{"1min":0.69799999999999995,"5min":0.61099999999999999,"15min":0.59699999999999998},"min":{"1min":0.48199999999999998,"5min":0.39300000000000002,"15min":0.39300000000000002},"max":{"1min":0.98199999999999998,"5min":0.98199999999999998,"15min":0.98199999999999998},"last":0.81999999999999995}]},{"osd":6,"last update":"Fri Jul 14 19:28:38 2023","interfaces":[{"interface":"back","average":{"1min":0.61299999999999999,"5min":0.501,"15min":0.48199999999999998},"min":{"1min":0.39100000000000001,"5min":0.34699999999999998,"15min":0.34699999999999998},"max":{"1min":0.82599999999999996,"5min":0.82599999999999996,"15min":0.82599999999999996},"last":0.58299999999999996},{"interface":"front","average":{"1min":0.51500000000000001,"5min":0.52000000000000002,"15min":0.52100000000000002},"min":{"1min":0.36499999999999999,"5min":0.36299999999999999,"15min":0.36299999999999999},"max":{"1min":0.67300000000000004,"5min":0.67300000000000004,"15min":0.67300000000000004},"last":0.44800000000000001}]},{"osd":7,"last update":"Fri Jul 14 19:28:57 2023","interfaces":[{"interface":"back","average":{"1min":0.69699999999999995,"5min":0.68000000000000005,"15min":0.67700000000000005},"min":{"1min":0.48999999999999999,"5min":0.41699999999999998,"15min":0.41699999999999998},"max":{"1min":0.96399999999999997,"5min":1.196,"15min":1.196},"last":0.73699999999999999},{"interface":"front","average":{"1min":0.78700000000000003,"5min":0.69199999999999995,"15min":0.67600000000000005},"min":{"1min":0.58499999999999996,"5min":0.46100000000000002,"15min":0.46100000000000002},"max":{"1min":1.0289999999999999,"5min":1.0289999999999999,"15min":1.0289999999999999},"last":0.85699999999999998}]},{"osd":8,"last update":"Fri Jul 14 19:28:17 2023","interfaces":[{"interface":"back","average":{"1min":0.69899999999999995,"5min":0.69899999999999995,"15min":0.69899999999999995},"min":{"1min":0.48199999999999998,"5min":0.48199999999999998,"15min":0.48199999999999998},"max":{"1min":0.98799999999999999,"5min":0.98799999999999999,"15min":0.98799999999999999},"last":0.751},{"interface":"front","average":{"1min":0.749,"5min":0.749,"15min":0.749},"min":{"1min":0.51800000000000002,"5min":0.51800000000000002,"15min":0.51800000000000002},"max":{"1min":0.92500000000000004,"5min":0.92500000000000004,"15min":0.92500000000000004},"last":0.68100000000000005}]},{"osd":9,"last update":"Fri Jul 14 19:28:38 2023","interfaces":[{"interface":"back","average":{"1min":0.76600000000000001,"5min":0.76600000000000001,"15min":0.76600000000000001},"min":{"1min":0.42099999999999999,"5min":0.42099999999999999,"15min":0.42099999999999999},"max":{"1min":1.0549999999999999,"5min":1.0549999999999999,"15min":1.0549999999999999},"last":0.79100000000000004},{"interface":"front","average":{"1min":0.64900000000000002,"5min":0.64900000000000002,"15min":0.64900000000000002},"min":{"1min":0.40799999999999997,"5min":0.40799999999999997,"15min":0.40799999999999997},"max":{"1min":0.95099999999999996,"5min":0.95099999999999996,"15min":0.95099999999999996},"last":0.49099999999999999}]},{"osd":10,"last update":"Fri Jul 14 19:29:02 2023","interfaces":[{"interface":"back","average":{"1min":0.72699999999999998,"5min":0.72699999999999998,"15min":0.72699999999999998},"min":{"1min":0.56499999999999995,"5min":0.56499999999999995,"15min":0.56499999999999995},"max":{"1min":0.92700000000000005,"5min":0.92700000000000005,"15min":0.92700000000000005},"last":0.83399999999999996},{"interface":"front","average":{"1min":0.82099999999999995,"5min":0.82099999999999995,"15min":0.82099999999999995},"min":{"1min":0.59399999999999997,"5min":0.59399999999999997,"15min":0.59399999999999997},"max":{"1min":1.0880000000000001,"5min":1.0880000000000001,"15min":1.0880000000000001},"last":0.89700000000000002}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.80800000000000005}]}]},{"osd":4,"up_from":32,"seq":137438953513,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27892,"kb_used_data":876,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717260,"statfs":{"total":95995035648,"available":95966474240,"internally_reserved":0,"allocated":897024,"data_stored":428479,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":27654411},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jul 14 19:28:54 2023","interfaces":[{"interface":"back","average":{"1min":0.57599999999999996,"5min":0.52300000000000002,"15min":0.50900000000000001},"min":{"1min":0.39200000000000002,"5min":0.27400000000000002,"15min":0.27400000000000002},"max":{"1min":0.85299999999999998,"5min":0.85299999999999998,"15min":0.85299999999999998},"last":0.496},{"interface":"front","average":{"1min":0.56799999999999995,"5min":0.51700000000000002,"15min":0.497},"min":{"1min":0.35599999999999998,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":0.77600000000000002,"5min":0.90800000000000003,"15min":0.90800000000000003},"last":0.40699999999999997}]},{"osd":1,"last update":"Fri Jul 14 19:28:54 2023","interfaces":[{"interface":"back","average":{"1min":0.42799999999999999,"5min":0.378,"15min":0.35899999999999999},"min":{"1min":0.28699999999999998,"5min":0.224,"15min":0.224},"max":{"1min":0.77200000000000002,"5min":0.77200000000000002,"15min":0.77200000000000002},"last":0.40400000000000003},{"interface":"front","average":{"1min":0.48699999999999999,"5min":0.40500000000000003,"15min":0.38700000000000001},"min":{"1min":0.33800000000000002,"5min":0.247,"15min":0.247},"max":{"1min":0.72999999999999998,"5min":0.72999999999999998,"15min":0.72999999999999998},"last":0.29799999999999999}]},{"osd":2,"last update":"Fri Jul 14 19:28:54 2023","interfaces":[{"interface":"back","average":{"1min":0.61099999999999999,"5min":0.59799999999999998,"15min":0.59599999999999997},"min":{"1min":0.373,"5min":0.32800000000000001,"15min":0.32800000000000001},"max":{"1min":0.83799999999999997,"5min":0.86399999999999999,"15min":0.86399999999999999},"last":0.63800000000000001},{"interface":"front","average":{"1min":0.65200000000000002,"5min":0.59699999999999998,"15min":0.58299999999999996},"min":{"1min":0.43099999999999999,"5min":0.43099999999999999,"15min":0.43099999999999999},"max":{"1min":0.97099999999999997,"5min":0.97099999999999997,"15min":0.97099999999999997},"last":0.65700000000000003}]},{"osd":3,"last update":"Fri Jul 14 19:28:54 2023","interfaces":[{"interface":"back","average":{"1min":0.61299999999999999,"5min":0.55700000000000005,"15min":0.54400000000000004},"min":{"1min":0.442,"5min":0.36599999999999999,"15min":0.36599999999999999},"max":{"1min":0.98599999999999999,"5min":0.98599999999999999,"15min":0.98599999999999999},"last":0.52400000000000002},{"interface":"front","average":{"1min":0.59399999999999997,"5min":0.54600000000000004,"15min":0.53000000000000003},"min":{"1min":0.36399999999999999,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.876,"5min":0.876,"15min":0.876},"last":0.36699999999999999}]},{"osd":5,"last update":"Fri Jul 14 19:28:12 2023","interfaces":[{"interface":"back","average":{"1min":0.625,"5min":0.59699999999999998,"15min":0.59299999999999997},"min":{"1min":0.40000000000000002,"5min":0.34699999999999998,"15min":0.34699999999999998},"max":{"1min":0.81200000000000006,"5min":0.88200000000000001,"15min":0.88200000000000001},"last":0.73699999999999999},{"interface":"front","average":{"1min":0.61599999999999999,"5min":0.58899999999999997,"15min":0.58499999999999996},"min":{"1min":0.44500000000000001,"5min":0.314,"15min":0.314},"max":{"1min":0.749,"5min":0.97099999999999997,"15min":0.97099999999999997},"last":0.621}]},{"osd":6,"last update":"Fri Jul 14 19:28:36 2023","interfaces":[{"interface":"back","average":{"1min":0.61199999999999999,"5min":0.59199999999999997,"15min":0.58799999999999997},"min":{"1min":0.378,"5min":0.378,"15min":0.378},"max":{"1min":0.79900000000000004,"5min":0.93000000000000005,"15min":0.93000000000000005},"last":0.433},{"interface":"front","average":{"1min":0.66000000000000003,"5min":0.627,"15min":0.621},"min":{"1min":0.42699999999999999,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":0.89500000000000002,"5min":0.89500000000000002,"15min":0.89500000000000002},"last":0.60299999999999998}]},{"osd":7,"last update":"Fri Jul 14 19:28:59 2023","interfaces":[{"interface":"back","average":{"1min":0.61199999999999999,"5min":0.58299999999999996,"15min":0.57799999999999996},"min":{"1min":0.373,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":0.86499999999999999,"5min":0.89800000000000002,"15min":0.89800000000000002},"last":0.54200000000000004},{"interface":"front","average":{"1min":0.59199999999999997,"5min":0.55600000000000005,"15min":0.55000000000000004},"min":{"1min":0.38400000000000001,"5min":0.36199999999999999,"15min":0.36199999999999999},"max":{"1min":0.96899999999999997,"5min":0.96899999999999997,"15min":0.96899999999999997},"last":0.47199999999999998}]},{"osd":8,"last update":"Fri Jul 14 19:28:18 2023","interfaces":[{"interface":"back","average":{"1min":0.66500000000000004,"5min":0.66500000000000004,"15min":0.66500000000000004},"min":{"1min":0.52500000000000002,"5min":0.52500000000000002,"15min":0.52500000000000002},"max":{"1min":0.84499999999999997,"5min":0.84499999999999997,"15min":0.84499999999999997},"last":0.51000000000000001},{"interface":"front","average":{"1min":0.70899999999999996,"5min":0.70899999999999996,"15min":0.70899999999999996},"min":{"1min":0.49099999999999999,"5min":0.49099999999999999,"15min":0.49099999999999999},"max":{"1min":0.95899999999999996,"5min":0.95899999999999996,"15min":0.95899999999999996},"last":0.70499999999999996}]},{"osd":9,"last update":"Fri Jul 14 19:28:37 2023","interfaces":[{"interface":"back","average":{"1min":0.69799999999999995,"5min":0.69799999999999995,"15min":0.69799999999999995},"min":{"1min":0.49299999999999999,"5min":0.49299999999999999,"15min":0.49299999999999999},"max":{"1min":0.878,"5min":0.878,"15min":0.878},"last":0.45200000000000001},{"interface":"front","average":{"1min":0.74199999999999999,"5min":0.74199999999999999,"15min":0.74199999999999999},"min":{"1min":0.52000000000000002,"5min":0.52000000000000002,"15min":0.52000000000000002},"max":{"1min":1.04,"5min":1.04,"15min":1.04},"last":0.76200000000000001}]},{"osd":10,"last update":"Fri Jul 14 19:28:59 2023","interfaces":[{"interface":"back","average":{"1min":0.68300000000000005,"5min":0.68300000000000005,"15min":0.68300000000000005},"min":{"1min":0.47299999999999998,"5min":0.47299999999999998,"15min":0.47299999999999998},"max":{"1min":0.95299999999999996,"5min":0.95299999999999996,"15min":0.95299999999999996},"last":0.58399999999999996},{"interface":"front","average":{"1min":0.68000000000000005,"5min":0.68000000000000005,"15min":0.68000000000000005},"min":{"1min":0.443,"5min":0.443,"15min":0.443},"max":{"1min":0.995,"5min":0.995,"15min":0.995},"last":0.56399999999999995}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.67300000000000004}]}]},{"osd":5,"up_from":37,"seq":158913789989,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28496,"kb_used_data":1480,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716656,"statfs":{"total":95995035648,"available":95965855744,"internally_reserved":0,"allocated":1515520,"data_stored":1020643,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":27654411},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jul 14 19:28:13 2023","interfaces":[{"interface":"back","average":{"1min":0.54100000000000004,"5min":0.47499999999999998,"15min":0.46400000000000002},"min":{"1min":0.34899999999999998,"5min":0.28499999999999998,"15min":0.28499999999999998},"max":{"1min":0.79300000000000004,"5min":0.86399999999999999,"15min":0.86399999999999999},"last":0.66900000000000004},{"interface":"front","average":{"1min":0.55600000000000005,"5min":0.51100000000000001,"15min":0.503},"min":{"1min":0.35699999999999998,"5min":0.32800000000000001,"15min":0.32800000000000001},"max":{"1min":0.70299999999999996,"5min":0.84099999999999997,"15min":0.84099999999999997},"last":0.46899999999999997}]},{"osd":1,"last update":"Fri Jul 14 19:28:13 2023","interfaces":[{"interface":"back","average":{"1min":0.56799999999999995,"5min":0.54100000000000004,"15min":0.53700000000000003},"min":{"1min":0.38900000000000001,"5min":0.38500000000000001,"15min":0.38500000000000001},"max":{"1min":1.0620000000000001,"5min":1.0620000000000001,"15min":1.0620000000000001},"last":0.72599999999999998},{"interface":"front","average":{"1min":0.57099999999999995,"5min":0.55800000000000005,"15min":0.55600000000000005},"min":{"1min":0.40799999999999997,"5min":0.36099999999999999,"15min":0.36099999999999999},"max":{"1min":0.93899999999999995,"5min":0.93899999999999995,"15min":0.93899999999999995},"last":0.74299999999999999}]},{"osd":2,"last update":"Fri Jul 14 19:28:13 2023","interfaces":[{"interface":"back","average":{"1min":0.45400000000000001,"5min":0.442,"15min":0.44},"min":{"1min":0.307,"5min":0.217,"15min":0.217},"max":{"1min":0.67400000000000004,"5min":0.67400000000000004,"15min":0.67400000000000004},"last":0.50600000000000001},{"interface":"front","average":{"1min":0.48999999999999999,"5min":0.45500000000000002,"15min":0.44900000000000001},"min":{"1min":0.313,"5min":0.23699999999999999,"15min":0.23699999999999999},"max":{"1min":0.82499999999999996,"5min":0.82499999999999996,"15min":0.82499999999999996},"last":0.55200000000000005}]},{"osd":3,"last update":"Fri Jul 14 19:28:13 2023","interfaces":[{"interface":"back","average":{"1min":0.57499999999999996,"5min":0.57099999999999995,"15min":0.56999999999999995},"min":{"1min":0.36499999999999999,"5min":0.32200000000000001,"15min":0.32200000000000001},"max":{"1min":0.76400000000000001,"5min":0.90800000000000003,"15min":0.90800000000000003},"last":0.59299999999999997},{"interface":"front","average":{"1min":0.60899999999999999,"5min":0.57699999999999996,"15min":0.57199999999999995},"min":{"1min":0.41099999999999998,"5min":0.33900000000000002,"15min":0.33900000000000002},"max":{"1min":0.85499999999999998,"5min":0.94399999999999995,"15min":0.94399999999999995},"last":0.626}]},{"osd":4,"last update":"Fri Jul 14 19:28:13 2023","interfaces":[{"interface":"back","average":{"1min":0.627,"5min":0.60299999999999998,"15min":0.59899999999999998},"min":{"1min":0.45800000000000002,"5min":0.45800000000000002,"15min":0.45800000000000002},"max":{"1min":0.86899999999999999,"5min":0.86899999999999999,"15min":0.86899999999999999},"last":0.79700000000000004},{"interface":"front","average":{"1min":0.57699999999999996,"5min":0.59899999999999998,"15min":0.60299999999999998},"min":{"1min":0.38,"5min":0.38,"15min":0.38},"max":{"1min":0.745,"5min":0.79500000000000004,"15min":0.79500000000000004},"last":0.65200000000000002}]},{"osd":6,"last update":"Fri Jul 14 19:28:38 2023","interfaces":[{"interface":"back","average":{"1min":0.64900000000000002,"5min":0.60499999999999998,"15min":0.59699999999999998},"min":{"1min":0.443,"5min":0.38300000000000001,"15min":0.38300000000000001},"max":{"1min":0.85799999999999998,"5min":0.85799999999999998,"15min":0.85799999999999998},"last":0.77500000000000002},{"interface":"front","average":{"1min":0.70299999999999996,"5min":0.64500000000000002,"15min":0.63500000000000001},"min":{"1min":0.39900000000000002,"5min":0.36099999999999999,"15min":0.36099999999999999},"max":{"1min":1.0189999999999999,"5min":1.0189999999999999,"15min":1.0189999999999999},"last":0.81000000000000005}]},{"osd":7,"last update":"Fri Jul 14 19:28:59 2023","interfaces":[{"interface":"back","average":{"1min":0.65600000000000003,"5min":0.66400000000000003,"15min":0.66600000000000004},"min":{"1min":0.49099999999999999,"5min":0.49099999999999999,"15min":0.49099999999999999},"max":{"1min":0.88,"5min":0.95399999999999996,"15min":0.95399999999999996},"last":0.82599999999999996},{"interface":"front","average":{"1min":0.66800000000000004,"5min":0.68500000000000005,"15min":0.68799999999999994},"min":{"1min":0.46999999999999997,"5min":0.45800000000000002,"15min":0.45800000000000002},"max":{"1min":0.89500000000000002,"5min":0.96099999999999997,"15min":0.96099999999999997},"last":0.84099999999999997}]},{"osd":8,"last update":"Fri Jul 14 19:28:18 2023","interfaces":[{"interface":"back","average":{"1min":0.65200000000000002,"5min":0.65200000000000002,"15min":0.65200000000000002},"min":{"1min":0.41099999999999998,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":0.997,"5min":0.997,"15min":0.997},"last":0.85599999999999998},{"interface":"front","average":{"1min":0.60599999999999998,"5min":0.60599999999999998,"15min":0.60599999999999998},"min":{"1min":0.38600000000000001,"5min":0.38600000000000001,"15min":0.38600000000000001},"max":{"1min":0.80000000000000004,"5min":0.80000000000000004,"15min":0.80000000000000004},"last":0.68600000000000005}]},{"osd":9,"last update":"Fri Jul 14 19:28:38 2023","interfaces":[{"interface":"back","average":{"1min":0.70299999999999996,"5min":0.70299999999999996,"15min":0.70299999999999996},"min":{"1min":0.50600000000000001,"5min":0.50600000000000001,"15min":0.50600000000000001},"max":{"1min":0.88700000000000001,"5min":0.88700000000000001,"15min":0.88700000000000001},"last":0.76200000000000001},{"interface":"front","average":{"1min":0.77800000000000002,"5min":0.77800000000000002,"15min":0.77800000000000002},"min":{"1min":0.504,"5min":0.504,"15min":0.504},"max":{"1min":1.111,"5min":1.111,"15min":1.111},"last":0.877}]},{"osd":10,"last update":"Fri Jul 14 19:28:59 2023","interfaces":[{"interface":"back","average":{"1min":0.71699999999999997,"5min":0.71699999999999997,"15min":0.71699999999999997},"min":{"1min":0.46899999999999997,"5min":0.46899999999999997,"15min":0.46899999999999997},"max":{"1min":0.96599999999999997,"5min":0.96599999999999997,"15min":0.96599999999999997},"last":0.89000000000000001},{"interface":"front","average":{"1min":0.73199999999999998,"5min":0.73199999999999998,"15min":0.73199999999999998},"min":{"1min":0.54000000000000004,"5min":0.54000000000000004,"15min":0.54000000000000004},"max":{"1min":0.95099999999999996,"5min":0.95099999999999996,"15min":0.95099999999999996},"last":0.90100000000000002}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1478,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1564,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1598,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":596,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1818,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1156,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":738,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1758,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-07-14T19:29:09.312 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2023-07-14T19:29:09.313 INFO:tasks.cephadm:Setting up client nodes... 2023-07-14T19:29:09.313 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81c9387d2e7576da065747ec9c16d1c7d69016ec shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 9adea648-227b-11ee-9b33-001a4aab830c -- ceph auth get-or-create client.0 mon 'allow *' osd 'allow *' mds 'allow *' mgr 'allow *' 2023-07-14T19:29:09.890 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/9adea648-227b-11ee-9b33-001a4aab830c/mon.a/config 2023-07-14T19:29:09.941 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:29:09 smithi101 ceph-mon[128807]: from='client.14784 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-07-14T19:29:09.970 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:29:09 smithi139 ceph-mon[129293]: from='client.14784 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-07-14T19:29:10.031 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:29:09 smithi098 ceph-mon[127486]: from='client.14784 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-07-14T19:29:10.941 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:29:10 smithi101 ceph-mon[128807]: from='client.14790 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-07-14T19:29:10.941 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:29:10 smithi101 ceph-mon[128807]: pgmap v257: 97 pgs: 97 active+clean; 616 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 5.9 KiB/s rd, 8.9 KiB/s wr, 17 op/s 2023-07-14T19:29:10.959 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:29:10 smithi098 ceph-mon[127486]: from='client.14790 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-07-14T19:29:10.959 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:29:10 smithi098 ceph-mon[127486]: pgmap v257: 97 pgs: 97 active+clean; 616 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 5.9 KiB/s rd, 8.9 KiB/s wr, 17 op/s 2023-07-14T19:29:10.970 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:29:10 smithi139 ceph-mon[129293]: from='client.14790 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-07-14T19:29:10.971 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:29:10 smithi139 ceph-mon[129293]: pgmap v257: 97 pgs: 97 active+clean; 616 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 5.9 KiB/s rd, 8.9 KiB/s wr, 17 op/s 2023-07-14T19:29:11.109 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:11.107+0000 7f1d46f23700 1 -- 172.21.15.98:0/2752847689 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1d400ffde0 msgr2=0x7f1d40100230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:29:11.109 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:11.107+0000 7f1d46f23700 1 --2- 172.21.15.98:0/2752847689 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1d400ffde0 0x7f1d40100230 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f1d2c009990 tx=0x7f1d2c009ca0 comp rx=0 tx=0).stop 2023-07-14T19:29:11.109 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:11.107+0000 7f1d46f23700 1 -- 172.21.15.98:0/2752847689 shutdown_connections 2023-07-14T19:29:11.109 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:11.107+0000 7f1d46f23700 1 --2- 172.21.15.98:0/2752847689 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f1d400febe0 0x7f1d400feff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:29:11.109 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:11.107+0000 7f1d46f23700 1 --2- 172.21.15.98:0/2752847689 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f1d40100770 0x7f1d40107030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:29:11.110 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:11.107+0000 7f1d46f23700 1 --2- 172.21.15.98:0/2752847689 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1d400ffde0 0x7f1d40100230 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:29:11.110 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:11.107+0000 7f1d46f23700 1 -- 172.21.15.98:0/2752847689 >> 172.21.15.98:0/2752847689 conn(0x7f1d400fa210 msgr2=0x7f1d400fc660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:29:11.110 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:11.107+0000 7f1d46f23700 1 -- 172.21.15.98:0/2752847689 shutdown_connections 2023-07-14T19:29:11.110 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:11.107+0000 7f1d46f23700 1 -- 172.21.15.98:0/2752847689 wait complete. 2023-07-14T19:29:11.112 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:11.107+0000 7f1d46f23700 1 Processor -- start 2023-07-14T19:29:11.112 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:11.108+0000 7f1d46f23700 1 -- start start 2023-07-14T19:29:11.112 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:11.108+0000 7f1d46f23700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1d400febe0 0x7f1d40068c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:29:11.112 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:11.108+0000 7f1d46f23700 1 --2- >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f1d400ffde0 0x7f1d4019a7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:29:11.112 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:11.108+0000 7f1d46f23700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f1d40100770 0x7f1d4019cdf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:29:11.112 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:11.108+0000 7f1d46f23700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1d4019d590 con 0x7f1d400febe0 2023-07-14T19:29:11.113 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:11.108+0000 7f1d46f23700 1 -- --> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1d4019d700 con 0x7f1d400ffde0 2023-07-14T19:29:11.113 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:11.108+0000 7f1d46f23700 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1d4019d8c0 con 0x7f1d40100770 2023-07-14T19:29:11.113 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:11.108+0000 7f1d44cbf700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1d400febe0 0x7f1d40068c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:29:11.113 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:11.108+0000 7f1d44cbf700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1d400febe0 0x7f1d40068c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:45878/0 (socket says 172.21.15.98:45878) 2023-07-14T19:29:11.113 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:11.108+0000 7f1d44cbf700 1 -- 172.21.15.98:0/1479700177 learned_addr learned my addr 172.21.15.98:0/1479700177 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-07-14T19:29:11.113 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:11.108+0000 7f1d3ffff700 1 --2- 172.21.15.98:0/1479700177 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f1d400ffde0 0x7f1d4019a7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:29:11.113 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:11.108+0000 7f1d454c0700 1 --2- 172.21.15.98:0/1479700177 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f1d40100770 0x7f1d4019cdf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:29:11.114 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:11.108+0000 7f1d44cbf700 1 -- 172.21.15.98:0/1479700177 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f1d40100770 msgr2=0x7f1d4019cdf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:29:11.114 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:11.108+0000 7f1d44cbf700 1 --2- 172.21.15.98:0/1479700177 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f1d40100770 0x7f1d4019cdf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:29:11.114 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:11.108+0000 7f1d44cbf700 1 -- 172.21.15.98:0/1479700177 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f1d400ffde0 msgr2=0x7f1d4019a7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:29:11.114 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:11.108+0000 7f1d44cbf700 1 --2- 172.21.15.98:0/1479700177 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f1d400ffde0 0x7f1d4019a7b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:29:11.114 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:11.108+0000 7f1d44cbf700 1 -- 172.21.15.98:0/1479700177 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1d2c0095f0 con 0x7f1d400febe0 2023-07-14T19:29:11.114 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:11.108+0000 7f1d454c0700 1 --2- 172.21.15.98:0/1479700177 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f1d40100770 0x7f1d4019cdf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-07-14T19:29:11.114 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:11.108+0000 7f1d44cbf700 1 --2- 172.21.15.98:0/1479700177 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1d400febe0 0x7f1d40068c60 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f1d3400b500 tx=0x7f1d3400b810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:29:11.115 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:11.108+0000 7f1d3dffb700 1 -- 172.21.15.98:0/1479700177 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1d34011830 con 0x7f1d400febe0 2023-07-14T19:29:11.115 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:11.109+0000 7f1d46f23700 1 -- 172.21.15.98:0/1479700177 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1d4019dba0 con 0x7f1d400febe0 2023-07-14T19:29:11.115 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:11.109+0000 7f1d3dffb700 1 -- 172.21.15.98:0/1479700177 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f1d34011e50 con 0x7f1d400febe0 2023-07-14T19:29:11.115 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:11.109+0000 7f1d3dffb700 1 -- 172.21.15.98:0/1479700177 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1d3400f2e0 con 0x7f1d400febe0 2023-07-14T19:29:11.115 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:11.109+0000 7f1d46f23700 1 -- 172.21.15.98:0/1479700177 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1d4019e0a0 con 0x7f1d400febe0 2023-07-14T19:29:11.115 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:11.110+0000 7f1d46f23700 1 -- 172.21.15.98:0/1479700177 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1d40105b70 con 0x7f1d400febe0 2023-07-14T19:29:11.115 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:11.110+0000 7f1d3dffb700 1 -- 172.21.15.98:0/1479700177 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 17) v1 ==== 135766+0+0 (secure 0 0 0) 0x7f1d34010580 con 0x7f1d400febe0 2023-07-14T19:29:11.116 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:11.111+0000 7f1d3dffb700 1 --2- 172.21.15.98:0/1479700177 >> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] conn(0x7f1d300a11e0 0x7f1d300a3690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:29:11.116 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:11.111+0000 7f1d3ffff700 1 --2- 172.21.15.98:0/1479700177 >> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] conn(0x7f1d300a11e0 0x7f1d300a3690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:29:11.116 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:11.111+0000 7f1d3dffb700 1 -- 172.21.15.98:0/1479700177 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 7979+0+0 (secure 0 0 0) 0x7f1d340cc220 con 0x7f1d400febe0 2023-07-14T19:29:11.117 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:11.111+0000 7f1d3ffff700 1 --2- 172.21.15.98:0/1479700177 >> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] conn(0x7f1d300a11e0 0x7f1d300a3690 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f1d400691e0 tx=0x7f1d2c005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:29:11.117 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:11.115+0000 7f1d3dffb700 1 -- 172.21.15.98:0/1479700177 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f1d34095ad0 con 0x7f1d400febe0 2023-07-14T19:29:11.383 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:11.382+0000 7f1d46f23700 1 -- 172.21.15.98:0/1479700177 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]} v 0) v1 -- 0x7f1d4002c8e0 con 0x7f1d400febe0 2023-07-14T19:29:11.387 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:11.386+0000 7f1d3dffb700 1 -- 172.21.15.98:0/1479700177 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]=0 v31) v1 ==== 170+0+59 (secure 0 0 0) 0x7f1d340990f0 con 0x7f1d400febe0 2023-07-14T19:29:11.388 INFO:teuthology.orchestra.run.smithi098.stdout:[client.0] 2023-07-14T19:29:11.388 INFO:teuthology.orchestra.run.smithi098.stdout: key = AQAHorFk49fkFhAACDuWWTY+CZI+oyp15NZR1A== 2023-07-14T19:29:11.390 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:11.388+0000 7f1d46f23700 1 -- 172.21.15.98:0/1479700177 >> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] conn(0x7f1d300a11e0 msgr2=0x7f1d300a3690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:29:11.390 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:11.388+0000 7f1d46f23700 1 --2- 172.21.15.98:0/1479700177 >> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] conn(0x7f1d300a11e0 0x7f1d300a3690 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f1d400691e0 tx=0x7f1d2c005fd0 comp rx=0 tx=0).stop 2023-07-14T19:29:11.390 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:11.388+0000 7f1d46f23700 1 -- 172.21.15.98:0/1479700177 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1d400febe0 msgr2=0x7f1d40068c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:29:11.390 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:11.388+0000 7f1d46f23700 1 --2- 172.21.15.98:0/1479700177 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1d400febe0 0x7f1d40068c60 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f1d3400b500 tx=0x7f1d3400b810 comp rx=0 tx=0).stop 2023-07-14T19:29:11.391 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:11.388+0000 7f1d46f23700 1 -- 172.21.15.98:0/1479700177 shutdown_connections 2023-07-14T19:29:11.391 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:11.388+0000 7f1d46f23700 1 --2- 172.21.15.98:0/1479700177 >> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] conn(0x7f1d300a11e0 0x7f1d300a3690 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:29:11.391 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:11.388+0000 7f1d46f23700 1 --2- 172.21.15.98:0/1479700177 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f1d40100770 0x7f1d4019cdf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:29:11.391 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:11.388+0000 7f1d46f23700 1 --2- 172.21.15.98:0/1479700177 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1d400febe0 0x7f1d40068c60 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:29:11.391 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:11.388+0000 7f1d46f23700 1 --2- 172.21.15.98:0/1479700177 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f1d400ffde0 0x7f1d4019a7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:29:11.391 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:11.388+0000 7f1d46f23700 1 -- 172.21.15.98:0/1479700177 >> 172.21.15.98:0/1479700177 conn(0x7f1d400fa210 msgr2=0x7f1d400fc4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:29:11.391 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:11.389+0000 7f1d46f23700 1 -- 172.21.15.98:0/1479700177 shutdown_connections 2023-07-14T19:29:11.391 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:11.389+0000 7f1d46f23700 1 -- 172.21.15.98:0/1479700177 wait complete. 2023-07-14T19:29:11.749 DEBUG:teuthology.orchestra.run.smithi098:> set -ex 2023-07-14T19:29:11.750 DEBUG:teuthology.orchestra.run.smithi098:> sudo dd of=/etc/ceph/ceph.client.0.keyring 2023-07-14T19:29:11.750 DEBUG:teuthology.orchestra.run.smithi098:> sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2023-07-14T19:29:11.791 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2023-07-14T19:29:11.792 INFO:tasks.cephadm.ceph_manager.ceph:waiting for mgr available 2023-07-14T19:29:11.793 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81c9387d2e7576da065747ec9c16d1c7d69016ec shell --fsid 9adea648-227b-11ee-9b33-001a4aab830c -- ceph mgr dump --format=json 2023-07-14T19:29:12.380 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/9adea648-227b-11ee-9b33-001a4aab830c/mon.a/config 2023-07-14T19:29:12.591 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:29:12 smithi098 ceph-mon[127486]: from='client.? 172.21.15.98:0/1479700177' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2023-07-14T19:29:12.591 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:29:12 smithi098 ceph-mon[127486]: from='client.? 172.21.15.98:0/1479700177' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2023-07-14T19:29:12.592 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:29:12 smithi098 ceph-mon[127486]: pgmap v258: 97 pgs: 97 active+clean; 616 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 7.7 KiB/s wr, 18 op/s 2023-07-14T19:29:12.691 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:29:12 smithi101 ceph-mon[128807]: from='client.? 172.21.15.98:0/1479700177' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2023-07-14T19:29:12.691 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:29:12 smithi101 ceph-mon[128807]: from='client.? 172.21.15.98:0/1479700177' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2023-07-14T19:29:12.692 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:29:12 smithi101 ceph-mon[128807]: pgmap v258: 97 pgs: 97 active+clean; 616 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 7.7 KiB/s wr, 18 op/s 2023-07-14T19:29:12.721 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:29:12 smithi139 ceph-mon[129293]: from='client.? 172.21.15.98:0/1479700177' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2023-07-14T19:29:12.721 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:29:12 smithi139 ceph-mon[129293]: from='client.? 172.21.15.98:0/1479700177' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2023-07-14T19:29:12.721 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:29:12 smithi139 ceph-mon[129293]: pgmap v258: 97 pgs: 97 active+clean; 616 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 7.7 KiB/s wr, 18 op/s 2023-07-14T19:29:13.564 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:13.561+0000 7f83a337f700 1 -- 172.21.15.98:0/2233470501 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f839c104d10 msgr2=0x7f839c1050e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:29:13.565 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:13.561+0000 7f83a337f700 1 --2- 172.21.15.98:0/2233470501 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f839c104d10 0x7f839c1050e0 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f838c0099c0 tx=0x7f838c009cd0 comp rx=0 tx=0).stop 2023-07-14T19:29:13.565 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:13.561+0000 7f83a337f700 1 -- 172.21.15.98:0/2233470501 shutdown_connections 2023-07-14T19:29:13.565 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:13.561+0000 7f83a337f700 1 --2- 172.21.15.98:0/2233470501 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f839c0ffbe0 0x7f839c10ba50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:29:13.565 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:13.561+0000 7f83a337f700 1 --2- 172.21.15.98:0/2233470501 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f839c104d10 0x7f839c1050e0 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:29:13.565 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:13.561+0000 7f83a337f700 1 --2- 172.21.15.98:0/2233470501 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f839c105620 0x7f839c0ff4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:29:13.566 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:13.561+0000 7f83a337f700 1 -- 172.21.15.98:0/2233470501 >> 172.21.15.98:0/2233470501 conn(0x7f839c076370 msgr2=0x7f839c076770 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:29:13.566 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:13.561+0000 7f83a337f700 1 -- 172.21.15.98:0/2233470501 shutdown_connections 2023-07-14T19:29:13.566 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:13.561+0000 7f83a337f700 1 -- 172.21.15.98:0/2233470501 wait complete. 2023-07-14T19:29:13.566 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:13.562+0000 7f83a337f700 1 Processor -- start 2023-07-14T19:29:13.566 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:13.562+0000 7f83a337f700 1 -- start start 2023-07-14T19:29:13.566 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:13.562+0000 7f83a337f700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f839c0ffbe0 0x7f839c195e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:29:13.566 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:13.562+0000 7f83a337f700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f839c104d10 0x7f839c1963a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:29:13.567 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:13.562+0000 7f83a337f700 1 --2- >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f839c19a810 0x7f839c19ac80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:29:13.567 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:13.562+0000 7f83a337f700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f839c19b1f0 con 0x7f839c104d10 2023-07-14T19:29:13.567 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:13.562+0000 7f83a337f700 1 -- --> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f839c19b360 con 0x7f839c19a810 2023-07-14T19:29:13.567 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:13.562+0000 7f83a337f700 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f839c19b520 con 0x7f839c0ffbe0 2023-07-14T19:29:13.567 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:13.562+0000 7f83a091a700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f839c104d10 0x7f839c1963a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:29:13.567 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:13.562+0000 7f83a111b700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f839c0ffbe0 0x7f839c195e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:29:13.567 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:13.562+0000 7f83a091a700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f839c104d10 0x7f839c1963a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:45894/0 (socket says 172.21.15.98:45894) 2023-07-14T19:29:13.568 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:13.562+0000 7f83a091a700 1 -- 172.21.15.98:0/21467624 learned_addr learned my addr 172.21.15.98:0/21467624 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-07-14T19:29:13.568 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:13.563+0000 7f83a191c700 1 --2- 172.21.15.98:0/21467624 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f839c19a810 0x7f839c19ac80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:29:13.568 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:13.563+0000 7f83a091a700 1 -- 172.21.15.98:0/21467624 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f839c0ffbe0 msgr2=0x7f839c195e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:29:13.568 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:13.563+0000 7f83a091a700 1 --2- 172.21.15.98:0/21467624 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f839c0ffbe0 0x7f839c195e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:29:13.568 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:13.563+0000 7f83a091a700 1 -- 172.21.15.98:0/21467624 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f839c19a810 msgr2=0x7f839c19ac80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:29:13.568 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:13.563+0000 7f83a091a700 1 --2- 172.21.15.98:0/21467624 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f839c19a810 0x7f839c19ac80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:29:13.568 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:13.563+0000 7f83a091a700 1 -- 172.21.15.98:0/21467624 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f838c0095f0 con 0x7f839c104d10 2023-07-14T19:29:13.568 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:13.563+0000 7f83a091a700 1 --2- 172.21.15.98:0/21467624 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f839c104d10 0x7f839c1963a0 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f839800d980 tx=0x7f839800dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:29:13.569 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:13.563+0000 7f83927fc700 1 -- 172.21.15.98:0/21467624 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f839800f890 con 0x7f839c104d10 2023-07-14T19:29:13.570 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:13.563+0000 7f83a337f700 1 -- 172.21.15.98:0/21467624 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f839c1a2ac0 con 0x7f839c104d10 2023-07-14T19:29:13.570 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:13.563+0000 7f83927fc700 1 -- 172.21.15.98:0/21467624 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f8398010460 con 0x7f839c104d10 2023-07-14T19:29:13.570 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:13.563+0000 7f83927fc700 1 -- 172.21.15.98:0/21467624 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f839800e610 con 0x7f839c104d10 2023-07-14T19:29:13.570 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:13.563+0000 7f83a337f700 1 -- 172.21.15.98:0/21467624 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f839c1a3010 con 0x7f839c104d10 2023-07-14T19:29:13.570 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:13.564+0000 7f83a337f700 1 -- 172.21.15.98:0/21467624 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f839c04e340 con 0x7f839c104d10 2023-07-14T19:29:13.571 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:13.565+0000 7f83927fc700 1 -- 172.21.15.98:0/21467624 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 17) v1 ==== 135766+0+0 (secure 0 0 0) 0x7f83980105d0 con 0x7f839c104d10 2023-07-14T19:29:13.571 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:13.566+0000 7f83927fc700 1 --2- 172.21.15.98:0/21467624 >> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] conn(0x7f83880a11e0 0x7f83880a3690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:29:13.571 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:13.566+0000 7f83a111b700 1 --2- 172.21.15.98:0/21467624 >> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] conn(0x7f83880a11e0 0x7f83880a3690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:29:13.571 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:13.566+0000 7f83927fc700 1 -- 172.21.15.98:0/21467624 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 7979+0+0 (secure 0 0 0) 0x7f83980cc350 con 0x7f839c104d10 2023-07-14T19:29:13.571 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:13.566+0000 7f83a111b700 1 --2- 172.21.15.98:0/21467624 >> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] conn(0x7f83880a11e0 0x7f83880a3690 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f838c000c00 tx=0x7f838c003680 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:29:13.572 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:13.570+0000 7f83927fc700 1 -- 172.21.15.98:0/21467624 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f8398095af0 con 0x7f839c104d10 2023-07-14T19:29:13.813 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:13.811+0000 7f83a337f700 1 -- 172.21.15.98:0/21467624 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "mgr dump", "format": "json"} v 0) v1 -- 0x7f839c1057a0 con 0x7f839c104d10 2023-07-14T19:29:13.818 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:13.816+0000 7f83927fc700 1 -- 172.21.15.98:0/21467624 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "mgr dump", "format": "json"}]=0 v17) v1 ==== 74+0+260122 (secure 0 0 0) 0x7f8398099110 con 0x7f839c104d10 2023-07-14T19:29:13.821 INFO:teuthology.orchestra.run.smithi098.stdout: 2023-07-14T19:29:13.822 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:13.819+0000 7f83a337f700 1 -- 172.21.15.98:0/21467624 >> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] conn(0x7f83880a11e0 msgr2=0x7f83880a3690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:29:13.822 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:13.819+0000 7f83a337f700 1 --2- 172.21.15.98:0/21467624 >> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] conn(0x7f83880a11e0 0x7f83880a3690 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f838c000c00 tx=0x7f838c003680 comp rx=0 tx=0).stop 2023-07-14T19:29:13.822 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:13.819+0000 7f83a337f700 1 -- 172.21.15.98:0/21467624 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f839c104d10 msgr2=0x7f839c1963a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:29:13.823 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:13.819+0000 7f83a337f700 1 --2- 172.21.15.98:0/21467624 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f839c104d10 0x7f839c1963a0 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f839800d980 tx=0x7f839800dc90 comp rx=0 tx=0).stop 2023-07-14T19:29:13.823 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:13.819+0000 7f83a337f700 1 -- 172.21.15.98:0/21467624 shutdown_connections 2023-07-14T19:29:13.823 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:13.819+0000 7f83a337f700 1 --2- 172.21.15.98:0/21467624 >> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] conn(0x7f83880a11e0 0x7f83880a3690 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:29:13.823 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:13.819+0000 7f83a337f700 1 --2- 172.21.15.98:0/21467624 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f839c0ffbe0 0x7f839c195e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:29:13.823 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:13.819+0000 7f83a337f700 1 --2- 172.21.15.98:0/21467624 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f839c19a810 0x7f839c19ac80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:29:13.823 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:13.819+0000 7f83a337f700 1 --2- 172.21.15.98:0/21467624 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f839c104d10 0x7f839c1963a0 unknown :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:29:13.823 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:13.819+0000 7f83a337f700 1 -- 172.21.15.98:0/21467624 >> 172.21.15.98:0/21467624 conn(0x7f839c076370 msgr2=0x7f839c102b40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:29:13.824 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:13.819+0000 7f83a337f700 1 -- 172.21.15.98:0/21467624 shutdown_connections 2023-07-14T19:29:13.824 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:13.819+0000 7f83a337f700 1 -- 172.21.15.98:0/21467624 wait complete. 2023-07-14T19:29:14.232 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":17,"active_gid":14152,"active_name":"x","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6800","nonce":452042164},{"type":"v1","addr":"172.21.15.98:6801","nonce":452042164}]},"active_addr":"172.21.15.98:6801/452042164","active_change":"2023-07-14T19:23:01.520420+0000","active_mgr_features":4540138322906710015,"available":true,"standbys":[{"gid":24106,"name":"z","mgr_features":4540138322906710015,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"alertmanager_web_password":{"name":"alertmanager_web_password","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"Alertmanager web password","long_desc":"","tags":[],"see_also":[]},"alertmanager_web_user":{"name":"alertmanager_web_user","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"Alertmanager web user","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.7","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"secs","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"prometheus_web_password":{"name":"prometheus_web_password","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"Prometheus web password","long_desc":"","tags":[],"see_also":[]},"prometheus_web_user":{"name":"prometheus_web_user","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"Prometheus web user","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"noautoscale":{"name":"noautoscale","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"global autoscale flag","long_desc":"Option to turn on/off the autoscaler for all pools","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"drive_group_interval":{"name":"drive_group_interval","type":"float","level":"advanced","flags":0,"default_value":"300.0","min":"","max":"","enum_allowed":[],"desc":"interval in seconds between re-application of applied drive_groups","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]},{"gid":24116,"name":"y","mgr_features":4540138322906710015,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"alertmanager_web_password":{"name":"alertmanager_web_password","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"Alertmanager web password","long_desc":"","tags":[],"see_also":[]},"alertmanager_web_user":{"name":"alertmanager_web_user","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"Alertmanager web user","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.7","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"secs","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"prometheus_web_password":{"name":"prometheus_web_password","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"Prometheus web password","long_desc":"","tags":[],"see_also":[]},"prometheus_web_user":{"name":"prometheus_web_user","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"Prometheus web user","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"noautoscale":{"name":"noautoscale","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"global autoscale flag","long_desc":"Option to turn on/off the autoscaler for all pools","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"drive_group_interval":{"name":"drive_group_interval","type":"float","level":"advanced","flags":0,"default_value":"300.0","min":"","max":"","enum_allowed":[],"desc":"interval in seconds between re-application of applied drive_groups","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]}],"modules":["cephadm","dashboard","iostat","nfs","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"alertmanager_web_password":{"name":"alertmanager_web_password","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"Alertmanager web password","long_desc":"","tags":[],"see_also":[]},"alertmanager_web_user":{"name":"alertmanager_web_user","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"Alertmanager web user","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.7","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"secs","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"prometheus_web_password":{"name":"prometheus_web_password","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"Prometheus web password","long_desc":"","tags":[],"see_also":[]},"prometheus_web_user":{"name":"prometheus_web_user","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"Prometheus web user","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"noautoscale":{"name":"noautoscale","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"global autoscale flag","long_desc":"Option to turn on/off the autoscaler for all pools","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"drive_group_interval":{"name":"drive_group_interval","type":"float","level":"advanced","flags":0,"default_value":"300.0","min":"","max":"","enum_allowed":[],"desc":"interval in seconds between re-application of applied drive_groups","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{"dashboard":"https://172.21.15.98:8443/"},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"reef":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":3,"active_clients":[{"name":"devicehealth","addrvec":[{"type":"v2","addr":"172.21.15.98:0","nonce":3882996047}]},{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.98:0","nonce":2099188469}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.98:0","nonce":3733111864}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.98:0","nonce":3608774060}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.98:0","nonce":538926571}]}]} 2023-07-14T19:29:14.240 INFO:tasks.cephadm.ceph_manager.ceph:mgr available! 2023-07-14T19:29:14.240 INFO:tasks.cephadm.ceph_manager.ceph:waiting for all up 2023-07-14T19:29:14.241 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81c9387d2e7576da065747ec9c16d1c7d69016ec shell --fsid 9adea648-227b-11ee-9b33-001a4aab830c -- ceph osd dump --format=json 2023-07-14T19:29:14.792 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/9adea648-227b-11ee-9b33-001a4aab830c/mon.a/config 2023-07-14T19:29:14.941 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:29:14 smithi101 ceph-mon[128807]: pgmap v259: 97 pgs: 97 active+clean; 616 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 6.7 KiB/s wr, 16 op/s 2023-07-14T19:29:14.941 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:29:14 smithi101 ceph-mon[128807]: from='client.? 172.21.15.98:0/21467624' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2023-07-14T19:29:14.970 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:29:14 smithi139 ceph-mon[129293]: pgmap v259: 97 pgs: 97 active+clean; 616 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 6.7 KiB/s wr, 16 op/s 2023-07-14T19:29:14.970 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:29:14 smithi139 ceph-mon[129293]: from='client.? 172.21.15.98:0/21467624' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2023-07-14T19:29:14.983 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:29:14 smithi098 ceph-mon[127486]: pgmap v259: 97 pgs: 97 active+clean; 616 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 6.7 KiB/s wr, 16 op/s 2023-07-14T19:29:14.983 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:29:14 smithi098 ceph-mon[127486]: from='client.? 172.21.15.98:0/21467624' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2023-07-14T19:29:15.880 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:15.877+0000 7f8a906cc700 1 -- 172.21.15.98:0/2725052850 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8a880fe4b0 msgr2=0x7f8a880fe880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:29:15.880 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:15.877+0000 7f8a906cc700 1 --2- 172.21.15.98:0/2725052850 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8a880fe4b0 0x7f8a880fe880 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f8a780099c0 tx=0x7f8a78009cd0 comp rx=0 tx=0).stop 2023-07-14T19:29:15.881 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:15.877+0000 7f8a906cc700 1 -- 172.21.15.98:0/2725052850 shutdown_connections 2023-07-14T19:29:15.881 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:15.877+0000 7f8a906cc700 1 --2- 172.21.15.98:0/2725052850 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f8a88105010 0x7f8a8810cbc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:29:15.881 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:15.877+0000 7f8a906cc700 1 --2- 172.21.15.98:0/2725052850 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8a880fe4b0 0x7f8a880fe880 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:29:15.881 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:15.877+0000 7f8a906cc700 1 --2- 172.21.15.98:0/2725052850 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f8a880fee50 0x7f8a88104930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:29:15.881 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:15.877+0000 7f8a906cc700 1 -- 172.21.15.98:0/2725052850 >> 172.21.15.98:0/2725052850 conn(0x7f8a88075280 msgr2=0x7f8a88075680 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:29:15.881 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:15.877+0000 7f8a906cc700 1 -- 172.21.15.98:0/2725052850 shutdown_connections 2023-07-14T19:29:15.881 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:15.877+0000 7f8a906cc700 1 -- 172.21.15.98:0/2725052850 wait complete. 2023-07-14T19:29:15.882 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:15.878+0000 7f8a906cc700 1 Processor -- start 2023-07-14T19:29:15.882 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:15.878+0000 7f8a906cc700 1 -- start start 2023-07-14T19:29:15.882 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:15.878+0000 7f8a906cc700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8a880fe4b0 0x7f8a88195db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:29:15.882 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:15.878+0000 7f8a906cc700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f8a880fee50 0x7f8a881962f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:29:15.882 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:15.878+0000 7f8a906cc700 1 --2- >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f8a88105010 0x7f8a8819a760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:29:15.882 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:15.878+0000 7f8a906cc700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8a8819ad70 con 0x7f8a880fe4b0 2023-07-14T19:29:15.882 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:15.878+0000 7f8a906cc700 1 -- --> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8a8819aee0 con 0x7f8a88105010 2023-07-14T19:29:15.883 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:15.878+0000 7f8a906cc700 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8a8819b0a0 con 0x7f8a880fee50 2023-07-14T19:29:15.883 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:15.878+0000 7f8a8e468700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8a880fe4b0 0x7f8a88195db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:29:15.883 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:15.878+0000 7f8a8e468700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8a880fe4b0 0x7f8a88195db0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:45904/0 (socket says 172.21.15.98:45904) 2023-07-14T19:29:15.883 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:15.878+0000 7f8a8e468700 1 -- 172.21.15.98:0/2505448407 learned_addr learned my addr 172.21.15.98:0/2505448407 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-07-14T19:29:15.883 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:15.878+0000 7f8a8ec69700 1 --2- 172.21.15.98:0/2505448407 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f8a88105010 0x7f8a8819a760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:29:15.883 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:15.878+0000 7f8a8dc67700 1 --2- 172.21.15.98:0/2505448407 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f8a880fee50 0x7f8a881962f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:29:15.883 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:15.878+0000 7f8a8e468700 1 -- 172.21.15.98:0/2505448407 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f8a880fee50 msgr2=0x7f8a881962f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:29:15.883 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:15.878+0000 7f8a8e468700 1 --2- 172.21.15.98:0/2505448407 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f8a880fee50 0x7f8a881962f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:29:15.884 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:15.878+0000 7f8a8e468700 1 -- 172.21.15.98:0/2505448407 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f8a88105010 msgr2=0x7f8a8819a760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:29:15.884 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:15.878+0000 7f8a8e468700 1 --2- 172.21.15.98:0/2505448407 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f8a88105010 0x7f8a8819a760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:29:15.884 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:15.878+0000 7f8a8e468700 1 -- 172.21.15.98:0/2505448407 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8a780095f0 con 0x7f8a880fe4b0 2023-07-14T19:29:15.884 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:15.878+0000 7f8a8ec69700 1 --2- 172.21.15.98:0/2505448407 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f8a88105010 0x7f8a8819a760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-07-14T19:29:15.884 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:15.878+0000 7f8a8e468700 1 --2- 172.21.15.98:0/2505448407 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8a880fe4b0 0x7f8a88195db0 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f8a78004710 tx=0x7f8a780038c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:29:15.884 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:15.879+0000 7f8a7f7fe700 1 -- 172.21.15.98:0/2505448407 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8a7801d070 con 0x7f8a880fe4b0 2023-07-14T19:29:15.884 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:15.879+0000 7f8a7f7fe700 1 -- 172.21.15.98:0/2505448407 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f8a780041e0 con 0x7f8a880fe4b0 2023-07-14T19:29:15.885 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:15.879+0000 7f8a906cc700 1 -- 172.21.15.98:0/2505448407 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8a8819b320 con 0x7f8a880fe4b0 2023-07-14T19:29:15.885 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:15.879+0000 7f8a7f7fe700 1 -- 172.21.15.98:0/2505448407 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8a78021450 con 0x7f8a880fe4b0 2023-07-14T19:29:15.885 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:15.879+0000 7f8a906cc700 1 -- 172.21.15.98:0/2505448407 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8a881a2be0 con 0x7f8a880fe4b0 2023-07-14T19:29:15.886 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:15.880+0000 7f8a906cc700 1 -- 172.21.15.98:0/2505448407 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8a8804e340 con 0x7f8a880fe4b0 2023-07-14T19:29:15.886 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:15.881+0000 7f8a7f7fe700 1 -- 172.21.15.98:0/2505448407 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 17) v1 ==== 135766+0+0 (secure 0 0 0) 0x7f8a7800f460 con 0x7f8a880fe4b0 2023-07-14T19:29:15.886 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:15.881+0000 7f8a7f7fe700 1 --2- 172.21.15.98:0/2505448407 >> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] conn(0x7f8a740a11e0 0x7f8a740a3690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:29:15.886 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:15.881+0000 7f8a8dc67700 1 --2- 172.21.15.98:0/2505448407 >> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] conn(0x7f8a740a11e0 0x7f8a740a3690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:29:15.886 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:15.881+0000 7f8a7f7fe700 1 -- 172.21.15.98:0/2505448407 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 7979+0+0 (secure 0 0 0) 0x7f8a780cdac0 con 0x7f8a880fe4b0 2023-07-14T19:29:15.886 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:15.882+0000 7f8a8dc67700 1 --2- 172.21.15.98:0/2505448407 >> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] conn(0x7f8a740a11e0 0x7f8a740a3690 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f8a84006fd0 tx=0x7f8a84008040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:29:15.887 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:15.885+0000 7f8a7f7fe700 1 -- 172.21.15.98:0/2505448407 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f8a780972f0 con 0x7f8a880fe4b0 2023-07-14T19:29:16.073 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:16.071+0000 7f8a906cc700 1 -- 172.21.15.98:0/2505448407 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f8a881a2d70 con 0x7f8a880fe4b0 2023-07-14T19:29:16.074 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:16.073+0000 7f8a7f7fe700 1 -- 172.21.15.98:0/2505448407 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v83) v1 ==== 74+0+22117 (secure 0 0 0) 0x7f8a7809a910 con 0x7f8a880fe4b0 2023-07-14T19:29:16.075 INFO:teuthology.orchestra.run.smithi098.stdout: 2023-07-14T19:29:16.075 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":83,"fsid":"9adea648-227b-11ee-9b33-001a4aab830c","created":"2023-07-14T19:22:07.178854+0000","modified":"2023-07-14T19:29:02.293099+0000","last_up_change":"2023-07-14T19:28:19.646827+0000","last_in_change":"2023-07-14T19:28:02.018325+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-07-14T19:25:09.845827+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2023-07-14T19:28:58.952921+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"82","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2023-07-14T19:29:00.228272+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"83","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"a3797e89-1754-41cf-9517-227e06ffdd72","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6802","nonce":120507228},{"type":"v1","addr":"172.21.15.98:6803","nonce":120507228}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6804","nonce":120507228},{"type":"v1","addr":"172.21.15.98:6805","nonce":120507228}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6808","nonce":120507228},{"type":"v1","addr":"172.21.15.98:6809","nonce":120507228}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6806","nonce":120507228},{"type":"v1","addr":"172.21.15.98:6807","nonce":120507228}]},"public_addr":"172.21.15.98:6803/120507228","cluster_addr":"172.21.15.98:6805/120507228","heartbeat_back_addr":"172.21.15.98:6809/120507228","heartbeat_front_addr":"172.21.15.98:6807/120507228","state":["exists","up"]},{"osd":1,"uuid":"2b001add-1aaa-4496-a403-7ef9725b80a0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.101:6800","nonce":2450584584},{"type":"v1","addr":"172.21.15.101:6801","nonce":2450584584}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.101:6802","nonce":2450584584},{"type":"v1","addr":"172.21.15.101:6803","nonce":2450584584}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.101:6806","nonce":2450584584},{"type":"v1","addr":"172.21.15.101:6807","nonce":2450584584}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.101:6804","nonce":2450584584},{"type":"v1","addr":"172.21.15.101:6805","nonce":2450584584}]},"public_addr":"172.21.15.101:6801/2450584584","cluster_addr":"172.21.15.101:6803/2450584584","heartbeat_back_addr":"172.21.15.101:6807/2450584584","heartbeat_front_addr":"172.21.15.101:6805/2450584584","state":["exists","up"]},{"osd":2,"uuid":"48e6d499-3ca9-424b-b0fb-3ba5f33b9fc9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":945038995},{"type":"v1","addr":"172.21.15.139:6801","nonce":945038995}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":945038995},{"type":"v1","addr":"172.21.15.139:6803","nonce":945038995}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":945038995},{"type":"v1","addr":"172.21.15.139:6807","nonce":945038995}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":945038995},{"type":"v1","addr":"172.21.15.139:6805","nonce":945038995}]},"public_addr":"172.21.15.139:6801/945038995","cluster_addr":"172.21.15.139:6803/945038995","heartbeat_back_addr":"172.21.15.139:6807/945038995","heartbeat_front_addr":"172.21.15.139:6805/945038995","state":["exists","up"]},{"osd":3,"uuid":"5530b265-31a5-447b-a1ea-17ebae9cfddf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6810","nonce":3095249665},{"type":"v1","addr":"172.21.15.98:6811","nonce":3095249665}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6812","nonce":3095249665},{"type":"v1","addr":"172.21.15.98:6813","nonce":3095249665}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6816","nonce":3095249665},{"type":"v1","addr":"172.21.15.98:6817","nonce":3095249665}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6814","nonce":3095249665},{"type":"v1","addr":"172.21.15.98:6815","nonce":3095249665}]},"public_addr":"172.21.15.98:6811/3095249665","cluster_addr":"172.21.15.98:6813/3095249665","heartbeat_back_addr":"172.21.15.98:6817/3095249665","heartbeat_front_addr":"172.21.15.98:6815/3095249665","state":["exists","up"]},{"osd":4,"uuid":"615d4642-a997-4f95-9e38-9fbdaa831400","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":32,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.101:6808","nonce":2844761406},{"type":"v1","addr":"172.21.15.101:6809","nonce":2844761406}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.101:6810","nonce":2844761406},{"type":"v1","addr":"172.21.15.101:6811","nonce":2844761406}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.101:6814","nonce":2844761406},{"type":"v1","addr":"172.21.15.101:6815","nonce":2844761406}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.101:6812","nonce":2844761406},{"type":"v1","addr":"172.21.15.101:6813","nonce":2844761406}]},"public_addr":"172.21.15.101:6809/2844761406","cluster_addr":"172.21.15.101:6811/2844761406","heartbeat_back_addr":"172.21.15.101:6815/2844761406","heartbeat_front_addr":"172.21.15.101:6813/2844761406","state":["exists","up"]},{"osd":5,"uuid":"415de5fa-607a-45a8-b732-28daf3be6dd3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":37,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":3532263042},{"type":"v1","addr":"172.21.15.139:6809","nonce":3532263042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6810","nonce":3532263042},{"type":"v1","addr":"172.21.15.139:6811","nonce":3532263042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6814","nonce":3532263042},{"type":"v1","addr":"172.21.15.139:6815","nonce":3532263042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":3532263042},{"type":"v1","addr":"172.21.15.139:6813","nonce":3532263042}]},"public_addr":"172.21.15.139:6809/3532263042","cluster_addr":"172.21.15.139:6811/3532263042","heartbeat_back_addr":"172.21.15.139:6815/3532263042","heartbeat_front_addr":"172.21.15.139:6813/3532263042","state":["exists","up"]},{"osd":6,"uuid":"d7f9d4ba-c309-4b75-a836-5a36384b528e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":45,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6818","nonce":658948743},{"type":"v1","addr":"172.21.15.98:6819","nonce":658948743}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6820","nonce":658948743},{"type":"v1","addr":"172.21.15.98:6821","nonce":658948743}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6824","nonce":658948743},{"type":"v1","addr":"172.21.15.98:6825","nonce":658948743}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6822","nonce":658948743},{"type":"v1","addr":"172.21.15.98:6823","nonce":658948743}]},"public_addr":"172.21.15.98:6819/658948743","cluster_addr":"172.21.15.98:6821/658948743","heartbeat_back_addr":"172.21.15.98:6825/658948743","heartbeat_front_addr":"172.21.15.98:6823/658948743","state":["exists","up"]},{"osd":7,"uuid":"eedb94d9-c5cd-4a93-8a8c-2648804b7195","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":50,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.101:6816","nonce":2881214066},{"type":"v1","addr":"172.21.15.101:6817","nonce":2881214066}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.101:6818","nonce":2881214066},{"type":"v1","addr":"172.21.15.101:6819","nonce":2881214066}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.101:6822","nonce":2881214066},{"type":"v1","addr":"172.21.15.101:6823","nonce":2881214066}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.101:6820","nonce":2881214066},{"type":"v1","addr":"172.21.15.101:6821","nonce":2881214066}]},"public_addr":"172.21.15.101:6817/2881214066","cluster_addr":"172.21.15.101:6819/2881214066","heartbeat_back_addr":"172.21.15.101:6823/2881214066","heartbeat_front_addr":"172.21.15.101:6821/2881214066","state":["exists","up"]},{"osd":8,"uuid":"6bfd77c2-5770-466e-aa6f-84b50c74ffb6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":57,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":3669514524},{"type":"v1","addr":"172.21.15.139:6817","nonce":3669514524}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":3669514524},{"type":"v1","addr":"172.21.15.139:6819","nonce":3669514524}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":3669514524},{"type":"v1","addr":"172.21.15.139:6823","nonce":3669514524}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6820","nonce":3669514524},{"type":"v1","addr":"172.21.15.139:6821","nonce":3669514524}]},"public_addr":"172.21.15.139:6817/3669514524","cluster_addr":"172.21.15.139:6819/3669514524","heartbeat_back_addr":"172.21.15.139:6823/3669514524","heartbeat_front_addr":"172.21.15.139:6821/3669514524","state":["exists","up"]},{"osd":9,"uuid":"19c5b54a-2242-4b23-b721-63be9559653d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":65,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6826","nonce":2893994080},{"type":"v1","addr":"172.21.15.98:6827","nonce":2893994080}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6828","nonce":2893994080},{"type":"v1","addr":"172.21.15.98:6829","nonce":2893994080}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6832","nonce":2893994080},{"type":"v1","addr":"172.21.15.98:6833","nonce":2893994080}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6830","nonce":2893994080},{"type":"v1","addr":"172.21.15.98:6831","nonce":2893994080}]},"public_addr":"172.21.15.98:6827/2893994080","cluster_addr":"172.21.15.98:6829/2893994080","heartbeat_back_addr":"172.21.15.98:6833/2893994080","heartbeat_front_addr":"172.21.15.98:6831/2893994080","state":["exists","up"]},{"osd":10,"uuid":"326f0209-ba9d-4b59-97cc-d61a758a8f29","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":70,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.101:6824","nonce":3545244225},{"type":"v1","addr":"172.21.15.101:6825","nonce":3545244225}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.101:6826","nonce":3545244225},{"type":"v1","addr":"172.21.15.101:6827","nonce":3545244225}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.101:6830","nonce":3545244225},{"type":"v1","addr":"172.21.15.101:6831","nonce":3545244225}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.101:6828","nonce":3545244225},{"type":"v1","addr":"172.21.15.101:6829","nonce":3545244225}]},"public_addr":"172.21.15.101:6825/3545244225","cluster_addr":"172.21.15.101:6827/3545244225","heartbeat_back_addr":"172.21.15.101:6831/3545244225","heartbeat_front_addr":"172.21.15.101:6829/3545244225","state":["exists","up"]},{"osd":11,"uuid":"c61c0320-1057-4f17-9bf1-7ad306ed7ed3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":78,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":2630023454},{"type":"v1","addr":"172.21.15.139:6825","nonce":2630023454}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6826","nonce":2630023454},{"type":"v1","addr":"172.21.15.139:6827","nonce":2630023454}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":2630023454},{"type":"v1","addr":"172.21.15.139:6831","nonce":2630023454}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":2630023454},{"type":"v1","addr":"172.21.15.139:6829","nonce":2630023454}]},"public_addr":"172.21.15.139:6825/2630023454","cluster_addr":"172.21.15.139:6827/2630023454","heartbeat_back_addr":"172.21.15.139:6831/2630023454","heartbeat_front_addr":"172.21.15.139:6829/2630023454","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T19:24:23.620028+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T19:24:45.779846+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T19:25:06.877949+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T19:25:27.994199+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T19:25:49.461139+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T19:26:10.409282+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T19:26:32.116390+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T19:26:53.536474+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T19:27:14.384839+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T19:27:35.635921+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T19:27:56.858744+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T19:28:18.342519+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.98:0/2164590253":"2023-07-15T19:23:01.520302+0000","172.21.15.98:0/2094734896":"2023-07-15T19:23:01.520302+0000","172.21.15.98:0/784745009":"2023-07-15T19:22:39.481502+0000","172.21.15.98:0/3657190818":"2023-07-15T19:22:39.481502+0000","172.21.15.98:6800/1732769087":"2023-07-15T19:23:01.520302+0000","172.21.15.98:0/3393542419":"2023-07-15T19:22:39.481502+0000","172.21.15.98:6800/749234665":"2023-07-15T19:22:39.481502+0000","172.21.15.98:6801/1732769087":"2023-07-15T19:23:01.520302+0000","172.21.15.98:6801/749234665":"2023-07-15T19:22:39.481502+0000","172.21.15.98:0/187822873":"2023-07-15T19:23:01.520302+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-07-14T19:29:16.077 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:16.075+0000 7f8a906cc700 1 -- 172.21.15.98:0/2505448407 >> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] conn(0x7f8a740a11e0 msgr2=0x7f8a740a3690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:29:16.077 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:16.075+0000 7f8a906cc700 1 --2- 172.21.15.98:0/2505448407 >> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] conn(0x7f8a740a11e0 0x7f8a740a3690 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f8a84006fd0 tx=0x7f8a84008040 comp rx=0 tx=0).stop 2023-07-14T19:29:16.077 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:16.075+0000 7f8a906cc700 1 -- 172.21.15.98:0/2505448407 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8a880fe4b0 msgr2=0x7f8a88195db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:29:16.078 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:16.075+0000 7f8a906cc700 1 --2- 172.21.15.98:0/2505448407 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8a880fe4b0 0x7f8a88195db0 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f8a78004710 tx=0x7f8a780038c0 comp rx=0 tx=0).stop 2023-07-14T19:29:16.078 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:16.075+0000 7f8a906cc700 1 -- 172.21.15.98:0/2505448407 shutdown_connections 2023-07-14T19:29:16.078 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:16.075+0000 7f8a906cc700 1 --2- 172.21.15.98:0/2505448407 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f8a88105010 0x7f8a8819a760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:29:16.078 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:16.075+0000 7f8a906cc700 1 --2- 172.21.15.98:0/2505448407 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8a880fe4b0 0x7f8a88195db0 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:29:16.079 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:16.075+0000 7f8a906cc700 1 --2- 172.21.15.98:0/2505448407 >> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] conn(0x7f8a740a11e0 0x7f8a740a3690 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:29:16.079 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:16.075+0000 7f8a906cc700 1 --2- 172.21.15.98:0/2505448407 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f8a880fee50 0x7f8a881962f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:29:16.079 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:16.075+0000 7f8a906cc700 1 -- 172.21.15.98:0/2505448407 >> 172.21.15.98:0/2505448407 conn(0x7f8a88075280 msgr2=0x7f8a880fcb90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:29:16.079 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:16.075+0000 7f8a906cc700 1 -- 172.21.15.98:0/2505448407 shutdown_connections 2023-07-14T19:29:16.080 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:16.075+0000 7f8a906cc700 1 -- 172.21.15.98:0/2505448407 wait complete. 2023-07-14T19:29:16.434 INFO:tasks.cephadm.ceph_manager.ceph:all up! 2023-07-14T19:29:16.434 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81c9387d2e7576da065747ec9c16d1c7d69016ec shell --fsid 9adea648-227b-11ee-9b33-001a4aab830c -- ceph osd dump --format=json 2023-07-14T19:29:16.781 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:29:16 smithi098 ceph-mon[127486]: pgmap v260: 97 pgs: 97 active+clean; 621 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 7.2 KiB/s rd, 6.7 KiB/s wr, 16 op/s 2023-07-14T19:29:16.782 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:29:16 smithi098 ceph-mon[127486]: from='client.? 172.21.15.98:0/2505448407' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-07-14T19:29:16.941 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:29:16 smithi101 ceph-mon[128807]: pgmap v260: 97 pgs: 97 active+clean; 621 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 7.2 KiB/s rd, 6.7 KiB/s wr, 16 op/s 2023-07-14T19:29:16.941 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:29:16 smithi101 ceph-mon[128807]: from='client.? 172.21.15.98:0/2505448407' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-07-14T19:29:16.970 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:29:16 smithi139 ceph-mon[129293]: pgmap v260: 97 pgs: 97 active+clean; 621 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 7.2 KiB/s rd, 6.7 KiB/s wr, 16 op/s 2023-07-14T19:29:16.970 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:29:16 smithi139 ceph-mon[129293]: from='client.? 172.21.15.98:0/2505448407' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-07-14T19:29:16.986 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/9adea648-227b-11ee-9b33-001a4aab830c/mon.a/config 2023-07-14T19:29:18.141 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:18.139+0000 7f446a653700 1 -- 172.21.15.98:0/2532028377 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4464100f20 msgr2=0x7f44641012f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:29:18.141 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:18.139+0000 7f446a653700 1 --2- 172.21.15.98:0/2532028377 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4464100f20 0x7f44641012f0 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f4454009990 tx=0x7f4454009ca0 comp rx=0 tx=0).stop 2023-07-14T19:29:18.141 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:18.139+0000 7f446a653700 1 -- 172.21.15.98:0/2532028377 shutdown_connections 2023-07-14T19:29:18.141 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:18.139+0000 7f446a653700 1 --2- 172.21.15.98:0/2532028377 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f446410a270 0x7f446410c6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:29:18.142 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:18.139+0000 7f446a653700 1 --2- 172.21.15.98:0/2532028377 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4464100f20 0x7f44641012f0 unknown :-1 s=CLOSED pgs=240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:29:18.142 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:18.139+0000 7f446a653700 1 --2- 172.21.15.98:0/2532028377 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f4464101830 0x7f4464109d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:29:18.142 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:18.139+0000 7f446a653700 1 -- 172.21.15.98:0/2532028377 >> 172.21.15.98:0/2532028377 conn(0x7f4464075ca0 msgr2=0x7f44640760a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:29:18.142 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:18.139+0000 7f446a653700 1 -- 172.21.15.98:0/2532028377 shutdown_connections 2023-07-14T19:29:18.142 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:18.139+0000 7f446a653700 1 -- 172.21.15.98:0/2532028377 wait complete. 2023-07-14T19:29:18.142 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:18.139+0000 7f446a653700 1 Processor -- start 2023-07-14T19:29:18.142 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:18.139+0000 7f446a653700 1 -- start start 2023-07-14T19:29:18.144 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:18.140+0000 7f446a653700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f4464100f20 0x7f4464195d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:29:18.144 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:18.140+0000 7f446a653700 1 --2- >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f4464101830 0x7f4464196290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:29:18.144 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:18.140+0000 7f446a653700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f446410a270 0x7f446419a700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:29:18.144 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:18.140+0000 7f446a653700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f446419ad10 con 0x7f446410a270 2023-07-14T19:29:18.145 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:18.140+0000 7f446a653700 1 -- --> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f446419ae80 con 0x7f4464101830 2023-07-14T19:29:18.145 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:18.140+0000 7f446a653700 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f446419b040 con 0x7f4464100f20 2023-07-14T19:29:18.145 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:18.140+0000 7f4468bf0700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f446410a270 0x7f446419a700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:29:18.145 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:18.140+0000 7f4468bf0700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f446410a270 0x7f446419a700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:45920/0 (socket says 172.21.15.98:45920) 2023-07-14T19:29:18.145 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:18.140+0000 7f4468bf0700 1 -- 172.21.15.98:0/2359743977 learned_addr learned my addr 172.21.15.98:0/2359743977 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-07-14T19:29:18.145 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:18.140+0000 7f4463fff700 1 --2- 172.21.15.98:0/2359743977 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f4464100f20 0x7f4464195d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:29:18.145 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:18.140+0000 7f44637fe700 1 --2- 172.21.15.98:0/2359743977 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f4464101830 0x7f4464196290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:29:18.146 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:18.140+0000 7f4468bf0700 1 -- 172.21.15.98:0/2359743977 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f4464100f20 msgr2=0x7f4464195d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:29:18.146 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:18.140+0000 7f4468bf0700 1 --2- 172.21.15.98:0/2359743977 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f4464100f20 0x7f4464195d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:29:18.147 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:18.140+0000 7f4468bf0700 1 -- 172.21.15.98:0/2359743977 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f4464101830 msgr2=0x7f4464196290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:29:18.147 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:18.140+0000 7f4468bf0700 1 --2- 172.21.15.98:0/2359743977 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f4464101830 0x7f4464196290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:29:18.147 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:18.140+0000 7f4468bf0700 1 -- 172.21.15.98:0/2359743977 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f44540095f0 con 0x7f446410a270 2023-07-14T19:29:18.147 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:18.140+0000 7f44637fe700 1 --2- 172.21.15.98:0/2359743977 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f4464101830 0x7f4464196290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-07-14T19:29:18.147 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:18.140+0000 7f4468bf0700 1 --2- 172.21.15.98:0/2359743977 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f446410a270 0x7f446419a700 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f445c007f10 tx=0x7f445c00d3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:29:18.147 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:18.140+0000 7f44617fa700 1 -- 172.21.15.98:0/2359743977 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f445c017070 con 0x7f446410a270 2023-07-14T19:29:18.148 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:18.140+0000 7f44617fa700 1 -- 172.21.15.98:0/2359743977 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f445c00f040 con 0x7f446410a270 2023-07-14T19:29:18.148 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:18.140+0000 7f446a653700 1 -- 172.21.15.98:0/2359743977 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f446419b320 con 0x7f446410a270 2023-07-14T19:29:18.148 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:18.141+0000 7f44617fa700 1 -- 172.21.15.98:0/2359743977 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f445c005740 con 0x7f446410a270 2023-07-14T19:29:18.148 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:18.141+0000 7f446a653700 1 -- 172.21.15.98:0/2359743977 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f44641a2b80 con 0x7f446410a270 2023-07-14T19:29:18.148 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:18.141+0000 7f446a653700 1 -- 172.21.15.98:0/2359743977 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4464104ac0 con 0x7f446410a270 2023-07-14T19:29:18.149 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:18.143+0000 7f44617fa700 1 -- 172.21.15.98:0/2359743977 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 17) v1 ==== 135766+0+0 (secure 0 0 0) 0x7f445c00dc60 con 0x7f446410a270 2023-07-14T19:29:18.150 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:18.143+0000 7f44617fa700 1 --2- 172.21.15.98:0/2359743977 >> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] conn(0x7f444c0a1260 0x7f444c0a3710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:29:18.150 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:18.143+0000 7f4463fff700 1 --2- 172.21.15.98:0/2359743977 >> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] conn(0x7f444c0a1260 0x7f444c0a3710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:29:18.150 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:18.143+0000 7f44617fa700 1 -- 172.21.15.98:0/2359743977 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 7979+0+0 (secure 0 0 0) 0x7f445c0cc5a0 con 0x7f446410a270 2023-07-14T19:29:18.150 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:18.144+0000 7f4463fff700 1 --2- 172.21.15.98:0/2359743977 >> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] conn(0x7f444c0a1260 0x7f444c0a3710 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f445400b2e0 tx=0x7f4454005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:29:18.150 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:18.147+0000 7f44617fa700 1 -- 172.21.15.98:0/2359743977 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f445c095d40 con 0x7f446410a270 2023-07-14T19:29:18.333 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:18.331+0000 7f446a653700 1 -- 172.21.15.98:0/2359743977 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f446404e340 con 0x7f446410a270 2023-07-14T19:29:18.334 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:18.333+0000 7f44617fa700 1 -- 172.21.15.98:0/2359743977 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v83) v1 ==== 74+0+22117 (secure 0 0 0) 0x7f445c099360 con 0x7f446410a270 2023-07-14T19:29:18.335 INFO:teuthology.orchestra.run.smithi098.stdout: 2023-07-14T19:29:18.335 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":83,"fsid":"9adea648-227b-11ee-9b33-001a4aab830c","created":"2023-07-14T19:22:07.178854+0000","modified":"2023-07-14T19:29:02.293099+0000","last_up_change":"2023-07-14T19:28:19.646827+0000","last_in_change":"2023-07-14T19:28:02.018325+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-07-14T19:25:09.845827+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2023-07-14T19:28:58.952921+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"82","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2023-07-14T19:29:00.228272+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"83","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"a3797e89-1754-41cf-9517-227e06ffdd72","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6802","nonce":120507228},{"type":"v1","addr":"172.21.15.98:6803","nonce":120507228}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6804","nonce":120507228},{"type":"v1","addr":"172.21.15.98:6805","nonce":120507228}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6808","nonce":120507228},{"type":"v1","addr":"172.21.15.98:6809","nonce":120507228}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6806","nonce":120507228},{"type":"v1","addr":"172.21.15.98:6807","nonce":120507228}]},"public_addr":"172.21.15.98:6803/120507228","cluster_addr":"172.21.15.98:6805/120507228","heartbeat_back_addr":"172.21.15.98:6809/120507228","heartbeat_front_addr":"172.21.15.98:6807/120507228","state":["exists","up"]},{"osd":1,"uuid":"2b001add-1aaa-4496-a403-7ef9725b80a0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.101:6800","nonce":2450584584},{"type":"v1","addr":"172.21.15.101:6801","nonce":2450584584}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.101:6802","nonce":2450584584},{"type":"v1","addr":"172.21.15.101:6803","nonce":2450584584}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.101:6806","nonce":2450584584},{"type":"v1","addr":"172.21.15.101:6807","nonce":2450584584}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.101:6804","nonce":2450584584},{"type":"v1","addr":"172.21.15.101:6805","nonce":2450584584}]},"public_addr":"172.21.15.101:6801/2450584584","cluster_addr":"172.21.15.101:6803/2450584584","heartbeat_back_addr":"172.21.15.101:6807/2450584584","heartbeat_front_addr":"172.21.15.101:6805/2450584584","state":["exists","up"]},{"osd":2,"uuid":"48e6d499-3ca9-424b-b0fb-3ba5f33b9fc9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":945038995},{"type":"v1","addr":"172.21.15.139:6801","nonce":945038995}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":945038995},{"type":"v1","addr":"172.21.15.139:6803","nonce":945038995}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":945038995},{"type":"v1","addr":"172.21.15.139:6807","nonce":945038995}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":945038995},{"type":"v1","addr":"172.21.15.139:6805","nonce":945038995}]},"public_addr":"172.21.15.139:6801/945038995","cluster_addr":"172.21.15.139:6803/945038995","heartbeat_back_addr":"172.21.15.139:6807/945038995","heartbeat_front_addr":"172.21.15.139:6805/945038995","state":["exists","up"]},{"osd":3,"uuid":"5530b265-31a5-447b-a1ea-17ebae9cfddf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6810","nonce":3095249665},{"type":"v1","addr":"172.21.15.98:6811","nonce":3095249665}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6812","nonce":3095249665},{"type":"v1","addr":"172.21.15.98:6813","nonce":3095249665}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6816","nonce":3095249665},{"type":"v1","addr":"172.21.15.98:6817","nonce":3095249665}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6814","nonce":3095249665},{"type":"v1","addr":"172.21.15.98:6815","nonce":3095249665}]},"public_addr":"172.21.15.98:6811/3095249665","cluster_addr":"172.21.15.98:6813/3095249665","heartbeat_back_addr":"172.21.15.98:6817/3095249665","heartbeat_front_addr":"172.21.15.98:6815/3095249665","state":["exists","up"]},{"osd":4,"uuid":"615d4642-a997-4f95-9e38-9fbdaa831400","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":32,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.101:6808","nonce":2844761406},{"type":"v1","addr":"172.21.15.101:6809","nonce":2844761406}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.101:6810","nonce":2844761406},{"type":"v1","addr":"172.21.15.101:6811","nonce":2844761406}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.101:6814","nonce":2844761406},{"type":"v1","addr":"172.21.15.101:6815","nonce":2844761406}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.101:6812","nonce":2844761406},{"type":"v1","addr":"172.21.15.101:6813","nonce":2844761406}]},"public_addr":"172.21.15.101:6809/2844761406","cluster_addr":"172.21.15.101:6811/2844761406","heartbeat_back_addr":"172.21.15.101:6815/2844761406","heartbeat_front_addr":"172.21.15.101:6813/2844761406","state":["exists","up"]},{"osd":5,"uuid":"415de5fa-607a-45a8-b732-28daf3be6dd3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":37,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":3532263042},{"type":"v1","addr":"172.21.15.139:6809","nonce":3532263042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6810","nonce":3532263042},{"type":"v1","addr":"172.21.15.139:6811","nonce":3532263042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6814","nonce":3532263042},{"type":"v1","addr":"172.21.15.139:6815","nonce":3532263042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":3532263042},{"type":"v1","addr":"172.21.15.139:6813","nonce":3532263042}]},"public_addr":"172.21.15.139:6809/3532263042","cluster_addr":"172.21.15.139:6811/3532263042","heartbeat_back_addr":"172.21.15.139:6815/3532263042","heartbeat_front_addr":"172.21.15.139:6813/3532263042","state":["exists","up"]},{"osd":6,"uuid":"d7f9d4ba-c309-4b75-a836-5a36384b528e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":45,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6818","nonce":658948743},{"type":"v1","addr":"172.21.15.98:6819","nonce":658948743}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6820","nonce":658948743},{"type":"v1","addr":"172.21.15.98:6821","nonce":658948743}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6824","nonce":658948743},{"type":"v1","addr":"172.21.15.98:6825","nonce":658948743}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6822","nonce":658948743},{"type":"v1","addr":"172.21.15.98:6823","nonce":658948743}]},"public_addr":"172.21.15.98:6819/658948743","cluster_addr":"172.21.15.98:6821/658948743","heartbeat_back_addr":"172.21.15.98:6825/658948743","heartbeat_front_addr":"172.21.15.98:6823/658948743","state":["exists","up"]},{"osd":7,"uuid":"eedb94d9-c5cd-4a93-8a8c-2648804b7195","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":50,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.101:6816","nonce":2881214066},{"type":"v1","addr":"172.21.15.101:6817","nonce":2881214066}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.101:6818","nonce":2881214066},{"type":"v1","addr":"172.21.15.101:6819","nonce":2881214066}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.101:6822","nonce":2881214066},{"type":"v1","addr":"172.21.15.101:6823","nonce":2881214066}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.101:6820","nonce":2881214066},{"type":"v1","addr":"172.21.15.101:6821","nonce":2881214066}]},"public_addr":"172.21.15.101:6817/2881214066","cluster_addr":"172.21.15.101:6819/2881214066","heartbeat_back_addr":"172.21.15.101:6823/2881214066","heartbeat_front_addr":"172.21.15.101:6821/2881214066","state":["exists","up"]},{"osd":8,"uuid":"6bfd77c2-5770-466e-aa6f-84b50c74ffb6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":57,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":3669514524},{"type":"v1","addr":"172.21.15.139:6817","nonce":3669514524}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":3669514524},{"type":"v1","addr":"172.21.15.139:6819","nonce":3669514524}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":3669514524},{"type":"v1","addr":"172.21.15.139:6823","nonce":3669514524}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6820","nonce":3669514524},{"type":"v1","addr":"172.21.15.139:6821","nonce":3669514524}]},"public_addr":"172.21.15.139:6817/3669514524","cluster_addr":"172.21.15.139:6819/3669514524","heartbeat_back_addr":"172.21.15.139:6823/3669514524","heartbeat_front_addr":"172.21.15.139:6821/3669514524","state":["exists","up"]},{"osd":9,"uuid":"19c5b54a-2242-4b23-b721-63be9559653d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":65,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6826","nonce":2893994080},{"type":"v1","addr":"172.21.15.98:6827","nonce":2893994080}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6828","nonce":2893994080},{"type":"v1","addr":"172.21.15.98:6829","nonce":2893994080}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6832","nonce":2893994080},{"type":"v1","addr":"172.21.15.98:6833","nonce":2893994080}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6830","nonce":2893994080},{"type":"v1","addr":"172.21.15.98:6831","nonce":2893994080}]},"public_addr":"172.21.15.98:6827/2893994080","cluster_addr":"172.21.15.98:6829/2893994080","heartbeat_back_addr":"172.21.15.98:6833/2893994080","heartbeat_front_addr":"172.21.15.98:6831/2893994080","state":["exists","up"]},{"osd":10,"uuid":"326f0209-ba9d-4b59-97cc-d61a758a8f29","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":70,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.101:6824","nonce":3545244225},{"type":"v1","addr":"172.21.15.101:6825","nonce":3545244225}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.101:6826","nonce":3545244225},{"type":"v1","addr":"172.21.15.101:6827","nonce":3545244225}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.101:6830","nonce":3545244225},{"type":"v1","addr":"172.21.15.101:6831","nonce":3545244225}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.101:6828","nonce":3545244225},{"type":"v1","addr":"172.21.15.101:6829","nonce":3545244225}]},"public_addr":"172.21.15.101:6825/3545244225","cluster_addr":"172.21.15.101:6827/3545244225","heartbeat_back_addr":"172.21.15.101:6831/3545244225","heartbeat_front_addr":"172.21.15.101:6829/3545244225","state":["exists","up"]},{"osd":11,"uuid":"c61c0320-1057-4f17-9bf1-7ad306ed7ed3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":78,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":2630023454},{"type":"v1","addr":"172.21.15.139:6825","nonce":2630023454}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6826","nonce":2630023454},{"type":"v1","addr":"172.21.15.139:6827","nonce":2630023454}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":2630023454},{"type":"v1","addr":"172.21.15.139:6831","nonce":2630023454}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":2630023454},{"type":"v1","addr":"172.21.15.139:6829","nonce":2630023454}]},"public_addr":"172.21.15.139:6825/2630023454","cluster_addr":"172.21.15.139:6827/2630023454","heartbeat_back_addr":"172.21.15.139:6831/2630023454","heartbeat_front_addr":"172.21.15.139:6829/2630023454","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T19:24:23.620028+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T19:24:45.779846+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T19:25:06.877949+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T19:25:27.994199+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T19:25:49.461139+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T19:26:10.409282+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T19:26:32.116390+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T19:26:53.536474+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T19:27:14.384839+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T19:27:35.635921+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T19:27:56.858744+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T19:28:18.342519+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.98:0/2164590253":"2023-07-15T19:23:01.520302+0000","172.21.15.98:0/2094734896":"2023-07-15T19:23:01.520302+0000","172.21.15.98:0/784745009":"2023-07-15T19:22:39.481502+0000","172.21.15.98:0/3657190818":"2023-07-15T19:22:39.481502+0000","172.21.15.98:6800/1732769087":"2023-07-15T19:23:01.520302+0000","172.21.15.98:0/3393542419":"2023-07-15T19:22:39.481502+0000","172.21.15.98:6800/749234665":"2023-07-15T19:22:39.481502+0000","172.21.15.98:6801/1732769087":"2023-07-15T19:23:01.520302+0000","172.21.15.98:6801/749234665":"2023-07-15T19:22:39.481502+0000","172.21.15.98:0/187822873":"2023-07-15T19:23:01.520302+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-07-14T19:29:18.336 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:18.335+0000 7f446a653700 1 -- 172.21.15.98:0/2359743977 >> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] conn(0x7f444c0a1260 msgr2=0x7f444c0a3710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:29:18.337 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:18.335+0000 7f446a653700 1 --2- 172.21.15.98:0/2359743977 >> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] conn(0x7f444c0a1260 0x7f444c0a3710 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f445400b2e0 tx=0x7f4454005fd0 comp rx=0 tx=0).stop 2023-07-14T19:29:18.337 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:18.335+0000 7f446a653700 1 -- 172.21.15.98:0/2359743977 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f446410a270 msgr2=0x7f446419a700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:29:18.337 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:18.335+0000 7f446a653700 1 --2- 172.21.15.98:0/2359743977 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f446410a270 0x7f446419a700 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f445c007f10 tx=0x7f445c00d3b0 comp rx=0 tx=0).stop 2023-07-14T19:29:18.337 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:18.335+0000 7f446a653700 1 -- 172.21.15.98:0/2359743977 shutdown_connections 2023-07-14T19:29:18.337 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:18.335+0000 7f446a653700 1 --2- 172.21.15.98:0/2359743977 >> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] conn(0x7f444c0a1260 0x7f444c0a3710 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:29:18.337 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:18.335+0000 7f446a653700 1 --2- 172.21.15.98:0/2359743977 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f4464100f20 0x7f4464195d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:29:18.337 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:18.335+0000 7f446a653700 1 --2- 172.21.15.98:0/2359743977 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f446410a270 0x7f446419a700 unknown :-1 s=CLOSED pgs=241 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:29:18.338 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:18.335+0000 7f446a653700 1 --2- 172.21.15.98:0/2359743977 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f4464101830 0x7f4464196290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:29:18.338 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:18.335+0000 7f446a653700 1 -- 172.21.15.98:0/2359743977 >> 172.21.15.98:0/2359743977 conn(0x7f4464075ca0 msgr2=0x7f4464101c00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:29:18.338 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:18.335+0000 7f446a653700 1 -- 172.21.15.98:0/2359743977 shutdown_connections 2023-07-14T19:29:18.339 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:18.335+0000 7f446a653700 1 -- 172.21.15.98:0/2359743977 wait complete. 2023-07-14T19:29:18.728 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81c9387d2e7576da065747ec9c16d1c7d69016ec shell --fsid 9adea648-227b-11ee-9b33-001a4aab830c -- ceph tell osd.0 flush_pg_stats 2023-07-14T19:29:18.729 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81c9387d2e7576da065747ec9c16d1c7d69016ec shell --fsid 9adea648-227b-11ee-9b33-001a4aab830c -- ceph tell osd.1 flush_pg_stats 2023-07-14T19:29:18.730 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81c9387d2e7576da065747ec9c16d1c7d69016ec shell --fsid 9adea648-227b-11ee-9b33-001a4aab830c -- ceph tell osd.2 flush_pg_stats 2023-07-14T19:29:18.731 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81c9387d2e7576da065747ec9c16d1c7d69016ec shell --fsid 9adea648-227b-11ee-9b33-001a4aab830c -- ceph tell osd.3 flush_pg_stats 2023-07-14T19:29:18.735 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81c9387d2e7576da065747ec9c16d1c7d69016ec shell --fsid 9adea648-227b-11ee-9b33-001a4aab830c -- ceph tell osd.4 flush_pg_stats 2023-07-14T19:29:18.736 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81c9387d2e7576da065747ec9c16d1c7d69016ec shell --fsid 9adea648-227b-11ee-9b33-001a4aab830c -- ceph tell osd.5 flush_pg_stats 2023-07-14T19:29:18.737 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81c9387d2e7576da065747ec9c16d1c7d69016ec shell --fsid 9adea648-227b-11ee-9b33-001a4aab830c -- ceph tell osd.6 flush_pg_stats 2023-07-14T19:29:18.737 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81c9387d2e7576da065747ec9c16d1c7d69016ec shell --fsid 9adea648-227b-11ee-9b33-001a4aab830c -- ceph tell osd.7 flush_pg_stats 2023-07-14T19:29:18.738 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81c9387d2e7576da065747ec9c16d1c7d69016ec shell --fsid 9adea648-227b-11ee-9b33-001a4aab830c -- ceph tell osd.8 flush_pg_stats 2023-07-14T19:29:18.739 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81c9387d2e7576da065747ec9c16d1c7d69016ec shell --fsid 9adea648-227b-11ee-9b33-001a4aab830c -- ceph tell osd.9 flush_pg_stats 2023-07-14T19:29:18.739 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81c9387d2e7576da065747ec9c16d1c7d69016ec shell --fsid 9adea648-227b-11ee-9b33-001a4aab830c -- ceph tell osd.10 flush_pg_stats 2023-07-14T19:29:18.739 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81c9387d2e7576da065747ec9c16d1c7d69016ec shell --fsid 9adea648-227b-11ee-9b33-001a4aab830c -- ceph tell osd.11 flush_pg_stats 2023-07-14T19:29:18.941 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:29:18 smithi101 ceph-mon[128807]: pgmap v261: 97 pgs: 97 active+clean; 621 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 7.2 KiB/s rd, 5.8 KiB/s wr, 16 op/s 2023-07-14T19:29:18.941 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:29:18 smithi101 ceph-mon[128807]: from='client.? 172.21.15.98:0/2359743977' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-07-14T19:29:18.970 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:29:18 smithi139 ceph-mon[129293]: pgmap v261: 97 pgs: 97 active+clean; 621 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 7.2 KiB/s rd, 5.8 KiB/s wr, 16 op/s 2023-07-14T19:29:18.970 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:29:18 smithi139 ceph-mon[129293]: from='client.? 172.21.15.98:0/2359743977' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-07-14T19:29:19.031 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:29:18 smithi098 ceph-mon[127486]: pgmap v261: 97 pgs: 97 active+clean; 621 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 7.2 KiB/s rd, 5.8 KiB/s wr, 16 op/s 2023-07-14T19:29:19.032 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:29:18 smithi098 ceph-mon[127486]: from='client.? 172.21.15.98:0/2359743977' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-07-14T19:29:20.049 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/9adea648-227b-11ee-9b33-001a4aab830c/mon.a/config 2023-07-14T19:29:20.263 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/9adea648-227b-11ee-9b33-001a4aab830c/mon.a/config 2023-07-14T19:29:20.436 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/9adea648-227b-11ee-9b33-001a4aab830c/mon.a/config 2023-07-14T19:29:20.454 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/9adea648-227b-11ee-9b33-001a4aab830c/mon.a/config 2023-07-14T19:29:20.491 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/9adea648-227b-11ee-9b33-001a4aab830c/mon.a/config 2023-07-14T19:29:20.719 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/9adea648-227b-11ee-9b33-001a4aab830c/mon.a/config 2023-07-14T19:29:20.941 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:29:20 smithi101 ceph-mon[128807]: pgmap v262: 97 pgs: 97 active+clean; 633 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 8.7 KiB/s rd, 5.3 KiB/s wr, 13 op/s 2023-07-14T19:29:20.970 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:29:20 smithi139 ceph-mon[129293]: pgmap v262: 97 pgs: 97 active+clean; 633 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 8.7 KiB/s rd, 5.3 KiB/s wr, 13 op/s 2023-07-14T19:29:21.029 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/9adea648-227b-11ee-9b33-001a4aab830c/mon.a/config 2023-07-14T19:29:21.031 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:29:20 smithi098 ceph-mon[127486]: pgmap v262: 97 pgs: 97 active+clean; 633 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 8.7 KiB/s rd, 5.3 KiB/s wr, 13 op/s 2023-07-14T19:29:21.219 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/9adea648-227b-11ee-9b33-001a4aab830c/mon.a/config 2023-07-14T19:29:22.562 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:29:22 smithi098 ceph-mon[127486]: pgmap v263: 97 pgs: 97 active+clean; 633 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 2.1 KiB/s wr, 8 op/s 2023-07-14T19:29:22.691 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:29:22 smithi101 ceph-mon[128807]: pgmap v263: 97 pgs: 97 active+clean; 633 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 2.1 KiB/s wr, 8 op/s 2023-07-14T19:29:22.720 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:29:22 smithi139 ceph-mon[129293]: pgmap v263: 97 pgs: 97 active+clean; 633 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 2.1 KiB/s wr, 8 op/s 2023-07-14T19:29:22.929 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/9adea648-227b-11ee-9b33-001a4aab830c/mon.a/config 2023-07-14T19:29:22.979 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/9adea648-227b-11ee-9b33-001a4aab830c/mon.a/config 2023-07-14T19:29:23.343 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/9adea648-227b-11ee-9b33-001a4aab830c/mon.a/config 2023-07-14T19:29:23.412 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:23.410+0000 7efed2d2e700 1 -- 172.21.15.98:0/3617143363 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7efecc101150 msgr2=0x7efecc10bb40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:29:23.412 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:23.410+0000 7efed2d2e700 1 --2- 172.21.15.98:0/3617143363 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7efecc101150 0x7efecc10bb40 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7efec400b0d0 tx=0x7efec400b3e0 comp rx=0 tx=0).stop 2023-07-14T19:29:23.413 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:23.410+0000 7efed2d2e700 1 -- 172.21.15.98:0/3617143363 shutdown_connections 2023-07-14T19:29:23.413 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:23.410+0000 7efed2d2e700 1 --2- 172.21.15.98:0/3617143363 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7efecc101150 0x7efecc10bb40 unknown :-1 s=CLOSED pgs=242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:29:23.413 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:23.410+0000 7efed2d2e700 1 --2- 172.21.15.98:0/3617143363 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7efecc0fef40 0x7efecc0ff310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:29:23.413 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:23.410+0000 7efed2d2e700 1 --2- 172.21.15.98:0/3617143363 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7efecc1007a0 0x7efecc100c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:29:23.413 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:23.410+0000 7efed2d2e700 1 -- 172.21.15.98:0/3617143363 >> 172.21.15.98:0/3617143363 conn(0x7efecc075c90 msgr2=0x7efecc076090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:29:23.413 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:23.410+0000 7efed2d2e700 1 -- 172.21.15.98:0/3617143363 shutdown_connections 2023-07-14T19:29:23.413 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:23.410+0000 7efed2d2e700 1 -- 172.21.15.98:0/3617143363 wait complete. 2023-07-14T19:29:23.415 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:23.411+0000 7efed2d2e700 1 Processor -- start 2023-07-14T19:29:23.415 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:23.411+0000 7efed2d2e700 1 -- start start 2023-07-14T19:29:23.415 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:23.411+0000 7efed2d2e700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7efecc0fef40 0x7efecc196080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:29:23.415 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:23.411+0000 7efed2d2e700 1 --2- >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7efecc1007a0 0x7efecc1965c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:29:23.416 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:23.411+0000 7efed2d2e700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7efecc101150 0x7efecc19aa30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:29:23.416 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:23.411+0000 7efed2d2e700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efecc19b040 con 0x7efecc0fef40 2023-07-14T19:29:23.416 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:23.411+0000 7efed2d2e700 1 -- --> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efecc19b1b0 con 0x7efecc1007a0 2023-07-14T19:29:23.416 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:23.411+0000 7efed2d2e700 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efecc19b370 con 0x7efecc101150 2023-07-14T19:29:23.416 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:23.411+0000 7efed0aca700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7efecc0fef40 0x7efecc196080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:29:23.416 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:23.411+0000 7efed0aca700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7efecc0fef40 0x7efecc196080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:45930/0 (socket says 172.21.15.98:45930) 2023-07-14T19:29:23.416 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:23.411+0000 7efed0aca700 1 -- 172.21.15.98:0/2534886379 learned_addr learned my addr 172.21.15.98:0/2534886379 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-07-14T19:29:23.417 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:23.412+0000 7efed12cb700 1 --2- 172.21.15.98:0/2534886379 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7efecc101150 0x7efecc19aa30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:29:23.417 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:23.412+0000 7efecbfff700 1 --2- 172.21.15.98:0/2534886379 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7efecc1007a0 0x7efecc1965c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:29:23.417 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:23.412+0000 7efed0aca700 1 -- 172.21.15.98:0/2534886379 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7efecc101150 msgr2=0x7efecc19aa30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:29:23.417 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:23.412+0000 7efed0aca700 1 --2- 172.21.15.98:0/2534886379 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7efecc101150 0x7efecc19aa30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:29:23.417 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:23.412+0000 7efed0aca700 1 -- 172.21.15.98:0/2534886379 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7efecc1007a0 msgr2=0x7efecc1965c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:29:23.417 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:23.412+0000 7efed0aca700 1 --2- 172.21.15.98:0/2534886379 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7efecc1007a0 0x7efecc1965c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:29:23.417 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:23.412+0000 7efed0aca700 1 -- 172.21.15.98:0/2534886379 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efec4009d00 con 0x7efecc0fef40 2023-07-14T19:29:23.418 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:23.412+0000 7efecbfff700 1 --2- 172.21.15.98:0/2534886379 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7efecc1007a0 0x7efecc1965c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-07-14T19:29:23.418 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:23.412+0000 7efed0aca700 1 --2- 172.21.15.98:0/2534886379 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7efecc0fef40 0x7efecc196080 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7efec000b530 tx=0x7efec000b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:29:23.418 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:23.412+0000 7efec9ffb700 1 -- 172.21.15.98:0/2534886379 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efec0011830 con 0x7efecc0fef40 2023-07-14T19:29:23.418 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:23.412+0000 7efec9ffb700 1 -- 172.21.15.98:0/2534886379 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7efec0011e50 con 0x7efecc0fef40 2023-07-14T19:29:23.418 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:23.412+0000 7efec9ffb700 1 -- 172.21.15.98:0/2534886379 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efec000f2e0 con 0x7efecc0fef40 2023-07-14T19:29:23.418 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:23.412+0000 7efed2d2e700 1 -- 172.21.15.98:0/2534886379 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efecc19b650 con 0x7efecc0fef40 2023-07-14T19:29:23.418 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:23.412+0000 7efed2d2e700 1 -- 172.21.15.98:0/2534886379 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efecc1a2eb0 con 0x7efecc0fef40 2023-07-14T19:29:23.418 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:23.413+0000 7efed2d2e700 1 -- 172.21.15.98:0/2534886379 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7efecc110040 con 0x7efecc0fef40 2023-07-14T19:29:23.419 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:23.414+0000 7efec9ffb700 1 -- 172.21.15.98:0/2534886379 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 17) v1 ==== 135766+0+0 (secure 0 0 0) 0x7efec0010580 con 0x7efecc0fef40 2023-07-14T19:29:23.420 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:23.415+0000 7efec9ffb700 1 --2- 172.21.15.98:0/2534886379 >> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] conn(0x7efebc0a1260 0x7efebc0a3710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:29:23.420 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:23.415+0000 7efecbfff700 1 --2- 172.21.15.98:0/2534886379 >> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] conn(0x7efebc0a1260 0x7efebc0a3710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:29:23.420 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:23.415+0000 7efec9ffb700 1 -- 172.21.15.98:0/2534886379 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 7979+0+0 (secure 0 0 0) 0x7efec00cc8f0 con 0x7efecc0fef40 2023-07-14T19:29:23.420 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:23.415+0000 7efec9ffb700 1 --2- 172.21.15.98:0/2534886379 >> [v2:172.21.15.98:6802/120507228,v1:172.21.15.98:6803/120507228] conn(0x7efebc0aaa30 0x7efebc0ace40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:29:23.420 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:23.415+0000 7efec9ffb700 1 -- 172.21.15.98:0/2534886379 --> [v2:172.21.15.98:6802/120507228,v1:172.21.15.98:6803/120507228] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7efebc0ad4f0 con 0x7efebc0aaa30 2023-07-14T19:29:23.420 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:23.415+0000 7efec9ffb700 1 -- 172.21.15.98:0/2534886379 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_get_version_reply(handle=1 version=83) v2 ==== 24+0+0 (secure 0 0 0) 0x7efec00ccd90 con 0x7efecc0fef40 2023-07-14T19:29:23.421 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:23.415+0000 7efed12cb700 1 --2- 172.21.15.98:0/2534886379 >> [v2:172.21.15.98:6802/120507228,v1:172.21.15.98:6803/120507228] conn(0x7efebc0aaa30 0x7efebc0ace40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:29:23.421 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:23.415+0000 7efecbfff700 1 --2- 172.21.15.98:0/2534886379 >> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] conn(0x7efebc0a1260 0x7efebc0a3710 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7efecc1976a0 tx=0x7efeb8008040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:29:23.421 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:23.415+0000 7efed12cb700 1 --2- 172.21.15.98:0/2534886379 >> [v2:172.21.15.98:6802/120507228,v1:172.21.15.98:6803/120507228] conn(0x7efebc0aaa30 0x7efebc0ace40 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:29:23.421 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:23.416+0000 7efec9ffb700 1 -- 172.21.15.98:0/2534886379 <== osd.0 v2:172.21.15.98:6802/120507228 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25106 (crc 0 0 0) 0x7efebc0ad4f0 con 0x7efebc0aaa30 2023-07-14T19:29:23.433 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:23.431+0000 7efed2d2e700 1 -- 172.21.15.98:0/2534886379 --> [v2:172.21.15.98:6802/120507228,v1:172.21.15.98:6803/120507228] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7efecc04e340 con 0x7efebc0aaa30 2023-07-14T19:29:23.434 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:23.432+0000 7efec9ffb700 1 -- 172.21.15.98:0/2534886379 <== osd.0 v2:172.21.15.98:6802/120507228 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7efecc04e340 con 0x7efebc0aaa30 2023-07-14T19:29:23.435 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:23.432+0000 7efed2d2e700 1 -- 172.21.15.98:0/2534886379 >> [v2:172.21.15.98:6802/120507228,v1:172.21.15.98:6803/120507228] conn(0x7efebc0aaa30 msgr2=0x7efebc0ace40 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:29:23.435 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:23.432+0000 7efed2d2e700 1 --2- 172.21.15.98:0/2534886379 >> [v2:172.21.15.98:6802/120507228,v1:172.21.15.98:6803/120507228] conn(0x7efebc0aaa30 0x7efebc0ace40 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:29:23.436 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:23.432+0000 7efed2d2e700 1 -- 172.21.15.98:0/2534886379 >> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] conn(0x7efebc0a1260 msgr2=0x7efebc0a3710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:29:23.436 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:23.432+0000 7efed2d2e700 1 --2- 172.21.15.98:0/2534886379 >> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] conn(0x7efebc0a1260 0x7efebc0a3710 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7efecc1976a0 tx=0x7efeb8008040 comp rx=0 tx=0).stop 2023-07-14T19:29:23.436 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:23.432+0000 7efed2d2e700 1 -- 172.21.15.98:0/2534886379 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7efecc0fef40 msgr2=0x7efecc196080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:29:23.437 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:23.432+0000 7efed2d2e700 1 --2- 172.21.15.98:0/2534886379 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7efecc0fef40 0x7efecc196080 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7efec000b530 tx=0x7efec000b8f0 comp rx=0 tx=0).stop 2023-07-14T19:29:23.437 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:23.432+0000 7efed12cb700 1 -- 172.21.15.98:0/2534886379 reap_dead start 2023-07-14T19:29:23.437 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:23.432+0000 7efed2d2e700 1 -- 172.21.15.98:0/2534886379 shutdown_connections 2023-07-14T19:29:23.437 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:23.432+0000 7efed2d2e700 1 -- 172.21.15.98:0/2534886379 >> 172.21.15.98:0/2534886379 conn(0x7efecc075c90 msgr2=0x7efecc0fc4f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:29:23.438 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:23.432+0000 7efed2d2e700 1 -- 172.21.15.98:0/2534886379 shutdown_connections 2023-07-14T19:29:23.438 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:23.432+0000 7efed2d2e700 1 -- 172.21.15.98:0/2534886379 wait complete. 2023-07-14T19:29:23.551 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/9adea648-227b-11ee-9b33-001a4aab830c/mon.a/config 2023-07-14T19:29:23.967 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:23.638+0000 7f03eb05c700 1 -- 172.21.15.98:0/3961503541 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f03e4104fd0 msgr2=0x7f03e410cb50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:29:23.967 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:23.638+0000 7f03eb05c700 1 --2- 172.21.15.98:0/3961503541 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f03e4104fd0 0x7f03e410cb50 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f03dc00b0d0 tx=0x7f03dc00b3e0 comp rx=0 tx=0).stop 2023-07-14T19:29:23.967 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:23.638+0000 7f03eb05c700 1 -- 172.21.15.98:0/3961503541 shutdown_connections 2023-07-14T19:29:23.967 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:23.638+0000 7f03eb05c700 1 --2- 172.21.15.98:0/3961503541 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f03e4104fd0 0x7f03e410cb50 unknown :-1 s=CLOSED pgs=244 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:29:23.967 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:23.638+0000 7f03eb05c700 1 --2- 172.21.15.98:0/3961503541 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f03e40fff30 0x7f03e4100300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:29:23.968 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:23.638+0000 7f03eb05c700 1 --2- 172.21.15.98:0/3961503541 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f03e41008d0 0x7f03e41048a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:29:23.968 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:23.638+0000 7f03eb05c700 1 -- 172.21.15.98:0/3961503541 >> 172.21.15.98:0/3961503541 conn(0x7f03e40fb790 msgr2=0x7f03e40fdba0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:29:23.968 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:23.638+0000 7f03eb05c700 1 -- 172.21.15.98:0/3961503541 shutdown_connections 2023-07-14T19:29:23.968 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:23.638+0000 7f03eb05c700 1 -- 172.21.15.98:0/3961503541 wait complete. 2023-07-14T19:29:23.968 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:23.639+0000 7f03eb05c700 1 Processor -- start 2023-07-14T19:29:23.968 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:23.639+0000 7f03eb05c700 1 -- start start 2023-07-14T19:29:23.969 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:23.639+0000 7f03eb05c700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f03e40fff30 0x7f03e4195e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:29:23.969 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:23.639+0000 7f03eb05c700 1 --2- >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f03e41008d0 0x7f03e41963c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:29:23.969 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:23.639+0000 7f03eb05c700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f03e4104fd0 0x7f03e419a830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:29:23.969 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:23.639+0000 7f03eb05c700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f03e419ae40 con 0x7f03e40fff30 2023-07-14T19:29:23.969 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:23.639+0000 7f03eb05c700 1 -- --> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f03e419afb0 con 0x7f03e41008d0 2023-07-14T19:29:23.969 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:23.639+0000 7f03eb05c700 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f03e419b170 con 0x7f03e4104fd0 2023-07-14T19:29:23.970 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:23.639+0000 7f03e8df8700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f03e40fff30 0x7f03e4195e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:29:23.970 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:23.639+0000 7f03e8df8700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f03e40fff30 0x7f03e4195e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:45946/0 (socket says 172.21.15.98:45946) 2023-07-14T19:29:23.970 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:23.639+0000 7f03e8df8700 1 -- 172.21.15.98:0/2972943878 learned_addr learned my addr 172.21.15.98:0/2972943878 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-07-14T19:29:23.970 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:23.639+0000 7f03e3fff700 1 --2- 172.21.15.98:0/2972943878 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f03e41008d0 0x7f03e41963c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:29:23.970 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:23.639+0000 7f03e8df8700 1 -- 172.21.15.98:0/2972943878 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f03e4104fd0 msgr2=0x7f03e419a830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:29:23.970 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:23.639+0000 7f03e8df8700 1 --2- 172.21.15.98:0/2972943878 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f03e4104fd0 0x7f03e419a830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:29:23.970 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:23.639+0000 7f03e8df8700 1 -- 172.21.15.98:0/2972943878 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f03e41008d0 msgr2=0x7f03e41963c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:29:23.970 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:23.639+0000 7f03e8df8700 1 --2- 172.21.15.98:0/2972943878 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f03e41008d0 0x7f03e41963c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:29:23.971 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:23.639+0000 7f03e8df8700 1 -- 172.21.15.98:0/2972943878 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f03dc009d00 con 0x7f03e40fff30 2023-07-14T19:29:23.971 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:23.639+0000 7f03e3fff700 1 --2- 172.21.15.98:0/2972943878 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f03e41008d0 0x7f03e41963c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-07-14T19:29:23.971 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:23.640+0000 7f03e8df8700 1 --2- 172.21.15.98:0/2972943878 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f03e40fff30 0x7f03e4195e80 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f03d800d950 tx=0x7f03d800dd10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:29:23.971 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:23.640+0000 7f03e1ffb700 1 -- 172.21.15.98:0/2972943878 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f03d8009b40 con 0x7f03e40fff30 2023-07-14T19:29:23.971 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:23.640+0000 7f03eb05c700 1 -- 172.21.15.98:0/2972943878 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f03e419b450 con 0x7f03e40fff30 2023-07-14T19:29:23.971 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:23.640+0000 7f03e1ffb700 1 -- 172.21.15.98:0/2972943878 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f03d8009ca0 con 0x7f03e40fff30 2023-07-14T19:29:23.971 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:23.640+0000 7f03e1ffb700 1 -- 172.21.15.98:0/2972943878 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f03d800f620 con 0x7f03e40fff30 2023-07-14T19:29:23.972 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:23.640+0000 7f03eb05c700 1 -- 172.21.15.98:0/2972943878 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f03e41a2cb0 con 0x7f03e40fff30 2023-07-14T19:29:23.972 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:23.641+0000 7f03eb05c700 1 -- 172.21.15.98:0/2972943878 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f03e404e340 con 0x7f03e40fff30 2023-07-14T19:29:23.972 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:23.642+0000 7f03e1ffb700 1 -- 172.21.15.98:0/2972943878 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 17) v1 ==== 135766+0+0 (secure 0 0 0) 0x7f03d8010460 con 0x7f03e40fff30 2023-07-14T19:29:23.972 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:23.642+0000 7f03e1ffb700 1 --2- 172.21.15.98:0/2972943878 >> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] conn(0x7f03d40a11e0 0x7f03d40a3690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:29:23.972 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:23.643+0000 7f03e3fff700 1 --2- 172.21.15.98:0/2972943878 >> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] conn(0x7f03d40a11e0 0x7f03d40a3690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:29:23.972 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:23.643+0000 7f03e1ffb700 1 -- 172.21.15.98:0/2972943878 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 7979+0+0 (secure 0 0 0) 0x7f03d80ccf30 con 0x7f03e40fff30 2023-07-14T19:29:23.972 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:23.643+0000 7f03e1ffb700 1 --2- 172.21.15.98:0/2972943878 >> [v2:172.21.15.101:6816/2881214066,v1:172.21.15.101:6817/2881214066] conn(0x7f03d40aa9b0 0x7f03d40acdc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:29:23.973 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:23.643+0000 7f03e1ffb700 1 -- 172.21.15.98:0/2972943878 --> [v2:172.21.15.101:6816/2881214066,v1:172.21.15.101:6817/2881214066] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f03d40ad470 con 0x7f03d40aa9b0 2023-07-14T19:29:23.973 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:23.643+0000 7f03e1ffb700 1 -- 172.21.15.98:0/2972943878 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_get_version_reply(handle=1 version=83) v2 ==== 24+0+0 (secure 0 0 0) 0x7f03d80cd2a0 con 0x7f03e40fff30 2023-07-14T19:29:23.973 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:23.643+0000 7f03e3fff700 1 --2- 172.21.15.98:0/2972943878 >> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] conn(0x7f03d40a11e0 0x7f03d40a3690 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f03e41974a0 tx=0x7f03d0009550 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:29:23.973 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:23.643+0000 7f03e95f9700 1 --2- 172.21.15.98:0/2972943878 >> [v2:172.21.15.101:6816/2881214066,v1:172.21.15.101:6817/2881214066] conn(0x7f03d40aa9b0 0x7f03d40acdc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:29:23.973 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:23.643+0000 7f03e95f9700 1 --2- 172.21.15.98:0/2972943878 >> [v2:172.21.15.101:6816/2881214066,v1:172.21.15.101:6817/2881214066] conn(0x7f03d40aa9b0 0x7f03d40acdc0 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:29:23.973 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:23.644+0000 7f03e1ffb700 1 -- 172.21.15.98:0/2972943878 <== osd.7 v2:172.21.15.101:6816/2881214066 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25106 (crc 0 0 0) 0x7f03d40ad470 con 0x7f03d40aa9b0 2023-07-14T19:29:23.973 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:23.662+0000 7f03eb05c700 1 -- 172.21.15.98:0/2972943878 --> [v2:172.21.15.101:6816/2881214066,v1:172.21.15.101:6817/2881214066] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f03e410fe70 con 0x7f03d40aa9b0 2023-07-14T19:29:23.973 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:23.663+0000 7f03e1ffb700 1 -- 172.21.15.98:0/2972943878 <== osd.7 v2:172.21.15.101:6816/2881214066 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+12 (crc 0 0 0) 0x7f03dc005a60 con 0x7f03d40aa9b0 2023-07-14T19:29:23.974 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:23.664+0000 7f03eb05c700 1 -- 172.21.15.98:0/2972943878 >> [v2:172.21.15.101:6816/2881214066,v1:172.21.15.101:6817/2881214066] conn(0x7f03d40aa9b0 msgr2=0x7f03d40acdc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:29:23.974 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:23.664+0000 7f03eb05c700 1 --2- 172.21.15.98:0/2972943878 >> [v2:172.21.15.101:6816/2881214066,v1:172.21.15.101:6817/2881214066] conn(0x7f03d40aa9b0 0x7f03d40acdc0 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:29:23.974 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:23.664+0000 7f03eb05c700 1 -- 172.21.15.98:0/2972943878 >> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] conn(0x7f03d40a11e0 msgr2=0x7f03d40a3690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:29:23.974 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:23.664+0000 7f03eb05c700 1 --2- 172.21.15.98:0/2972943878 >> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] conn(0x7f03d40a11e0 0x7f03d40a3690 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f03e41974a0 tx=0x7f03d0009550 comp rx=0 tx=0).stop 2023-07-14T19:29:23.975 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:23.664+0000 7f03eb05c700 1 -- 172.21.15.98:0/2972943878 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f03e40fff30 msgr2=0x7f03e4195e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:29:23.975 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:23.664+0000 7f03eb05c700 1 --2- 172.21.15.98:0/2972943878 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f03e40fff30 0x7f03e4195e80 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f03d800d950 tx=0x7f03d800dd10 comp rx=0 tx=0).stop 2023-07-14T19:29:23.975 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:23.664+0000 7f03e95f9700 1 -- 172.21.15.98:0/2972943878 reap_dead start 2023-07-14T19:29:23.975 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:23.664+0000 7f03eb05c700 1 -- 172.21.15.98:0/2972943878 shutdown_connections 2023-07-14T19:29:23.975 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:23.664+0000 7f03eb05c700 1 -- 172.21.15.98:0/2972943878 >> 172.21.15.98:0/2972943878 conn(0x7f03e40fb790 msgr2=0x7f03e40fc190 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:29:23.975 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:23.664+0000 7f03eb05c700 1 -- 172.21.15.98:0/2972943878 shutdown_connections 2023-07-14T19:29:23.975 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:23.664+0000 7f03eb05c700 1 -- 172.21.15.98:0/2972943878 wait complete. 2023-07-14T19:29:24.931 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:29:24 smithi098 ceph-mon[127486]: pgmap v264: 97 pgs: 97 active+clean; 633 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 1.8 KiB/s wr, 6 op/s 2023-07-14T19:29:24.941 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:29:24 smithi101 ceph-mon[128807]: pgmap v264: 97 pgs: 97 active+clean; 633 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 1.8 KiB/s wr, 6 op/s 2023-07-14T19:29:24.970 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:29:24 smithi139 ceph-mon[129293]: pgmap v264: 97 pgs: 97 active+clean; 633 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 1.8 KiB/s wr, 6 op/s 2023-07-14T19:29:24.974 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:24.126+0000 7fd1f1bd7700 1 -- 172.21.15.98:0/2452685820 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd1ec0fef40 msgr2=0x7fd1ec0ff310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:29:24.974 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:24.126+0000 7fd1f1bd7700 1 --2- 172.21.15.98:0/2452685820 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd1ec0fef40 0x7fd1ec0ff310 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7fd1dc0099c0 tx=0x7fd1dc009cd0 comp rx=0 tx=0).stop 2023-07-14T19:29:24.975 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:24.126+0000 7fd1f1bd7700 1 -- 172.21.15.98:0/2452685820 shutdown_connections 2023-07-14T19:29:24.975 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:24.126+0000 7fd1f1bd7700 1 --2- 172.21.15.98:0/2452685820 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7fd1ec101150 0x7fd1ec10bb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:29:24.975 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:24.126+0000 7fd1f1bd7700 1 --2- 172.21.15.98:0/2452685820 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd1ec0fef40 0x7fd1ec0ff310 unknown :-1 s=CLOSED pgs=246 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:29:24.975 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:24.126+0000 7fd1f1bd7700 1 --2- 172.21.15.98:0/2452685820 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fd1ec1007a0 0x7fd1ec100c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:29:24.975 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:24.126+0000 7fd1f1bd7700 1 -- 172.21.15.98:0/2452685820 >> 172.21.15.98:0/2452685820 conn(0x7fd1ec075c90 msgr2=0x7fd1ec076090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:29:24.975 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:24.126+0000 7fd1f1bd7700 1 -- 172.21.15.98:0/2452685820 shutdown_connections 2023-07-14T19:29:24.976 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:24.126+0000 7fd1f1bd7700 1 -- 172.21.15.98:0/2452685820 wait complete. 2023-07-14T19:29:24.976 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:24.127+0000 7fd1f1bd7700 1 Processor -- start 2023-07-14T19:29:24.976 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:24.127+0000 7fd1f1bd7700 1 -- start start 2023-07-14T19:29:24.976 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:24.127+0000 7fd1f1bd7700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fd1ec0fef40 0x7fd1ec194ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:29:24.976 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:24.127+0000 7fd1f1bd7700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd1ec1007a0 0x7fd1ec195410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:29:24.976 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:24.127+0000 7fd1f1bd7700 1 --2- >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7fd1ec101150 0x7fd1ec199880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:29:24.976 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:24.127+0000 7fd1f1bd7700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd1ec199e90 con 0x7fd1ec1007a0 2023-07-14T19:29:24.977 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:24.127+0000 7fd1f1bd7700 1 -- --> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd1ec19a000 con 0x7fd1ec101150 2023-07-14T19:29:24.977 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:24.127+0000 7fd1f1bd7700 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd1ec19a1c0 con 0x7fd1ec0fef40 2023-07-14T19:29:24.977 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:24.127+0000 7fd1eaffd700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd1ec1007a0 0x7fd1ec195410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:29:24.977 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:24.127+0000 7fd1eaffd700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd1ec1007a0 0x7fd1ec195410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:45964/0 (socket says 172.21.15.98:45964) 2023-07-14T19:29:24.977 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:24.127+0000 7fd1eaffd700 1 -- 172.21.15.98:0/3716558334 learned_addr learned my addr 172.21.15.98:0/3716558334 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-07-14T19:29:24.977 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:24.127+0000 7fd1ebfff700 1 --2- 172.21.15.98:0/3716558334 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7fd1ec101150 0x7fd1ec199880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:29:24.977 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:24.128+0000 7fd1eb7fe700 1 --2- 172.21.15.98:0/3716558334 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fd1ec0fef40 0x7fd1ec194ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:29:24.978 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:24.128+0000 7fd1eaffd700 1 -- 172.21.15.98:0/3716558334 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fd1ec0fef40 msgr2=0x7fd1ec194ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:29:24.978 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:24.128+0000 7fd1eaffd700 1 --2- 172.21.15.98:0/3716558334 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fd1ec0fef40 0x7fd1ec194ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:29:24.978 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:24.128+0000 7fd1eaffd700 1 -- 172.21.15.98:0/3716558334 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7fd1ec101150 msgr2=0x7fd1ec199880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:29:24.978 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:24.128+0000 7fd1eaffd700 1 --2- 172.21.15.98:0/3716558334 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7fd1ec101150 0x7fd1ec199880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:29:24.978 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:24.128+0000 7fd1eaffd700 1 -- 172.21.15.98:0/3716558334 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd1dc0095f0 con 0x7fd1ec1007a0 2023-07-14T19:29:24.979 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:24.128+0000 7fd1eb7fe700 1 --2- 172.21.15.98:0/3716558334 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fd1ec0fef40 0x7fd1ec194ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-07-14T19:29:24.979 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:24.128+0000 7fd1eaffd700 1 --2- 172.21.15.98:0/3716558334 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd1ec1007a0 0x7fd1ec195410 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7fd1e000d9e0 tx=0x7fd1e000dcf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:29:24.979 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:24.128+0000 7fd1e8ff9700 1 -- 172.21.15.98:0/3716558334 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd1e000ec50 con 0x7fd1ec1007a0 2023-07-14T19:29:24.979 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:24.128+0000 7fd1f1bd7700 1 -- 172.21.15.98:0/3716558334 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd1ec19a4a0 con 0x7fd1ec1007a0 2023-07-14T19:29:24.979 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:24.128+0000 7fd1e8ff9700 1 -- 172.21.15.98:0/3716558334 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fd1e000edb0 con 0x7fd1ec1007a0 2023-07-14T19:29:24.979 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:24.128+0000 7fd1e8ff9700 1 -- 172.21.15.98:0/3716558334 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd1e0014730 con 0x7fd1ec1007a0 2023-07-14T19:29:24.979 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:24.128+0000 7fd1f1bd7700 1 -- 172.21.15.98:0/3716558334 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd1ec068b60 con 0x7fd1ec1007a0 2023-07-14T19:29:24.980 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:24.129+0000 7fd1f1bd7700 1 -- 172.21.15.98:0/3716558334 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fd1ec10ee70 con 0x7fd1ec1007a0 2023-07-14T19:29:24.980 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:24.130+0000 7fd1e8ff9700 1 -- 172.21.15.98:0/3716558334 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 17) v1 ==== 135766+0+0 (secure 0 0 0) 0x7fd1e0016a90 con 0x7fd1ec1007a0 2023-07-14T19:29:24.980 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:24.130+0000 7fd1e8ff9700 1 --2- 172.21.15.98:0/3716558334 >> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] conn(0x7fd1d40a11e0 0x7fd1d40a3690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:29:24.980 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:24.131+0000 7fd1eb7fe700 1 --2- 172.21.15.98:0/3716558334 >> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] conn(0x7fd1d40a11e0 0x7fd1d40a3690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:29:24.981 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:24.131+0000 7fd1e8ff9700 1 -- 172.21.15.98:0/3716558334 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 7979+0+0 (secure 0 0 0) 0x7fd1e00ce940 con 0x7fd1ec1007a0 2023-07-14T19:29:24.981 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:24.131+0000 7fd1e8ff9700 1 --2- 172.21.15.98:0/3716558334 >> [v2:172.21.15.101:6808/2844761406,v1:172.21.15.101:6809/2844761406] conn(0x7fd1d40aa9b0 0x7fd1d40acdc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:29:24.981 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:24.131+0000 7fd1e8ff9700 1 -- 172.21.15.98:0/3716558334 --> [v2:172.21.15.101:6808/2844761406,v1:172.21.15.101:6809/2844761406] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fd1d40ad470 con 0x7fd1d40aa9b0 2023-07-14T19:29:24.981 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:24.131+0000 7fd1e8ff9700 1 -- 172.21.15.98:0/3716558334 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_get_version_reply(handle=1 version=83) v2 ==== 24+0+0 (secure 0 0 0) 0x7fd1e00cecb0 con 0x7fd1ec1007a0 2023-07-14T19:29:24.982 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:24.131+0000 7fd1eb7fe700 1 --2- 172.21.15.98:0/3716558334 >> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] conn(0x7fd1d40a11e0 0x7fd1d40a3690 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7fd1dc006010 tx=0x7fd1dc000bc0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:29:24.982 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:24.131+0000 7fd1ebfff700 1 --2- 172.21.15.98:0/3716558334 >> [v2:172.21.15.101:6808/2844761406,v1:172.21.15.101:6809/2844761406] conn(0x7fd1d40aa9b0 0x7fd1d40acdc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:29:24.982 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:24.131+0000 7fd1ebfff700 1 --2- 172.21.15.98:0/3716558334 >> [v2:172.21.15.101:6808/2844761406,v1:172.21.15.101:6809/2844761406] conn(0x7fd1d40aa9b0 0x7fd1d40acdc0 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:29:24.982 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:24.132+0000 7fd1e8ff9700 1 -- 172.21.15.98:0/3716558334 <== osd.4 v2:172.21.15.101:6808/2844761406 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25106 (crc 0 0 0) 0x7fd1d40ad470 con 0x7fd1d40aa9b0 2023-07-14T19:29:24.982 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:24.149+0000 7fd1f1bd7700 1 -- 172.21.15.98:0/3716558334 --> [v2:172.21.15.101:6808/2844761406,v1:172.21.15.101:6809/2844761406] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7fd1ec04e340 con 0x7fd1d40aa9b0 2023-07-14T19:29:24.982 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:24.150+0000 7fd1e8ff9700 1 -- 172.21.15.98:0/3716558334 <== osd.4 v2:172.21.15.101:6808/2844761406 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+12 (crc 0 0 0) 0x7fd1ec04e340 con 0x7fd1d40aa9b0 2023-07-14T19:29:24.982 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:24.150+0000 7fd1f1bd7700 1 -- 172.21.15.98:0/3716558334 >> [v2:172.21.15.101:6808/2844761406,v1:172.21.15.101:6809/2844761406] conn(0x7fd1d40aa9b0 msgr2=0x7fd1d40acdc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:29:24.983 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:24.150+0000 7fd1f1bd7700 1 --2- 172.21.15.98:0/3716558334 >> [v2:172.21.15.101:6808/2844761406,v1:172.21.15.101:6809/2844761406] conn(0x7fd1d40aa9b0 0x7fd1d40acdc0 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:29:24.983 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:24.150+0000 7fd1f1bd7700 1 -- 172.21.15.98:0/3716558334 >> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] conn(0x7fd1d40a11e0 msgr2=0x7fd1d40a3690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:29:24.983 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:24.150+0000 7fd1f1bd7700 1 --2- 172.21.15.98:0/3716558334 >> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] conn(0x7fd1d40a11e0 0x7fd1d40a3690 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7fd1dc006010 tx=0x7fd1dc000bc0 comp rx=0 tx=0).stop 2023-07-14T19:29:24.983 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:24.150+0000 7fd1f1bd7700 1 -- 172.21.15.98:0/3716558334 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd1ec1007a0 msgr2=0x7fd1ec195410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:29:24.983 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:24.150+0000 7fd1f1bd7700 1 --2- 172.21.15.98:0/3716558334 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd1ec1007a0 0x7fd1ec195410 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7fd1e000d9e0 tx=0x7fd1e000dcf0 comp rx=0 tx=0).stop 2023-07-14T19:29:24.983 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:24.150+0000 7fd1ebfff700 1 -- 172.21.15.98:0/3716558334 reap_dead start 2023-07-14T19:29:24.983 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:24.150+0000 7fd1f1bd7700 1 -- 172.21.15.98:0/3716558334 shutdown_connections 2023-07-14T19:29:24.983 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:24.150+0000 7fd1f1bd7700 1 -- 172.21.15.98:0/3716558334 >> 172.21.15.98:0/3716558334 conn(0x7fd1ec075c90 msgr2=0x7fd1ec0fc4f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:29:24.984 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:24.150+0000 7fd1f1bd7700 1 -- 172.21.15.98:0/3716558334 shutdown_connections 2023-07-14T19:29:24.984 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:24.150+0000 7fd1f1bd7700 1 -- 172.21.15.98:0/3716558334 wait complete. 2023-07-14T19:29:26.220 INFO:teuthology.orchestra.run.smithi098.stdout:34359738430 2023-07-14T19:29:26.220 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81c9387d2e7576da065747ec9c16d1c7d69016ec shell --fsid 9adea648-227b-11ee-9b33-001a4aab830c -- ceph osd last-stat-seq osd.0 2023-07-14T19:29:26.363 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:26.361+0000 7ff7fbd75700 1 -- 172.21.15.98:0/1389983089 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff7f4101830 msgr2=0x7ff7f4109d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:29:26.363 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:26.361+0000 7ff7fbd75700 1 --2- 172.21.15.98:0/1389983089 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff7f4101830 0x7ff7f4109d30 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7ff7e4009990 tx=0x7ff7e4009ca0 comp rx=0 tx=0).stop 2023-07-14T19:29:26.363 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:26.361+0000 7ff7fbd75700 1 -- 172.21.15.98:0/1389983089 shutdown_connections 2023-07-14T19:29:26.364 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:26.361+0000 7ff7fbd75700 1 --2- 172.21.15.98:0/1389983089 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7ff7f4100f20 0x7ff7f41012f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:29:26.364 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:26.361+0000 7ff7fbd75700 1 --2- 172.21.15.98:0/1389983089 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7ff7f410a270 0x7ff7f410c6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:29:26.364 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:26.361+0000 7ff7fbd75700 1 --2- 172.21.15.98:0/1389983089 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff7f4101830 0x7ff7f4109d30 unknown :-1 s=CLOSED pgs=248 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:29:26.364 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:26.361+0000 7ff7fbd75700 1 -- 172.21.15.98:0/1389983089 >> 172.21.15.98:0/1389983089 conn(0x7ff7f4075ca0 msgr2=0x7ff7f40760a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:29:26.364 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:26.361+0000 7ff7fbd75700 1 -- 172.21.15.98:0/1389983089 shutdown_connections 2023-07-14T19:29:26.364 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:26.361+0000 7ff7fbd75700 1 -- 172.21.15.98:0/1389983089 wait complete. 2023-07-14T19:29:26.366 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:26.362+0000 7ff7fbd75700 1 Processor -- start 2023-07-14T19:29:26.366 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:26.362+0000 7ff7fbd75700 1 -- start start 2023-07-14T19:29:26.366 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:26.362+0000 7ff7fbd75700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7ff7f4100f20 0x7ff7f40feb80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:29:26.366 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:26.362+0000 7ff7fbd75700 1 --2- >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7ff7f4101830 0x7ff7f40ff0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:29:26.367 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:26.362+0000 7ff7fbd75700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff7f410a270 0x7ff7f40ff640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:29:26.367 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:26.362+0000 7ff7fbd75700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff7f41a4240 con 0x7ff7f410a270 2023-07-14T19:29:26.367 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:26.362+0000 7ff7fbd75700 1 -- --> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff7f41a43b0 con 0x7ff7f4101830 2023-07-14T19:29:26.367 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:26.362+0000 7ff7fbd75700 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff7f41a4570 con 0x7ff7f4100f20 2023-07-14T19:29:26.367 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:26.362+0000 7ff7fa312700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff7f410a270 0x7ff7f40ff640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:29:26.367 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:26.362+0000 7ff7fa312700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff7f410a270 0x7ff7f40ff640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:45980/0 (socket says 172.21.15.98:45980) 2023-07-14T19:29:26.367 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:26.362+0000 7ff7fa312700 1 -- 172.21.15.98:0/3264723390 learned_addr learned my addr 172.21.15.98:0/3264723390 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-07-14T19:29:26.368 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:26.362+0000 7ff7f9310700 1 --2- 172.21.15.98:0/3264723390 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7ff7f4101830 0x7ff7f40ff0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:29:26.368 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:26.362+0000 7ff7f9b11700 1 --2- 172.21.15.98:0/3264723390 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7ff7f4100f20 0x7ff7f40feb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:29:26.368 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:26.362+0000 7ff7fa312700 1 -- 172.21.15.98:0/3264723390 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7ff7f4100f20 msgr2=0x7ff7f40feb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:29:26.368 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:26.363+0000 7ff7fa312700 1 --2- 172.21.15.98:0/3264723390 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7ff7f4100f20 0x7ff7f40feb80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:29:26.368 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:26.363+0000 7ff7fa312700 1 -- 172.21.15.98:0/3264723390 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7ff7f4101830 msgr2=0x7ff7f40ff0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:29:26.368 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:26.363+0000 7ff7fa312700 1 --2- 172.21.15.98:0/3264723390 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7ff7f4101830 0x7ff7f40ff0c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:29:26.368 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:26.363+0000 7ff7fa312700 1 -- 172.21.15.98:0/3264723390 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff7e40095f0 con 0x7ff7f410a270 2023-07-14T19:29:26.369 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:26.363+0000 7ff7fa312700 1 --2- 172.21.15.98:0/3264723390 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff7f410a270 0x7ff7f40ff640 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7ff7ec00ab50 tx=0x7ff7ec00ae60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:29:26.369 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:26.363+0000 7ff7eaffd700 1 -- 172.21.15.98:0/3264723390 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff7ec00f070 con 0x7ff7f410a270 2023-07-14T19:29:26.369 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:26.363+0000 7ff7fbd75700 1 -- 172.21.15.98:0/3264723390 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff7f41a4850 con 0x7ff7f410a270 2023-07-14T19:29:26.369 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:26.363+0000 7ff7eaffd700 1 -- 172.21.15.98:0/3264723390 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7ff7ec004bb0 con 0x7ff7f410a270 2023-07-14T19:29:26.369 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:26.363+0000 7ff7eaffd700 1 -- 172.21.15.98:0/3264723390 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff7ec005390 con 0x7ff7f410a270 2023-07-14T19:29:26.369 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:26.363+0000 7ff7fbd75700 1 -- 172.21.15.98:0/3264723390 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff7f41a4d00 con 0x7ff7f410a270 2023-07-14T19:29:26.369 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:26.364+0000 7ff7fbd75700 1 -- 172.21.15.98:0/3264723390 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7ff7f404e340 con 0x7ff7f410a270 2023-07-14T19:29:26.370 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:26.365+0000 7ff7eaffd700 1 -- 172.21.15.98:0/3264723390 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 17) v1 ==== 135766+0+0 (secure 0 0 0) 0x7ff7ec004d20 con 0x7ff7f410a270 2023-07-14T19:29:26.370 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:26.365+0000 7ff7eaffd700 1 --2- 172.21.15.98:0/3264723390 >> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] conn(0x7ff7e00a11e0 0x7ff7e00a3690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:29:26.371 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:26.366+0000 7ff7f9b11700 1 --2- 172.21.15.98:0/3264723390 >> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] conn(0x7ff7e00a11e0 0x7ff7e00a3690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:29:26.371 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:26.366+0000 7ff7eaffd700 1 -- 172.21.15.98:0/3264723390 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 7979+0+0 (secure 0 0 0) 0x7ff7ec0cc790 con 0x7ff7f410a270 2023-07-14T19:29:26.371 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:26.366+0000 7ff7eaffd700 1 --2- 172.21.15.98:0/3264723390 >> [v2:172.21.15.98:6810/3095249665,v1:172.21.15.98:6811/3095249665] conn(0x7ff7e00aa9b0 0x7ff7e00acdc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:29:26.371 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:26.366+0000 7ff7eaffd700 1 -- 172.21.15.98:0/3264723390 --> [v2:172.21.15.98:6810/3095249665,v1:172.21.15.98:6811/3095249665] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7ff7e00ad470 con 0x7ff7e00aa9b0 2023-07-14T19:29:26.371 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:26.366+0000 7ff7eaffd700 1 -- 172.21.15.98:0/3264723390 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_get_version_reply(handle=1 version=83) v2 ==== 24+0+0 (secure 0 0 0) 0x7ff7ec0ccc30 con 0x7ff7f410a270 2023-07-14T19:29:26.371 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:26.366+0000 7ff7f9310700 1 --2- 172.21.15.98:0/3264723390 >> [v2:172.21.15.98:6810/3095249665,v1:172.21.15.98:6811/3095249665] conn(0x7ff7e00aa9b0 0x7ff7e00acdc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:29:26.371 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:26.366+0000 7ff7f9b11700 1 --2- 172.21.15.98:0/3264723390 >> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] conn(0x7ff7e00a11e0 0x7ff7e00a3690 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7ff7f0005950 tx=0x7ff7f000b410 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:29:26.372 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:26.366+0000 7ff7f9310700 1 --2- 172.21.15.98:0/3264723390 >> [v2:172.21.15.98:6810/3095249665,v1:172.21.15.98:6811/3095249665] conn(0x7ff7e00aa9b0 0x7ff7e00acdc0 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:29:26.372 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:26.367+0000 7ff7eaffd700 1 -- 172.21.15.98:0/3264723390 <== osd.3 v2:172.21.15.98:6810/3095249665 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25106 (crc 0 0 0) 0x7ff7e00ad470 con 0x7ff7e00aa9b0 2023-07-14T19:29:26.384 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:26.382+0000 7ff7fbd75700 1 -- 172.21.15.98:0/3264723390 --> [v2:172.21.15.98:6810/3095249665,v1:172.21.15.98:6811/3095249665] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7ff7f410fea0 con 0x7ff7e00aa9b0 2023-07-14T19:29:26.385 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:26.383+0000 7ff7eaffd700 1 -- 172.21.15.98:0/3264723390 <== osd.3 v2:172.21.15.98:6810/3095249665 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+12 (crc 0 0 0) 0x7ff7e401e040 con 0x7ff7e00aa9b0 2023-07-14T19:29:26.386 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:26.383+0000 7ff7fbd75700 1 -- 172.21.15.98:0/3264723390 >> [v2:172.21.15.98:6810/3095249665,v1:172.21.15.98:6811/3095249665] conn(0x7ff7e00aa9b0 msgr2=0x7ff7e00acdc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:29:26.386 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:26.383+0000 7ff7fbd75700 1 --2- 172.21.15.98:0/3264723390 >> [v2:172.21.15.98:6810/3095249665,v1:172.21.15.98:6811/3095249665] conn(0x7ff7e00aa9b0 0x7ff7e00acdc0 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:29:26.386 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:26.383+0000 7ff7fbd75700 1 -- 172.21.15.98:0/3264723390 >> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] conn(0x7ff7e00a11e0 msgr2=0x7ff7e00a3690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:29:26.386 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:26.383+0000 7ff7fbd75700 1 --2- 172.21.15.98:0/3264723390 >> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] conn(0x7ff7e00a11e0 0x7ff7e00a3690 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7ff7f0005950 tx=0x7ff7f000b410 comp rx=0 tx=0).stop 2023-07-14T19:29:26.386 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:26.383+0000 7ff7fbd75700 1 -- 172.21.15.98:0/3264723390 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff7f410a270 msgr2=0x7ff7f40ff640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:29:26.386 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:26.383+0000 7ff7fbd75700 1 --2- 172.21.15.98:0/3264723390 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff7f410a270 0x7ff7f40ff640 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7ff7ec00ab50 tx=0x7ff7ec00ae60 comp rx=0 tx=0).stop 2023-07-14T19:29:26.387 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:26.384+0000 7ff7fa312700 1 -- 172.21.15.98:0/3264723390 reap_dead start 2023-07-14T19:29:26.387 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:26.384+0000 7ff7fbd75700 1 -- 172.21.15.98:0/3264723390 shutdown_connections 2023-07-14T19:29:26.387 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:26.384+0000 7ff7fbd75700 1 -- 172.21.15.98:0/3264723390 >> 172.21.15.98:0/3264723390 conn(0x7ff7f4075ca0 msgr2=0x7ff7f4101c40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:29:26.387 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:26.384+0000 7ff7fbd75700 1 -- 172.21.15.98:0/3264723390 shutdown_connections 2023-07-14T19:29:26.387 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:26.384+0000 7ff7fbd75700 1 -- 172.21.15.98:0/3264723390 wait complete. 2023-07-14T19:29:26.529 INFO:teuthology.orchestra.run.smithi098.stdout:137438953517 2023-07-14T19:29:26.530 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81c9387d2e7576da065747ec9c16d1c7d69016ec shell --fsid 9adea648-227b-11ee-9b33-001a4aab830c -- ceph osd last-stat-seq osd.4 2023-07-14T19:29:26.767 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:29:26 smithi098 ceph-mon[127486]: pgmap v265: 97 pgs: 97 active+clean; 633 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 1.7 KiB/s wr, 7 op/s 2023-07-14T19:29:26.941 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:29:26 smithi101 ceph-mon[128807]: pgmap v265: 97 pgs: 97 active+clean; 633 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 1.7 KiB/s wr, 7 op/s 2023-07-14T19:29:26.970 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:29:26 smithi139 ceph-mon[129293]: pgmap v265: 97 pgs: 97 active+clean; 633 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 1.7 KiB/s wr, 7 op/s 2023-07-14T19:29:27.731 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:27.727+0000 7f96327ef700 1 -- 172.21.15.98:0/264881277 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f962c103ee0 msgr2=0x7f962c10ba60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:29:27.731 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:27.727+0000 7f96327ef700 1 --2- 172.21.15.98:0/264881277 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f962c103ee0 0x7f962c10ba60 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f962400b0d0 tx=0x7f962400b3e0 comp rx=0 tx=0).stop 2023-07-14T19:29:27.731 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:27.727+0000 7f96327ef700 1 -- 172.21.15.98:0/264881277 shutdown_connections 2023-07-14T19:29:27.731 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:27.727+0000 7f96327ef700 1 --2- 172.21.15.98:0/264881277 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f962c0fee40 0x7f962c0ff210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:29:27.731 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:27.727+0000 7f96327ef700 1 --2- 172.21.15.98:0/264881277 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f962c103ee0 0x7f962c10ba60 unknown :-1 s=CLOSED pgs=250 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:29:27.732 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:27.727+0000 7f96327ef700 1 --2- 172.21.15.98:0/264881277 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f962c0ff7e0 0x7f962c1037b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:29:27.732 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:27.727+0000 7f96327ef700 1 -- 172.21.15.98:0/264881277 >> 172.21.15.98:0/264881277 conn(0x7f962c075ca0 msgr2=0x7f962c0760a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:29:27.732 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:27.727+0000 7f96327ef700 1 -- 172.21.15.98:0/264881277 shutdown_connections 2023-07-14T19:29:27.732 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:27.727+0000 7f96327ef700 1 -- 172.21.15.98:0/264881277 wait complete. 2023-07-14T19:29:27.732 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:27.728+0000 7f96327ef700 1 Processor -- start 2023-07-14T19:29:27.732 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:27.728+0000 7f96327ef700 1 -- start start 2023-07-14T19:29:27.732 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:27.728+0000 7f96327ef700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f962c0fee40 0x7f962c195f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:29:27.733 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:27.728+0000 7f96327ef700 1 --2- >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f962c0ff7e0 0x7f962c196450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:29:27.733 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:27.728+0000 7f96327ef700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f962c103ee0 0x7f962c19a8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:29:27.733 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:27.728+0000 7f96327ef700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f962c19aed0 con 0x7f962c0fee40 2023-07-14T19:29:27.734 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:27.728+0000 7f96327ef700 1 -- --> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f962c19b040 con 0x7f962c0ff7e0 2023-07-14T19:29:27.734 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:27.728+0000 7f96327ef700 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f962c19b200 con 0x7f962c103ee0 2023-07-14T19:29:27.734 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:27.728+0000 7f962bfff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f962c0fee40 0x7f962c195f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:29:27.735 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:27.728+0000 7f962bfff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f962c0fee40 0x7f962c195f10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:45992/0 (socket says 172.21.15.98:45992) 2023-07-14T19:29:27.735 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:27.728+0000 7f962bfff700 1 -- 172.21.15.98:0/2828100699 learned_addr learned my addr 172.21.15.98:0/2828100699 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-07-14T19:29:27.735 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:27.729+0000 7f9630d8c700 1 --2- 172.21.15.98:0/2828100699 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f962c103ee0 0x7f962c19a8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:29:27.735 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:27.729+0000 7f962b7fe700 1 --2- 172.21.15.98:0/2828100699 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f962c0ff7e0 0x7f962c196450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:29:27.736 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:27.729+0000 7f962bfff700 1 -- 172.21.15.98:0/2828100699 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f962c103ee0 msgr2=0x7f962c19a8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:29:27.736 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:27.729+0000 7f962bfff700 1 --2- 172.21.15.98:0/2828100699 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f962c103ee0 0x7f962c19a8c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:29:27.736 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:27.729+0000 7f962bfff700 1 -- 172.21.15.98:0/2828100699 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f962c0ff7e0 msgr2=0x7f962c196450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:29:27.736 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:27.729+0000 7f962bfff700 1 --2- 172.21.15.98:0/2828100699 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f962c0ff7e0 0x7f962c196450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:29:27.736 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:27.729+0000 7f962bfff700 1 -- 172.21.15.98:0/2828100699 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9624009d00 con 0x7f962c0fee40 2023-07-14T19:29:27.737 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:27.729+0000 7f9630d8c700 1 --2- 172.21.15.98:0/2828100699 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f962c103ee0 0x7f962c19a8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-07-14T19:29:27.737 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:27.729+0000 7f962bfff700 1 --2- 172.21.15.98:0/2828100699 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f962c0fee40 0x7f962c195f10 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f961400d950 tx=0x7f961400dd10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:29:27.737 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:27.729+0000 7f96297fa700 1 -- 172.21.15.98:0/2828100699 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9614009b40 con 0x7f962c0fee40 2023-07-14T19:29:27.737 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:27.729+0000 7f96327ef700 1 -- 172.21.15.98:0/2828100699 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f962c19b4e0 con 0x7f962c0fee40 2023-07-14T19:29:27.738 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:27.729+0000 7f96297fa700 1 -- 172.21.15.98:0/2828100699 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f9614009ca0 con 0x7f962c0fee40 2023-07-14T19:29:27.738 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:27.729+0000 7f96297fa700 1 -- 172.21.15.98:0/2828100699 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f961400f620 con 0x7f962c0fee40 2023-07-14T19:29:27.738 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:27.729+0000 7f96327ef700 1 -- 172.21.15.98:0/2828100699 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f962c1a2d40 con 0x7f962c0fee40 2023-07-14T19:29:27.740 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:27.730+0000 7f96327ef700 1 -- 172.21.15.98:0/2828100699 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f962c10ed40 con 0x7f962c0fee40 2023-07-14T19:29:27.741 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:27.731+0000 7f96297fa700 1 -- 172.21.15.98:0/2828100699 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 17) v1 ==== 135766+0+0 (secure 0 0 0) 0x7f9614010460 con 0x7f962c0fee40 2023-07-14T19:29:27.741 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:27.731+0000 7f96297fa700 1 --2- 172.21.15.98:0/2828100699 >> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] conn(0x7f96180a11e0 0x7f96180a3690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:29:27.741 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:27.732+0000 7f962b7fe700 1 --2- 172.21.15.98:0/2828100699 >> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] conn(0x7f96180a11e0 0x7f96180a3690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:29:27.741 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:27.732+0000 7f96297fa700 1 -- 172.21.15.98:0/2828100699 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 7979+0+0 (secure 0 0 0) 0x7f96140ccdd0 con 0x7f962c0fee40 2023-07-14T19:29:27.742 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:27.732+0000 7f96297fa700 1 --2- 172.21.15.98:0/2828100699 >> [v2:172.21.15.139:6816/3669514524,v1:172.21.15.139:6817/3669514524] conn(0x7f96180aa9a0 0x7f96180acdb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:29:27.742 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:27.732+0000 7f96297fa700 1 -- 172.21.15.98:0/2828100699 --> [v2:172.21.15.139:6816/3669514524,v1:172.21.15.139:6817/3669514524] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f96180ad460 con 0x7f96180aa9a0 2023-07-14T19:29:27.742 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:27.732+0000 7f96297fa700 1 -- 172.21.15.98:0/2828100699 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_get_version_reply(handle=1 version=83) v2 ==== 24+0+0 (secure 0 0 0) 0x7f96140cd270 con 0x7f962c0fee40 2023-07-14T19:29:27.743 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:27.732+0000 7f962b7fe700 1 --2- 172.21.15.98:0/2828100699 >> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] conn(0x7f96180a11e0 0x7f96180a3690 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f961c005fd0 tx=0x7f961c005db0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:29:27.743 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:27.732+0000 7f9630d8c700 1 --2- 172.21.15.98:0/2828100699 >> [v2:172.21.15.139:6816/3669514524,v1:172.21.15.139:6817/3669514524] conn(0x7f96180aa9a0 0x7f96180acdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:29:27.743 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:27.733+0000 7f9630d8c700 1 --2- 172.21.15.98:0/2828100699 >> [v2:172.21.15.139:6816/3669514524,v1:172.21.15.139:6817/3669514524] conn(0x7f96180aa9a0 0x7f96180acdb0 crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.8 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:29:27.743 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:27.734+0000 7f96297fa700 1 -- 172.21.15.98:0/2828100699 <== osd.8 v2:172.21.15.139:6816/3669514524 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25106 (crc 0 0 0) 0x7f96180ad460 con 0x7f96180aa9a0 2023-07-14T19:29:27.773 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:27.749+0000 7f96327ef700 1 -- 172.21.15.98:0/2828100699 --> [v2:172.21.15.139:6816/3669514524,v1:172.21.15.139:6817/3669514524] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f962c04e340 con 0x7f96180aa9a0 2023-07-14T19:29:27.774 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:27.750+0000 7f96297fa700 1 -- 172.21.15.98:0/2828100699 <== osd.8 v2:172.21.15.139:6816/3669514524 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+12 (crc 0 0 0) 0x7f962c04e340 con 0x7f96180aa9a0 2023-07-14T19:29:27.774 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:27.750+0000 7f96327ef700 1 -- 172.21.15.98:0/2828100699 >> [v2:172.21.15.139:6816/3669514524,v1:172.21.15.139:6817/3669514524] conn(0x7f96180aa9a0 msgr2=0x7f96180acdb0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:29:27.775 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:27.750+0000 7f96327ef700 1 --2- 172.21.15.98:0/2828100699 >> [v2:172.21.15.139:6816/3669514524,v1:172.21.15.139:6817/3669514524] conn(0x7f96180aa9a0 0x7f96180acdb0 crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:29:27.775 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:27.750+0000 7f96327ef700 1 -- 172.21.15.98:0/2828100699 >> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] conn(0x7f96180a11e0 msgr2=0x7f96180a3690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:29:27.775 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:27.750+0000 7f96327ef700 1 --2- 172.21.15.98:0/2828100699 >> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] conn(0x7f96180a11e0 0x7f96180a3690 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f961c005fd0 tx=0x7f961c005db0 comp rx=0 tx=0).stop 2023-07-14T19:29:27.775 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:27.750+0000 7f96327ef700 1 -- 172.21.15.98:0/2828100699 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f962c0fee40 msgr2=0x7f962c195f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:29:27.776 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:27.750+0000 7f96327ef700 1 --2- 172.21.15.98:0/2828100699 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f962c0fee40 0x7f962c195f10 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f961400d950 tx=0x7f961400dd10 comp rx=0 tx=0).stop 2023-07-14T19:29:27.776 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:27.750+0000 7f9630d8c700 1 -- 172.21.15.98:0/2828100699 reap_dead start 2023-07-14T19:29:27.776 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:27.750+0000 7f96327ef700 1 -- 172.21.15.98:0/2828100699 shutdown_connections 2023-07-14T19:29:27.777 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:27.750+0000 7f96327ef700 1 -- 172.21.15.98:0/2828100699 >> 172.21.15.98:0/2828100699 conn(0x7f962c075ca0 msgr2=0x7f962c102990 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:29:27.777 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:27.750+0000 7f96327ef700 1 -- 172.21.15.98:0/2828100699 shutdown_connections 2023-07-14T19:29:27.777 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:27.750+0000 7f96327ef700 1 -- 172.21.15.98:0/2828100699 wait complete. 2023-07-14T19:29:27.825 INFO:teuthology.orchestra.run.smithi098.stdout:214748364832 2023-07-14T19:29:27.825 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81c9387d2e7576da065747ec9c16d1c7d69016ec shell --fsid 9adea648-227b-11ee-9b33-001a4aab830c -- ceph osd last-stat-seq osd.7 2023-07-14T19:29:28.341 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:27.918+0000 7fefa332a700 1 -- 172.21.15.98:0/3903472715 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fef9c101150 msgr2=0x7fef9c10bb40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:29:28.341 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:27.918+0000 7fefa332a700 1 --2- 172.21.15.98:0/3903472715 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fef9c101150 0x7fef9c10bb40 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7fef9400b0d0 tx=0x7fef9400b3e0 comp rx=0 tx=0).stop 2023-07-14T19:29:28.341 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:27.918+0000 7fefa332a700 1 -- 172.21.15.98:0/3903472715 shutdown_connections 2023-07-14T19:29:28.341 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:27.918+0000 7fefa332a700 1 --2- 172.21.15.98:0/3903472715 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fef9c101150 0x7fef9c10bb40 unknown :-1 s=CLOSED pgs=252 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:29:28.341 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:27.918+0000 7fefa332a700 1 --2- 172.21.15.98:0/3903472715 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7fef9c0fef40 0x7fef9c0ff310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:29:28.342 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:27.918+0000 7fefa332a700 1 --2- 172.21.15.98:0/3903472715 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fef9c1007a0 0x7fef9c100c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:29:28.342 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:27.918+0000 7fefa332a700 1 -- 172.21.15.98:0/3903472715 >> 172.21.15.98:0/3903472715 conn(0x7fef9c075c90 msgr2=0x7fef9c076090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:29:28.342 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:27.918+0000 7fefa332a700 1 -- 172.21.15.98:0/3903472715 shutdown_connections 2023-07-14T19:29:28.342 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:27.918+0000 7fefa332a700 1 -- 172.21.15.98:0/3903472715 wait complete. 2023-07-14T19:29:28.342 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:27.919+0000 7fefa332a700 1 Processor -- start 2023-07-14T19:29:28.342 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:27.919+0000 7fefa332a700 1 -- start start 2023-07-14T19:29:28.342 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:27.919+0000 7fefa332a700 1 --2- >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7fef9c0fef40 0x7fef9c195f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:29:28.343 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:27.919+0000 7fefa332a700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fef9c1007a0 0x7fef9c196490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:29:28.343 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:27.919+0000 7fefa332a700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fef9c101150 0x7fef9c19a900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:29:28.343 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:27.919+0000 7fefa332a700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fef9c19af10 con 0x7fef9c101150 2023-07-14T19:29:28.343 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:27.919+0000 7fefa332a700 1 -- --> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fef9c19b080 con 0x7fef9c0fef40 2023-07-14T19:29:28.343 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:27.919+0000 7fefa332a700 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fef9c19b240 con 0x7fef9c1007a0 2023-07-14T19:29:28.343 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:27.919+0000 7fefa18c7700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fef9c101150 0x7fef9c19a900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:29:28.343 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:27.919+0000 7fefa18c7700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fef9c101150 0x7fef9c19a900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:46012/0 (socket says 172.21.15.98:46012) 2023-07-14T19:29:28.343 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:27.919+0000 7fefa18c7700 1 -- 172.21.15.98:0/398224155 learned_addr learned my addr 172.21.15.98:0/398224155 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-07-14T19:29:28.344 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:27.919+0000 7fefa10c6700 1 --2- 172.21.15.98:0/398224155 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7fef9c0fef40 0x7fef9c195f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:29:28.344 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:27.919+0000 7fefa08c5700 1 --2- 172.21.15.98:0/398224155 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fef9c1007a0 0x7fef9c196490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:29:28.344 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:27.919+0000 7fefa18c7700 1 -- 172.21.15.98:0/398224155 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fef9c1007a0 msgr2=0x7fef9c196490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:29:28.344 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:27.919+0000 7fefa18c7700 1 --2- 172.21.15.98:0/398224155 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fef9c1007a0 0x7fef9c196490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:29:28.344 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:27.919+0000 7fefa18c7700 1 -- 172.21.15.98:0/398224155 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7fef9c0fef40 msgr2=0x7fef9c195f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:29:28.344 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:27.919+0000 7fefa18c7700 1 --2- 172.21.15.98:0/398224155 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7fef9c0fef40 0x7fef9c195f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:29:28.345 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:27.919+0000 7fefa18c7700 1 -- 172.21.15.98:0/398224155 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fef94009d00 con 0x7fef9c101150 2023-07-14T19:29:28.345 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:27.919+0000 7fefa10c6700 1 --2- 172.21.15.98:0/398224155 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7fef9c0fef40 0x7fef9c195f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-07-14T19:29:28.345 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:27.919+0000 7fefa08c5700 1 --2- 172.21.15.98:0/398224155 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fef9c1007a0 0x7fef9c196490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-07-14T19:29:28.345 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:27.919+0000 7fefa18c7700 1 --2- 172.21.15.98:0/398224155 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fef9c101150 0x7fef9c19a900 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7fef94007870 tx=0x7fef94007950 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:29:28.345 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:27.920+0000 7fef927fc700 1 -- 172.21.15.98:0/398224155 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fef9400e040 con 0x7fef9c101150 2023-07-14T19:29:28.345 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:27.920+0000 7fef927fc700 1 -- 172.21.15.98:0/398224155 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fef94005070 con 0x7fef9c101150 2023-07-14T19:29:28.345 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:27.920+0000 7fefa332a700 1 -- 172.21.15.98:0/398224155 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fef9c19b4c0 con 0x7fef9c101150 2023-07-14T19:29:28.346 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:27.920+0000 7fef927fc700 1 -- 172.21.15.98:0/398224155 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fef9401b6c0 con 0x7fef9c101150 2023-07-14T19:29:28.346 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:27.920+0000 7fefa332a700 1 -- 172.21.15.98:0/398224155 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fef9c1a2bb0 con 0x7fef9c101150 2023-07-14T19:29:28.346 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:27.921+0000 7fefa332a700 1 -- 172.21.15.98:0/398224155 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fef9c04e340 con 0x7fef9c101150 2023-07-14T19:29:28.346 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:27.922+0000 7fef927fc700 1 -- 172.21.15.98:0/398224155 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 17) v1 ==== 135766+0+0 (secure 0 0 0) 0x7fef940051e0 con 0x7fef9c101150 2023-07-14T19:29:28.346 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:27.922+0000 7fef927fc700 1 --2- 172.21.15.98:0/398224155 >> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] conn(0x7fef880a1260 0x7fef880a3710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:29:28.347 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:27.923+0000 7fefa10c6700 1 --2- 172.21.15.98:0/398224155 >> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] conn(0x7fef880a1260 0x7fef880a3710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:29:28.347 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:27.923+0000 7fef927fc700 1 -- 172.21.15.98:0/398224155 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 7979+0+0 (secure 0 0 0) 0x7fef940ce220 con 0x7fef9c101150 2023-07-14T19:29:28.347 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:27.923+0000 7fef927fc700 1 --2- 172.21.15.98:0/398224155 >> [v2:172.21.15.139:6824/2630023454,v1:172.21.15.139:6825/2630023454] conn(0x7fef880aa900 0x7fef880acd10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:29:28.347 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:27.923+0000 7fef927fc700 1 -- 172.21.15.98:0/398224155 --> [v2:172.21.15.139:6824/2630023454,v1:172.21.15.139:6825/2630023454] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fef880ad3c0 con 0x7fef880aa900 2023-07-14T19:29:28.348 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:27.923+0000 7fef927fc700 1 -- 172.21.15.98:0/398224155 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_get_version_reply(handle=1 version=83) v2 ==== 24+0+0 (secure 0 0 0) 0x7fef940ce5d0 con 0x7fef9c101150 2023-07-14T19:29:28.348 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:27.923+0000 7fefa10c6700 1 --2- 172.21.15.98:0/398224155 >> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] conn(0x7fef880a1260 0x7fef880a3710 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7fef98009cb0 tx=0x7fef98009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:29:28.348 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:27.923+0000 7fefa08c5700 1 --2- 172.21.15.98:0/398224155 >> [v2:172.21.15.139:6824/2630023454,v1:172.21.15.139:6825/2630023454] conn(0x7fef880aa900 0x7fef880acd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:29:28.348 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:27.924+0000 7fefa08c5700 1 --2- 172.21.15.98:0/398224155 >> [v2:172.21.15.139:6824/2630023454,v1:172.21.15.139:6825/2630023454] conn(0x7fef880aa900 0x7fef880acd10 crc :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.11 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:29:28.348 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:27.924+0000 7fef927fc700 1 -- 172.21.15.98:0/398224155 <== osd.11 v2:172.21.15.139:6824/2630023454 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25106 (crc 0 0 0) 0x7fef880ad3c0 con 0x7fef880aa900 2023-07-14T19:29:28.348 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:27.941+0000 7fefa332a700 1 -- 172.21.15.98:0/398224155 --> [v2:172.21.15.139:6824/2630023454,v1:172.21.15.139:6825/2630023454] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7fef9c110000 con 0x7fef880aa900 2023-07-14T19:29:28.349 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:27.942+0000 7fef927fc700 1 -- 172.21.15.98:0/398224155 <== osd.11 v2:172.21.15.139:6824/2630023454 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+12 (crc 0 0 0) 0x7fef8c0099c0 con 0x7fef880aa900 2023-07-14T19:29:28.349 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:27.942+0000 7fefa332a700 1 -- 172.21.15.98:0/398224155 >> [v2:172.21.15.139:6824/2630023454,v1:172.21.15.139:6825/2630023454] conn(0x7fef880aa900 msgr2=0x7fef880acd10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:29:28.349 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:27.942+0000 7fefa332a700 1 --2- 172.21.15.98:0/398224155 >> [v2:172.21.15.139:6824/2630023454,v1:172.21.15.139:6825/2630023454] conn(0x7fef880aa900 0x7fef880acd10 crc :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:29:28.349 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:27.942+0000 7fefa332a700 1 -- 172.21.15.98:0/398224155 >> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] conn(0x7fef880a1260 msgr2=0x7fef880a3710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:29:28.349 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:27.942+0000 7fefa332a700 1 --2- 172.21.15.98:0/398224155 >> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] conn(0x7fef880a1260 0x7fef880a3710 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7fef98009cb0 tx=0x7fef98009260 comp rx=0 tx=0).stop 2023-07-14T19:29:28.349 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:27.942+0000 7fefa332a700 1 -- 172.21.15.98:0/398224155 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fef9c101150 msgr2=0x7fef9c19a900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:29:28.349 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:27.942+0000 7fefa332a700 1 --2- 172.21.15.98:0/398224155 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fef9c101150 0x7fef9c19a900 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7fef94007870 tx=0x7fef94007950 comp rx=0 tx=0).stop 2023-07-14T19:29:28.349 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:27.942+0000 7fefa18c7700 1 -- 172.21.15.98:0/398224155 reap_dead start 2023-07-14T19:29:28.350 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:27.942+0000 7fefa332a700 1 -- 172.21.15.98:0/398224155 shutdown_connections 2023-07-14T19:29:28.350 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:27.942+0000 7fefa332a700 1 -- 172.21.15.98:0/398224155 >> 172.21.15.98:0/398224155 conn(0x7fef9c075c90 msgr2=0x7fef9c0fc4f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:29:28.350 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:27.942+0000 7fefa332a700 1 -- 172.21.15.98:0/398224155 shutdown_connections 2023-07-14T19:29:28.350 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:27.942+0000 7fefa332a700 1 -- 172.21.15.98:0/398224155 wait complete. 2023-07-14T19:29:28.407 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:28.186+0000 7fe9b5f62700 1 -- 172.21.15.98:0/567613702 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe9b00ff7e0 msgr2=0x7fe9b01037b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:29:28.408 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:28.186+0000 7fe9b5f62700 1 --2- 172.21.15.98:0/567613702 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe9b00ff7e0 0x7fe9b01037b0 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7fe9a40099c0 tx=0x7fe9a4009cd0 comp rx=0 tx=0).stop 2023-07-14T19:29:28.408 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:28.186+0000 7fe9b5f62700 1 -- 172.21.15.98:0/567613702 shutdown_connections 2023-07-14T19:29:28.408 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:28.186+0000 7fe9b5f62700 1 --2- 172.21.15.98:0/567613702 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fe9b00fee40 0x7fe9b00ff210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:29:28.408 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:28.186+0000 7fe9b5f62700 1 --2- 172.21.15.98:0/567613702 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7fe9b0103ee0 0x7fe9b010ba60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:29:28.408 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:28.186+0000 7fe9b5f62700 1 --2- 172.21.15.98:0/567613702 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe9b00ff7e0 0x7fe9b01037b0 unknown :-1 s=CLOSED pgs=254 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:29:28.408 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:28.186+0000 7fe9b5f62700 1 -- 172.21.15.98:0/567613702 >> 172.21.15.98:0/567613702 conn(0x7fe9b0075ca0 msgr2=0x7fe9b00760a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:29:28.409 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:28.186+0000 7fe9b5f62700 1 -- 172.21.15.98:0/567613702 shutdown_connections 2023-07-14T19:29:28.409 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:28.186+0000 7fe9b5f62700 1 -- 172.21.15.98:0/567613702 wait complete. 2023-07-14T19:29:28.409 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:28.187+0000 7fe9b5f62700 1 Processor -- start 2023-07-14T19:29:28.409 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:28.187+0000 7fe9b5f62700 1 -- start start 2023-07-14T19:29:28.409 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:28.187+0000 7fe9b5f62700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe9b00fee40 0x7fe9b0196c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:29:28.409 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:28.187+0000 7fe9b5f62700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fe9b00ff7e0 0x7fe9b0197150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:29:28.409 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:28.187+0000 7fe9b5f62700 1 --2- >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7fe9b0103ee0 0x7fe9b01a1c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:29:28.410 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:28.187+0000 7fe9b5f62700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe9b0197a60 con 0x7fe9b00fee40 2023-07-14T19:29:28.410 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:28.187+0000 7fe9b5f62700 1 -- --> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe9b01a21d0 con 0x7fe9b0103ee0 2023-07-14T19:29:28.410 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:28.187+0000 7fe9b5f62700 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe9b01a2340 con 0x7fe9b00ff7e0 2023-07-14T19:29:28.410 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:28.187+0000 7fe9af7fe700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe9b00fee40 0x7fe9b0196c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:29:28.410 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:28.187+0000 7fe9af7fe700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe9b00fee40 0x7fe9b0196c10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:46022/0 (socket says 172.21.15.98:46022) 2023-07-14T19:29:28.410 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:28.187+0000 7fe9af7fe700 1 -- 172.21.15.98:0/294053084 learned_addr learned my addr 172.21.15.98:0/294053084 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-07-14T19:29:28.410 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:28.187+0000 7fe9aeffd700 1 --2- 172.21.15.98:0/294053084 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fe9b00ff7e0 0x7fe9b0197150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:29:28.410 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:28.188+0000 7fe9af7fe700 1 -- 172.21.15.98:0/294053084 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fe9b00ff7e0 msgr2=0x7fe9b0197150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:29:28.411 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:28.188+0000 7fe9af7fe700 1 --2- 172.21.15.98:0/294053084 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fe9b00ff7e0 0x7fe9b0197150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:29:28.411 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:28.188+0000 7fe9af7fe700 1 -- 172.21.15.98:0/294053084 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7fe9b0103ee0 msgr2=0x7fe9b01a1c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:29:28.411 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:28.188+0000 7fe9af7fe700 1 --2- 172.21.15.98:0/294053084 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7fe9b0103ee0 0x7fe9b01a1c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:29:28.411 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:28.188+0000 7fe9af7fe700 1 -- 172.21.15.98:0/294053084 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe9a40095f0 con 0x7fe9b00fee40 2023-07-14T19:29:28.411 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:28.188+0000 7fe9aeffd700 1 --2- 172.21.15.98:0/294053084 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fe9b00ff7e0 0x7fe9b0197150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-07-14T19:29:28.411 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:28.188+0000 7fe9af7fe700 1 --2- 172.21.15.98:0/294053084 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe9b00fee40 0x7fe9b0196c10 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7fe9a000e9a0 tx=0x7fe9a000ed60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:29:28.411 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:28.188+0000 7fe9acff9700 1 -- 172.21.15.98:0/294053084 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe9a0009790 con 0x7fe9b00fee40 2023-07-14T19:29:28.412 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:28.188+0000 7fe9b5f62700 1 -- 172.21.15.98:0/294053084 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe9b01a2620 con 0x7fe9b00fee40 2023-07-14T19:29:28.412 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:28.188+0000 7fe9acff9700 1 -- 172.21.15.98:0/294053084 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fe9a000cb50 con 0x7fe9b00fee40 2023-07-14T19:29:28.412 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:28.188+0000 7fe9acff9700 1 -- 172.21.15.98:0/294053084 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe9a0018680 con 0x7fe9b00fee40 2023-07-14T19:29:28.412 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:28.188+0000 7fe9b5f62700 1 -- 172.21.15.98:0/294053084 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe9b01a2b40 con 0x7fe9b00fee40 2023-07-14T19:29:28.412 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:28.189+0000 7fe9b5f62700 1 -- 172.21.15.98:0/294053084 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fe9b010ed80 con 0x7fe9b00fee40 2023-07-14T19:29:28.412 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:28.190+0000 7fe9acff9700 1 -- 172.21.15.98:0/294053084 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 17) v1 ==== 135766+0+0 (secure 0 0 0) 0x7fe9a000ccc0 con 0x7fe9b00fee40 2023-07-14T19:29:28.413 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:28.190+0000 7fe9acff9700 1 --2- 172.21.15.98:0/294053084 >> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] conn(0x7fe9980a1260 0x7fe9980a3710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:29:28.413 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:28.191+0000 7fe9aeffd700 1 --2- 172.21.15.98:0/294053084 >> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] conn(0x7fe9980a1260 0x7fe9980a3710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:29:28.413 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:28.191+0000 7fe9acff9700 1 -- 172.21.15.98:0/294053084 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 7979+0+0 (secure 0 0 0) 0x7fe9a0014070 con 0x7fe9b00fee40 2023-07-14T19:29:28.413 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:28.191+0000 7fe9acff9700 1 --2- 172.21.15.98:0/294053084 >> [v2:172.21.15.101:6800/2450584584,v1:172.21.15.101:6801/2450584584] conn(0x7fe9980aaa30 0x7fe9980ace40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:29:28.413 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:28.191+0000 7fe9acff9700 1 -- 172.21.15.98:0/294053084 --> [v2:172.21.15.101:6800/2450584584,v1:172.21.15.101:6801/2450584584] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fe9980ad4f0 con 0x7fe9980aaa30 2023-07-14T19:29:28.413 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:28.191+0000 7fe9acff9700 1 -- 172.21.15.98:0/294053084 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_get_version_reply(handle=1 version=83) v2 ==== 24+0+0 (secure 0 0 0) 0x7fe9a00cd260 con 0x7fe9b00fee40 2023-07-14T19:29:28.414 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:28.191+0000 7fe9aeffd700 1 --2- 172.21.15.98:0/294053084 >> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] conn(0x7fe9980a1260 0x7fe9980a3710 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7fe9a4009fd0 tx=0x7fe9a401a040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:29:28.414 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:28.191+0000 7fe9affff700 1 --2- 172.21.15.98:0/294053084 >> [v2:172.21.15.101:6800/2450584584,v1:172.21.15.101:6801/2450584584] conn(0x7fe9980aaa30 0x7fe9980ace40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:29:28.414 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:28.192+0000 7fe9affff700 1 --2- 172.21.15.98:0/294053084 >> [v2:172.21.15.101:6800/2450584584,v1:172.21.15.101:6801/2450584584] conn(0x7fe9980aaa30 0x7fe9980ace40 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:29:28.414 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:28.192+0000 7fe9acff9700 1 -- 172.21.15.98:0/294053084 <== osd.1 v2:172.21.15.101:6800/2450584584 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25106 (crc 0 0 0) 0x7fe9980ad4f0 con 0x7fe9980aaa30 2023-07-14T19:29:28.414 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:28.207+0000 7fe9b5f62700 1 -- 172.21.15.98:0/294053084 --> [v2:172.21.15.101:6800/2450584584,v1:172.21.15.101:6801/2450584584] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7fe9b004e340 con 0x7fe9980aaa30 2023-07-14T19:29:28.414 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:28.208+0000 7fe9acff9700 1 -- 172.21.15.98:0/294053084 <== osd.1 v2:172.21.15.101:6800/2450584584 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7fe9b004e340 con 0x7fe9980aaa30 2023-07-14T19:29:28.415 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:28.208+0000 7fe9b5f62700 1 -- 172.21.15.98:0/294053084 >> [v2:172.21.15.101:6800/2450584584,v1:172.21.15.101:6801/2450584584] conn(0x7fe9980aaa30 msgr2=0x7fe9980ace40 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:29:28.415 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:28.208+0000 7fe9b5f62700 1 --2- 172.21.15.98:0/294053084 >> [v2:172.21.15.101:6800/2450584584,v1:172.21.15.101:6801/2450584584] conn(0x7fe9980aaa30 0x7fe9980ace40 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:29:28.415 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:28.208+0000 7fe9b5f62700 1 -- 172.21.15.98:0/294053084 >> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] conn(0x7fe9980a1260 msgr2=0x7fe9980a3710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:29:28.415 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:28.208+0000 7fe9b5f62700 1 --2- 172.21.15.98:0/294053084 >> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] conn(0x7fe9980a1260 0x7fe9980a3710 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7fe9a4009fd0 tx=0x7fe9a401a040 comp rx=0 tx=0).stop 2023-07-14T19:29:28.415 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:28.208+0000 7fe9b5f62700 1 -- 172.21.15.98:0/294053084 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe9b00fee40 msgr2=0x7fe9b0196c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:29:28.416 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:28.208+0000 7fe9b5f62700 1 --2- 172.21.15.98:0/294053084 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe9b00fee40 0x7fe9b0196c10 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7fe9a000e9a0 tx=0x7fe9a000ed60 comp rx=0 tx=0).stop 2023-07-14T19:29:28.416 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:28.208+0000 7fe9affff700 1 -- 172.21.15.98:0/294053084 reap_dead start 2023-07-14T19:29:28.416 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:28.209+0000 7fe9b5f62700 1 -- 172.21.15.98:0/294053084 shutdown_connections 2023-07-14T19:29:28.416 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:28.209+0000 7fe9b5f62700 1 -- 172.21.15.98:0/294053084 >> 172.21.15.98:0/294053084 conn(0x7fe9b0075ca0 msgr2=0x7fe9b00fc510 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:29:28.416 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:28.209+0000 7fe9b5f62700 1 -- 172.21.15.98:0/294053084 shutdown_connections 2023-07-14T19:29:28.416 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:28.209+0000 7fe9b5f62700 1 -- 172.21.15.98:0/294053084 wait complete. 2023-07-14T19:29:28.446 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:28.440+0000 7fd7ef3a8700 1 -- 172.21.15.98:0/2990739382 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd7e80fee50 msgr2=0x7fd7e80ff220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:29:28.447 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:28.440+0000 7fd7ef3a8700 1 --2- 172.21.15.98:0/2990739382 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd7e80fee50 0x7fd7e80ff220 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7fd7e00099c0 tx=0x7fd7e0009cd0 comp rx=0 tx=0).stop 2023-07-14T19:29:28.447 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:28.440+0000 7fd7ef3a8700 1 -- 172.21.15.98:0/2990739382 shutdown_connections 2023-07-14T19:29:28.447 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:28.440+0000 7fd7ef3a8700 1 --2- 172.21.15.98:0/2990739382 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fd7e8103ef0 0x7fd7e810ba70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:29:28.447 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:28.440+0000 7fd7ef3a8700 1 --2- 172.21.15.98:0/2990739382 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd7e80fee50 0x7fd7e80ff220 unknown :-1 s=CLOSED pgs=258 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:29:28.447 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:28.440+0000 7fd7ef3a8700 1 --2- 172.21.15.98:0/2990739382 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7fd7e80ff7f0 0x7fd7e81037c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:29:28.447 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:28.440+0000 7fd7ef3a8700 1 -- 172.21.15.98:0/2990739382 >> 172.21.15.98:0/2990739382 conn(0x7fd7e8075a80 msgr2=0x7fd7e8075e80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:29:28.448 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:28.440+0000 7fd7ef3a8700 1 -- 172.21.15.98:0/2990739382 shutdown_connections 2023-07-14T19:29:28.448 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:28.440+0000 7fd7ef3a8700 1 -- 172.21.15.98:0/2990739382 wait complete. 2023-07-14T19:29:28.448 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:28.441+0000 7fd7ef3a8700 1 Processor -- start 2023-07-14T19:29:28.448 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:28.441+0000 7fd7ef3a8700 1 -- start start 2023-07-14T19:29:28.448 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:28.441+0000 7fd7ef3a8700 1 --2- >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7fd7e80fee50 0x7fd7e8194e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:29:28.448 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:28.441+0000 7fd7ef3a8700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fd7e80ff7f0 0x7fd7e81953c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:29:28.449 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:28.441+0000 7fd7ef3a8700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd7e8103ef0 0x7fd7e8199790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:29:28.449 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:28.441+0000 7fd7ef3a8700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd7e8199da0 con 0x7fd7e8103ef0 2023-07-14T19:29:28.449 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:28.441+0000 7fd7ef3a8700 1 -- --> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd7e8199f10 con 0x7fd7e80fee50 2023-07-14T19:29:28.449 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:28.441+0000 7fd7ef3a8700 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd7e819a0d0 con 0x7fd7e80ff7f0 2023-07-14T19:29:28.449 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:28.441+0000 7fd7ed945700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd7e8103ef0 0x7fd7e8199790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:29:28.449 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:28.441+0000 7fd7ed945700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd7e8103ef0 0x7fd7e8199790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:46058/0 (socket says 172.21.15.98:46058) 2023-07-14T19:29:28.449 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:28.441+0000 7fd7ed945700 1 -- 172.21.15.98:0/1774409727 learned_addr learned my addr 172.21.15.98:0/1774409727 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-07-14T19:29:28.449 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:28.441+0000 7fd7ed144700 1 --2- 172.21.15.98:0/1774409727 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7fd7e80fee50 0x7fd7e8194e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:29:28.450 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:28.441+0000 7fd7ec943700 1 --2- 172.21.15.98:0/1774409727 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fd7e80ff7f0 0x7fd7e81953c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:29:28.450 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:28.441+0000 7fd7ed945700 1 -- 172.21.15.98:0/1774409727 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fd7e80ff7f0 msgr2=0x7fd7e81953c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:29:28.450 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:28.441+0000 7fd7ed945700 1 --2- 172.21.15.98:0/1774409727 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fd7e80ff7f0 0x7fd7e81953c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:29:28.450 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:28.441+0000 7fd7ed945700 1 -- 172.21.15.98:0/1774409727 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7fd7e80fee50 msgr2=0x7fd7e8194e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:29:28.450 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:28.441+0000 7fd7ed945700 1 --2- 172.21.15.98:0/1774409727 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7fd7e80fee50 0x7fd7e8194e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:29:28.450 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:28.441+0000 7fd7ed945700 1 -- 172.21.15.98:0/1774409727 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd7e00095f0 con 0x7fd7e8103ef0 2023-07-14T19:29:28.450 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:28.441+0000 7fd7ed144700 1 --2- 172.21.15.98:0/1774409727 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7fd7e80fee50 0x7fd7e8194e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-07-14T19:29:28.451 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:28.441+0000 7fd7ec943700 1 --2- 172.21.15.98:0/1774409727 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fd7e80ff7f0 0x7fd7e81953c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-07-14T19:29:28.451 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:28.442+0000 7fd7ed945700 1 --2- 172.21.15.98:0/1774409727 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd7e8103ef0 0x7fd7e8199790 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7fd7d800db90 tx=0x7fd7d800df50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:29:28.451 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:28.442+0000 7fd7e67fc700 1 -- 172.21.15.98:0/1774409727 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd7d800e800 con 0x7fd7e8103ef0 2023-07-14T19:29:28.451 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:28.442+0000 7fd7e67fc700 1 -- 172.21.15.98:0/1774409727 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fd7d800bac0 con 0x7fd7e8103ef0 2023-07-14T19:29:28.451 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:28.442+0000 7fd7ef3a8700 1 -- 172.21.15.98:0/1774409727 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd7e819a350 con 0x7fd7e8103ef0 2023-07-14T19:29:28.451 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:28.442+0000 7fd7e67fc700 1 -- 172.21.15.98:0/1774409727 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd7d80163f0 con 0x7fd7e8103ef0 2023-07-14T19:29:28.451 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:28.442+0000 7fd7ef3a8700 1 -- 172.21.15.98:0/1774409727 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd7e8068c70 con 0x7fd7e8103ef0 2023-07-14T19:29:28.452 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:28.442+0000 7fd7ef3a8700 1 -- 172.21.15.98:0/1774409727 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fd7e810ee50 con 0x7fd7e8103ef0 2023-07-14T19:29:28.452 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:28.444+0000 7fd7e67fc700 1 -- 172.21.15.98:0/1774409727 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 17) v1 ==== 135766+0+0 (secure 0 0 0) 0x7fd7d8009580 con 0x7fd7e8103ef0 2023-07-14T19:29:28.452 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:28.444+0000 7fd7e67fc700 1 --2- 172.21.15.98:0/1774409727 >> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] conn(0x7fd7d00a11e0 0x7fd7d00a3690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:29:28.452 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:28.444+0000 7fd7ed144700 1 --2- 172.21.15.98:0/1774409727 >> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] conn(0x7fd7d00a11e0 0x7fd7d00a3690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:29:28.452 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:28.444+0000 7fd7e67fc700 1 -- 172.21.15.98:0/1774409727 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 7979+0+0 (secure 0 0 0) 0x7fd7d8006e40 con 0x7fd7e8103ef0 2023-07-14T19:29:28.452 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:28.444+0000 7fd7e67fc700 1 --2- 172.21.15.98:0/1774409727 >> [v2:172.21.15.101:6824/3545244225,v1:172.21.15.101:6825/3545244225] conn(0x7fd7d00aa9b0 0x7fd7d00acdc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:29:28.452 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:28.444+0000 7fd7e67fc700 1 -- 172.21.15.98:0/1774409727 --> [v2:172.21.15.101:6824/3545244225,v1:172.21.15.101:6825/3545244225] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fd7d00ad470 con 0x7fd7d00aa9b0 2023-07-14T19:29:28.453 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:28.444+0000 7fd7e67fc700 1 -- 172.21.15.98:0/1774409727 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_get_version_reply(handle=1 version=83) v2 ==== 24+0+0 (secure 0 0 0) 0x7fd7d80ce4f0 con 0x7fd7e8103ef0 2023-07-14T19:29:28.453 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:28.445+0000 7fd7ed144700 1 --2- 172.21.15.98:0/1774409727 >> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] conn(0x7fd7d00a11e0 0x7fd7d00a3690 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7fd7e0000c00 tx=0x7fd7e0005fb0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:29:28.453 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:28.445+0000 7fd7ec943700 1 --2- 172.21.15.98:0/1774409727 >> [v2:172.21.15.101:6824/3545244225,v1:172.21.15.101:6825/3545244225] conn(0x7fd7d00aa9b0 0x7fd7d00acdc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:29:28.453 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:28.445+0000 7fd7ec943700 1 --2- 172.21.15.98:0/1774409727 >> [v2:172.21.15.101:6824/3545244225,v1:172.21.15.101:6825/3545244225] conn(0x7fd7d00aa9b0 0x7fd7d00acdc0 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.10 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:29:28.453 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:28.446+0000 7fd7e67fc700 1 -- 172.21.15.98:0/1774409727 <== osd.10 v2:172.21.15.101:6824/3545244225 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25106 (crc 0 0 0) 0x7fd7d00ad470 con 0x7fd7d00aa9b0 2023-07-14T19:29:28.463 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:28.462+0000 7fd7ef3a8700 1 -- 172.21.15.98:0/1774409727 --> [v2:172.21.15.101:6824/3545244225,v1:172.21.15.101:6825/3545244225] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7fd7e80682d0 con 0x7fd7d00aa9b0 2023-07-14T19:29:28.464 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:28.462+0000 7fd7e67fc700 1 -- 172.21.15.98:0/1774409727 <== osd.10 v2:172.21.15.101:6824/3545244225 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+12 (crc 0 0 0) 0x7fd7e80682d0 con 0x7fd7d00aa9b0 2023-07-14T19:29:28.464 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:28.463+0000 7fd7ef3a8700 1 -- 172.21.15.98:0/1774409727 >> [v2:172.21.15.101:6824/3545244225,v1:172.21.15.101:6825/3545244225] conn(0x7fd7d00aa9b0 msgr2=0x7fd7d00acdc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:29:28.465 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:28.463+0000 7fd7ef3a8700 1 --2- 172.21.15.98:0/1774409727 >> [v2:172.21.15.101:6824/3545244225,v1:172.21.15.101:6825/3545244225] conn(0x7fd7d00aa9b0 0x7fd7d00acdc0 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:29:28.465 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:28.463+0000 7fd7ef3a8700 1 -- 172.21.15.98:0/1774409727 >> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] conn(0x7fd7d00a11e0 msgr2=0x7fd7d00a3690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:29:28.465 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:28.463+0000 7fd7ef3a8700 1 --2- 172.21.15.98:0/1774409727 >> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] conn(0x7fd7d00a11e0 0x7fd7d00a3690 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7fd7e0000c00 tx=0x7fd7e0005fb0 comp rx=0 tx=0).stop 2023-07-14T19:29:28.465 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:28.463+0000 7fd7ef3a8700 1 -- 172.21.15.98:0/1774409727 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd7e8103ef0 msgr2=0x7fd7e8199790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:29:28.465 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:28.463+0000 7fd7ef3a8700 1 --2- 172.21.15.98:0/1774409727 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd7e8103ef0 0x7fd7e8199790 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7fd7d800db90 tx=0x7fd7d800df50 comp rx=0 tx=0).stop 2023-07-14T19:29:28.465 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:28.463+0000 7fd7ed945700 1 -- 172.21.15.98:0/1774409727 reap_dead start 2023-07-14T19:29:28.465 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:28.463+0000 7fd7ef3a8700 1 -- 172.21.15.98:0/1774409727 shutdown_connections 2023-07-14T19:29:28.466 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:28.463+0000 7fd7ef3a8700 1 -- 172.21.15.98:0/1774409727 >> 172.21.15.98:0/1774409727 conn(0x7fd7e8075a80 msgr2=0x7fd7e80fc790 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:29:28.466 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:28.463+0000 7fd7ef3a8700 1 -- 172.21.15.98:0/1774409727 shutdown_connections 2023-07-14T19:29:28.466 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:28.463+0000 7fd7ef3a8700 1 -- 172.21.15.98:0/1774409727 wait complete. 2023-07-14T19:29:28.524 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:28.357+0000 7f804c152700 1 -- 172.21.15.98:0/3094431676 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8044075bb0 msgr2=0x7f80440fef60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:29:28.524 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:28.357+0000 7f804c152700 1 --2- 172.21.15.98:0/3094431676 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8044075bb0 0x7f80440fef60 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f8040009990 tx=0x7f8040009ca0 comp rx=0 tx=0).stop 2023-07-14T19:29:28.524 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:28.358+0000 7f804c152700 1 -- 172.21.15.98:0/3094431676 shutdown_connections 2023-07-14T19:29:28.524 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:28.358+0000 7f804c152700 1 --2- 172.21.15.98:0/3094431676 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f804410a380 0x7f804410c780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:29:28.525 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:28.358+0000 7f804c152700 1 --2- 172.21.15.98:0/3094431676 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8044075bb0 0x7f80440fef60 unknown :-1 s=CLOSED pgs=256 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:29:28.525 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:28.358+0000 7f804c152700 1 --2- 172.21.15.98:0/3094431676 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f80440ff530 0x7f80440ff9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:29:28.525 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:28.358+0000 7f804c152700 1 -- 172.21.15.98:0/3094431676 >> 172.21.15.98:0/3094431676 conn(0x7f8044074550 msgr2=0x7f8044074950 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:29:28.525 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:28.358+0000 7f804c152700 1 -- 172.21.15.98:0/3094431676 shutdown_connections 2023-07-14T19:29:28.525 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:28.358+0000 7f804c152700 1 -- 172.21.15.98:0/3094431676 wait complete. 2023-07-14T19:29:28.525 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:28.358+0000 7f804c152700 1 Processor -- start 2023-07-14T19:29:28.525 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:28.358+0000 7f804c152700 1 -- start start 2023-07-14T19:29:28.526 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:28.359+0000 7f804c152700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f80440ff530 0x7f8044195de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:29:28.526 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:28.359+0000 7f804c152700 1 --2- >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f804410a380 0x7f8044196320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:29:28.526 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:28.359+0000 7f804c152700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f804419a740 0x7f804419abb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:29:28.526 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:28.359+0000 7f804c152700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f804419b120 con 0x7f804419a740 2023-07-14T19:29:28.526 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:28.359+0000 7f804c152700 1 -- --> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f804419b290 con 0x7f804410a380 2023-07-14T19:29:28.526 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:28.359+0000 7f804c152700 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f804419b450 con 0x7f80440ff530 2023-07-14T19:29:28.526 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:28.359+0000 7f804a6ef700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f804419a740 0x7f804419abb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:29:28.527 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:28.359+0000 7f8049eee700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f80440ff530 0x7f8044195de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:29:28.527 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:28.359+0000 7f804a6ef700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f804419a740 0x7f804419abb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:46042/0 (socket says 172.21.15.98:46042) 2023-07-14T19:29:28.527 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:28.359+0000 7f8049eee700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f80440ff530 0x7f8044195de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.139:3300/0 says I am v2:172.21.15.98:52992/0 (socket says 172.21.15.98:52992) 2023-07-14T19:29:28.527 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:28.359+0000 7f804a6ef700 1 -- 172.21.15.98:0/2765273256 learned_addr learned my addr 172.21.15.98:0/2765273256 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-07-14T19:29:28.527 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:28.359+0000 7f80496ed700 1 --2- 172.21.15.98:0/2765273256 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f804410a380 0x7f8044196320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:29:28.527 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:28.359+0000 7f804a6ef700 1 -- 172.21.15.98:0/2765273256 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f80440ff530 msgr2=0x7f8044195de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:29:28.527 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:28.359+0000 7f804a6ef700 1 --2- 172.21.15.98:0/2765273256 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f80440ff530 0x7f8044195de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:29:28.528 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:28.359+0000 7f804a6ef700 1 -- 172.21.15.98:0/2765273256 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f804410a380 msgr2=0x7f8044196320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:29:28.528 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:28.359+0000 7f804a6ef700 1 --2- 172.21.15.98:0/2765273256 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f804410a380 0x7f8044196320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:29:28.528 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:28.359+0000 7f804a6ef700 1 -- 172.21.15.98:0/2765273256 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f80400095f0 con 0x7f804419a740 2023-07-14T19:29:28.528 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:28.359+0000 7f8049eee700 1 --2- 172.21.15.98:0/2765273256 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f80440ff530 0x7f8044195de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-07-14T19:29:28.528 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:28.359+0000 7f804a6ef700 1 --2- 172.21.15.98:0/2765273256 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f804419a740 0x7f804419abb0 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f803c00ada0 tx=0x7f803c00c330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:29:28.529 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:28.359+0000 7f803affd700 1 -- 172.21.15.98:0/2765273256 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f803c00f070 con 0x7f804419a740 2023-07-14T19:29:28.529 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:28.360+0000 7f804c152700 1 -- 172.21.15.98:0/2765273256 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f80441a29f0 con 0x7f804419a740 2023-07-14T19:29:28.529 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:28.360+0000 7f803affd700 1 -- 172.21.15.98:0/2765273256 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f803c007b30 con 0x7f804419a740 2023-07-14T19:29:28.529 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:28.360+0000 7f803affd700 1 -- 172.21.15.98:0/2765273256 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f803c005280 con 0x7f804419a740 2023-07-14T19:29:28.529 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:28.360+0000 7f804c152700 1 -- 172.21.15.98:0/2765273256 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f80441a2ea0 con 0x7f804419a740 2023-07-14T19:29:28.530 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:28.360+0000 7f804c152700 1 -- 172.21.15.98:0/2765273256 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f804404e340 con 0x7f804419a740 2023-07-14T19:29:28.530 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:28.362+0000 7f803affd700 1 -- 172.21.15.98:0/2765273256 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 17) v1 ==== 135766+0+0 (secure 0 0 0) 0x7f803c00a080 con 0x7f804419a740 2023-07-14T19:29:28.530 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:28.362+0000 7f803affd700 1 --2- 172.21.15.98:0/2765273256 >> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] conn(0x7f80300a11e0 0x7f80300a3690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:29:28.531 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:28.362+0000 7f8049eee700 1 --2- 172.21.15.98:0/2765273256 >> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] conn(0x7f80300a11e0 0x7f80300a3690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:29:28.531 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:28.362+0000 7f803affd700 1 -- 172.21.15.98:0/2765273256 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 7979+0+0 (secure 0 0 0) 0x7f803c0cc840 con 0x7f804419a740 2023-07-14T19:29:28.531 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:28.363+0000 7f803affd700 1 --2- 172.21.15.98:0/2765273256 >> [v2:172.21.15.98:6826/2893994080,v1:172.21.15.98:6827/2893994080] conn(0x7f80300aa9b0 0x7f80300acdc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:29:28.531 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:28.363+0000 7f803affd700 1 -- 172.21.15.98:0/2765273256 --> [v2:172.21.15.98:6826/2893994080,v1:172.21.15.98:6827/2893994080] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f80300ad470 con 0x7f80300aa9b0 2023-07-14T19:29:28.531 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:28.363+0000 7f803affd700 1 -- 172.21.15.98:0/2765273256 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_get_version_reply(handle=1 version=83) v2 ==== 24+0+0 (secure 0 0 0) 0x7f803c0ccbb0 con 0x7f804419a740 2023-07-14T19:29:28.531 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:28.363+0000 7f80496ed700 1 --2- 172.21.15.98:0/2765273256 >> [v2:172.21.15.98:6826/2893994080,v1:172.21.15.98:6827/2893994080] conn(0x7f80300aa9b0 0x7f80300acdc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:29:28.531 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:28.363+0000 7f8049eee700 1 --2- 172.21.15.98:0/2765273256 >> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] conn(0x7f80300a11e0 0x7f80300a3690 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f804000b5c0 tx=0x7f8040005960 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:29:28.532 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:28.363+0000 7f80496ed700 1 --2- 172.21.15.98:0/2765273256 >> [v2:172.21.15.98:6826/2893994080,v1:172.21.15.98:6827/2893994080] conn(0x7f80300aa9b0 0x7f80300acdc0 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.9 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:29:28.532 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:28.364+0000 7f803affd700 1 -- 172.21.15.98:0/2765273256 <== osd.9 v2:172.21.15.98:6826/2893994080 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25106 (crc 0 0 0) 0x7f80300ad470 con 0x7f80300aa9b0 2023-07-14T19:29:28.532 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:28.393+0000 7f804c152700 1 -- 172.21.15.98:0/2765273256 --> [v2:172.21.15.98:6826/2893994080,v1:172.21.15.98:6827/2893994080] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f804410fc30 con 0x7f80300aa9b0 2023-07-14T19:29:28.532 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:28.394+0000 7f803affd700 1 -- 172.21.15.98:0/2765273256 <== osd.9 v2:172.21.15.98:6826/2893994080 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+12 (crc 0 0 0) 0x7f804410fc30 con 0x7f80300aa9b0 2023-07-14T19:29:28.532 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:28.394+0000 7f804c152700 1 -- 172.21.15.98:0/2765273256 >> [v2:172.21.15.98:6826/2893994080,v1:172.21.15.98:6827/2893994080] conn(0x7f80300aa9b0 msgr2=0x7f80300acdc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:29:28.532 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:28.394+0000 7f804c152700 1 --2- 172.21.15.98:0/2765273256 >> [v2:172.21.15.98:6826/2893994080,v1:172.21.15.98:6827/2893994080] conn(0x7f80300aa9b0 0x7f80300acdc0 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:29:28.533 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:28.394+0000 7f804c152700 1 -- 172.21.15.98:0/2765273256 >> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] conn(0x7f80300a11e0 msgr2=0x7f80300a3690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:29:28.533 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:28.394+0000 7f804c152700 1 --2- 172.21.15.98:0/2765273256 >> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] conn(0x7f80300a11e0 0x7f80300a3690 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f804000b5c0 tx=0x7f8040005960 comp rx=0 tx=0).stop 2023-07-14T19:29:28.533 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:28.394+0000 7f804c152700 1 -- 172.21.15.98:0/2765273256 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f804419a740 msgr2=0x7f804419abb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:29:28.533 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:28.394+0000 7f804c152700 1 --2- 172.21.15.98:0/2765273256 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f804419a740 0x7f804419abb0 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f803c00ada0 tx=0x7f803c00c330 comp rx=0 tx=0).stop 2023-07-14T19:29:28.533 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:28.394+0000 7f804a6ef700 1 -- 172.21.15.98:0/2765273256 reap_dead start 2023-07-14T19:29:28.533 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:28.394+0000 7f804c152700 1 -- 172.21.15.98:0/2765273256 shutdown_connections 2023-07-14T19:29:28.534 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:28.394+0000 7f804c152700 1 -- 172.21.15.98:0/2765273256 >> 172.21.15.98:0/2765273256 conn(0x7f8044074550 msgr2=0x7f8044102280 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:29:28.534 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:28.395+0000 7f804c152700 1 -- 172.21.15.98:0/2765273256 shutdown_connections 2023-07-14T19:29:28.534 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:28.395+0000 7f804c152700 1 -- 172.21.15.98:0/2765273256 wait complete. 2023-07-14T19:29:28.781 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:29:28 smithi098 ceph-mon[127486]: pgmap v266: 97 pgs: 97 active+clean; 633 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 1.3 KiB/s wr, 6 op/s 2023-07-14T19:29:28.929 INFO:teuthology.orchestra.run.smithi098.stdout:300647710741 2023-07-14T19:29:28.929 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81c9387d2e7576da065747ec9c16d1c7d69016ec shell --fsid 9adea648-227b-11ee-9b33-001a4aab830c -- ceph osd last-stat-seq osd.10 2023-07-14T19:29:28.941 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:29:28 smithi101 ceph-mon[128807]: pgmap v266: 97 pgs: 97 active+clean; 633 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 1.3 KiB/s wr, 6 op/s 2023-07-14T19:29:28.970 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:29:28 smithi139 ceph-mon[129293]: pgmap v266: 97 pgs: 97 active+clean; 633 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 1.3 KiB/s wr, 6 op/s 2023-07-14T19:29:29.574 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:29.546+0000 7fcfdc243700 1 -- 172.21.15.98:0/2754732566 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcfd40ff7e0 msgr2=0x7fcfd41037b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:29:29.574 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:29.546+0000 7fcfdc243700 1 --2- 172.21.15.98:0/2754732566 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcfd40ff7e0 0x7fcfd41037b0 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7fcfc40099c0 tx=0x7fcfc4009cd0 comp rx=0 tx=0).stop 2023-07-14T19:29:29.575 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:29.546+0000 7fcfdc243700 1 -- 172.21.15.98:0/2754732566 shutdown_connections 2023-07-14T19:29:29.575 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:29.546+0000 7fcfdc243700 1 --2- 172.21.15.98:0/2754732566 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fcfd40fee40 0x7fcfd40ff210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:29:29.575 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:29.546+0000 7fcfdc243700 1 --2- 172.21.15.98:0/2754732566 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7fcfd4103ee0 0x7fcfd410ba60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:29:29.575 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:29.546+0000 7fcfdc243700 1 --2- 172.21.15.98:0/2754732566 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcfd40ff7e0 0x7fcfd41037b0 unknown :-1 s=CLOSED pgs=260 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:29:29.575 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:29.546+0000 7fcfdc243700 1 -- 172.21.15.98:0/2754732566 >> 172.21.15.98:0/2754732566 conn(0x7fcfd4075ca0 msgr2=0x7fcfd40760a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:29:29.575 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:29.546+0000 7fcfdc243700 1 -- 172.21.15.98:0/2754732566 shutdown_connections 2023-07-14T19:29:29.575 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:29.546+0000 7fcfdc243700 1 -- 172.21.15.98:0/2754732566 wait complete. 2023-07-14T19:29:29.576 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:29.547+0000 7fcfdc243700 1 Processor -- start 2023-07-14T19:29:29.576 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:29.547+0000 7fcfdc243700 1 -- start start 2023-07-14T19:29:29.576 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:29.547+0000 7fcfdc243700 1 --2- >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7fcfd40fee40 0x7fcfd4195da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:29:29.576 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:29.547+0000 7fcfdc243700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fcfd40ff7e0 0x7fcfd41962e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:29:29.576 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:29.547+0000 7fcfdc243700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcfd4103ee0 0x7fcfd419a750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:29:29.576 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:29.547+0000 7fcfdc243700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcfd419ad60 con 0x7fcfd4103ee0 2023-07-14T19:29:29.576 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:29.547+0000 7fcfdc243700 1 -- --> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcfd419aed0 con 0x7fcfd40fee40 2023-07-14T19:29:29.577 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:29.547+0000 7fcfdc243700 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcfd419b090 con 0x7fcfd40ff7e0 2023-07-14T19:29:29.577 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:29.547+0000 7fcfda7e0700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcfd4103ee0 0x7fcfd419a750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:29:29.577 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:29.547+0000 7fcfda7e0700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcfd4103ee0 0x7fcfd419a750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:46074/0 (socket says 172.21.15.98:46074) 2023-07-14T19:29:29.577 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:29.547+0000 7fcfda7e0700 1 -- 172.21.15.98:0/3511881661 learned_addr learned my addr 172.21.15.98:0/3511881661 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-07-14T19:29:29.577 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:29.547+0000 7fcfd97de700 1 --2- 172.21.15.98:0/3511881661 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fcfd40ff7e0 0x7fcfd41962e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:29:29.577 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:29.548+0000 7fcfd9fdf700 1 --2- 172.21.15.98:0/3511881661 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7fcfd40fee40 0x7fcfd4195da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:29:29.577 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:29.548+0000 7fcfda7e0700 1 -- 172.21.15.98:0/3511881661 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fcfd40ff7e0 msgr2=0x7fcfd41962e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:29:29.578 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:29.548+0000 7fcfda7e0700 1 --2- 172.21.15.98:0/3511881661 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fcfd40ff7e0 0x7fcfd41962e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:29:29.578 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:29.548+0000 7fcfda7e0700 1 -- 172.21.15.98:0/3511881661 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7fcfd40fee40 msgr2=0x7fcfd4195da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:29:29.578 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:29.548+0000 7fcfda7e0700 1 --2- 172.21.15.98:0/3511881661 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7fcfd40fee40 0x7fcfd4195da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:29:29.578 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:29.548+0000 7fcfda7e0700 1 -- 172.21.15.98:0/3511881661 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcfc40095f0 con 0x7fcfd4103ee0 2023-07-14T19:29:29.579 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:29.548+0000 7fcfda7e0700 1 --2- 172.21.15.98:0/3511881661 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcfd4103ee0 0x7fcfd419a750 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7fcfcc00cca0 tx=0x7fcfcc00a060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:29:29.579 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:29.548+0000 7fcfcaffd700 1 -- 172.21.15.98:0/3511881661 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcfcc00aba0 con 0x7fcfd4103ee0 2023-07-14T19:29:29.579 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:29.548+0000 7fcfdc243700 1 -- 172.21.15.98:0/3511881661 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcfd419b370 con 0x7fcfd4103ee0 2023-07-14T19:29:29.579 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:29.548+0000 7fcfcaffd700 1 -- 172.21.15.98:0/3511881661 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fcfcc00ad00 con 0x7fcfd4103ee0 2023-07-14T19:29:29.579 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:29.548+0000 7fcfcaffd700 1 -- 172.21.15.98:0/3511881661 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcfcc0115e0 con 0x7fcfd4103ee0 2023-07-14T19:29:29.579 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:29.548+0000 7fcfdc243700 1 -- 172.21.15.98:0/3511881661 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcfd41a2bd0 con 0x7fcfd4103ee0 2023-07-14T19:29:29.579 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:29.549+0000 7fcfdc243700 1 -- 172.21.15.98:0/3511881661 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fcfd404e340 con 0x7fcfd4103ee0 2023-07-14T19:29:29.580 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:29.550+0000 7fcfcaffd700 1 -- 172.21.15.98:0/3511881661 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 17) v1 ==== 135766+0+0 (secure 0 0 0) 0x7fcfcc012420 con 0x7fcfd4103ee0 2023-07-14T19:29:29.580 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:29.550+0000 7fcfcaffd700 1 --2- 172.21.15.98:0/3511881661 >> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] conn(0x7fcfc00a11e0 0x7fcfc00a3690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:29:29.580 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:29.551+0000 7fcfd9fdf700 1 --2- 172.21.15.98:0/3511881661 >> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] conn(0x7fcfc00a11e0 0x7fcfc00a3690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:29:29.580 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:29.551+0000 7fcfcaffd700 1 -- 172.21.15.98:0/3511881661 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 7979+0+0 (secure 0 0 0) 0x7fcfcc0ccf70 con 0x7fcfd4103ee0 2023-07-14T19:29:29.580 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:29.551+0000 7fcfcaffd700 1 --2- 172.21.15.98:0/3511881661 >> [v2:172.21.15.98:6818/658948743,v1:172.21.15.98:6819/658948743] conn(0x7fcfc00aa9b0 0x7fcfc00acdc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:29:29.580 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:29.551+0000 7fcfcaffd700 1 -- 172.21.15.98:0/3511881661 --> [v2:172.21.15.98:6818/658948743,v1:172.21.15.98:6819/658948743] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fcfc00ad470 con 0x7fcfc00aa9b0 2023-07-14T19:29:29.580 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:29.551+0000 7fcfcaffd700 1 -- 172.21.15.98:0/3511881661 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_get_version_reply(handle=1 version=83) v2 ==== 24+0+0 (secure 0 0 0) 0x7fcfcc0cd410 con 0x7fcfd4103ee0 2023-07-14T19:29:29.581 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:29.551+0000 7fcfd97de700 1 --2- 172.21.15.98:0/3511881661 >> [v2:172.21.15.98:6818/658948743,v1:172.21.15.98:6819/658948743] conn(0x7fcfc00aa9b0 0x7fcfc00acdc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:29:29.581 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:29.551+0000 7fcfd9fdf700 1 --2- 172.21.15.98:0/3511881661 >> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] conn(0x7fcfc00a11e0 0x7fcfc00a3690 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7fcfd0006fd0 tx=0x7fcfd0008040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:29:29.581 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:29.551+0000 7fcfd97de700 1 --2- 172.21.15.98:0/3511881661 >> [v2:172.21.15.98:6818/658948743,v1:172.21.15.98:6819/658948743] conn(0x7fcfc00aa9b0 0x7fcfc00acdc0 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:29:29.581 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:29.552+0000 7fcfcaffd700 1 -- 172.21.15.98:0/3511881661 <== osd.6 v2:172.21.15.98:6818/658948743 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25106 (crc 0 0 0) 0x7fcfc00ad470 con 0x7fcfc00aa9b0 2023-07-14T19:29:29.582 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:29.567+0000 7fcfdc243700 1 -- 172.21.15.98:0/3511881661 --> [v2:172.21.15.98:6818/658948743,v1:172.21.15.98:6819/658948743] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7fcfd4068f30 con 0x7fcfc00aa9b0 2023-07-14T19:29:29.582 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:29.568+0000 7fcfcaffd700 1 -- 172.21.15.98:0/3511881661 <== osd.6 v2:172.21.15.98:6818/658948743 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+12 (crc 0 0 0) 0x7fcfc401e040 con 0x7fcfc00aa9b0 2023-07-14T19:29:29.582 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:29.568+0000 7fcfdc243700 1 -- 172.21.15.98:0/3511881661 >> [v2:172.21.15.98:6818/658948743,v1:172.21.15.98:6819/658948743] conn(0x7fcfc00aa9b0 msgr2=0x7fcfc00acdc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:29:29.582 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:29.568+0000 7fcfdc243700 1 --2- 172.21.15.98:0/3511881661 >> [v2:172.21.15.98:6818/658948743,v1:172.21.15.98:6819/658948743] conn(0x7fcfc00aa9b0 0x7fcfc00acdc0 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:29:29.582 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:29.568+0000 7fcfdc243700 1 -- 172.21.15.98:0/3511881661 >> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] conn(0x7fcfc00a11e0 msgr2=0x7fcfc00a3690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:29:29.582 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:29.568+0000 7fcfdc243700 1 --2- 172.21.15.98:0/3511881661 >> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] conn(0x7fcfc00a11e0 0x7fcfc00a3690 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7fcfd0006fd0 tx=0x7fcfd0008040 comp rx=0 tx=0).stop 2023-07-14T19:29:29.582 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:29.568+0000 7fcfdc243700 1 -- 172.21.15.98:0/3511881661 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcfd4103ee0 msgr2=0x7fcfd419a750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:29:29.583 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:29.568+0000 7fcfdc243700 1 --2- 172.21.15.98:0/3511881661 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcfd4103ee0 0x7fcfd419a750 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7fcfcc00cca0 tx=0x7fcfcc00a060 comp rx=0 tx=0).stop 2023-07-14T19:29:29.583 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:29.568+0000 7fcfda7e0700 1 -- 172.21.15.98:0/3511881661 reap_dead start 2023-07-14T19:29:29.583 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:29.568+0000 7fcfdc243700 1 -- 172.21.15.98:0/3511881661 shutdown_connections 2023-07-14T19:29:29.583 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:29.568+0000 7fcfdc243700 1 -- 172.21.15.98:0/3511881661 >> 172.21.15.98:0/3511881661 conn(0x7fcfd4075ca0 msgr2=0x7fcfd40fc4f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:29:29.583 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:29.568+0000 7fcfdc243700 1 -- 172.21.15.98:0/3511881661 shutdown_connections 2023-07-14T19:29:29.583 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:29.568+0000 7fcfdc243700 1 -- 172.21.15.98:0/3511881661 wait complete. 2023-07-14T19:29:29.599 INFO:teuthology.orchestra.run.smithi098.stdout:279172874265 2023-07-14T19:29:29.600 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81c9387d2e7576da065747ec9c16d1c7d69016ec shell --fsid 9adea648-227b-11ee-9b33-001a4aab830c -- ceph osd last-stat-seq osd.9 2023-07-14T19:29:30.029 INFO:teuthology.orchestra.run.smithi098.stdout:107374182450 2023-07-14T19:29:30.029 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81c9387d2e7576da065747ec9c16d1c7d69016ec shell --fsid 9adea648-227b-11ee-9b33-001a4aab830c -- ceph osd last-stat-seq osd.3 2023-07-14T19:29:30.046 INFO:teuthology.orchestra.run.smithi098.stdout:335007449104 2023-07-14T19:29:30.047 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81c9387d2e7576da065747ec9c16d1c7d69016ec shell --fsid 9adea648-227b-11ee-9b33-001a4aab830c -- ceph osd last-stat-seq osd.11 2023-07-14T19:29:30.429 INFO:teuthology.orchestra.run.smithi098.stdout:193273528358 2023-07-14T19:29:30.430 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81c9387d2e7576da065747ec9c16d1c7d69016ec shell --fsid 9adea648-227b-11ee-9b33-001a4aab830c -- ceph osd last-stat-seq osd.6 2023-07-14T19:29:30.941 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:29:30 smithi101 ceph-mon[128807]: pgmap v267: 97 pgs: 97 active+clean; 634 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 1.6 KiB/s wr, 7 op/s 2023-07-14T19:29:30.970 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:29:30 smithi139 ceph-mon[129293]: pgmap v267: 97 pgs: 97 active+clean; 634 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 1.6 KiB/s wr, 7 op/s 2023-07-14T19:29:31.031 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:29:30 smithi098 ceph-mon[127486]: pgmap v267: 97 pgs: 97 active+clean; 634 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 1.6 KiB/s wr, 7 op/s 2023-07-14T19:29:31.032 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:30.290+0000 7f1f6b0d2700 1 -- 172.21.15.98:0/808591687 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1f64100f80 msgr2=0x7f1f64101350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:29:31.032 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:30.290+0000 7f1f6b0d2700 1 --2- 172.21.15.98:0/808591687 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1f64100f80 0x7f1f64101350 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f1f58009990 tx=0x7f1f58009ca0 comp rx=0 tx=0).stop 2023-07-14T19:29:31.032 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:30.290+0000 7f1f6b0d2700 1 -- 172.21.15.98:0/808591687 shutdown_connections 2023-07-14T19:29:31.032 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:30.290+0000 7f1f6b0d2700 1 --2- 172.21.15.98:0/808591687 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f1f6410a2d0 0x7f1f6410c730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:29:31.032 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:30.290+0000 7f1f6b0d2700 1 --2- 172.21.15.98:0/808591687 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1f64100f80 0x7f1f64101350 unknown :-1 s=CLOSED pgs=262 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:29:31.033 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:30.290+0000 7f1f6b0d2700 1 --2- 172.21.15.98:0/808591687 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f1f64101890 0x7f1f64109d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:29:31.033 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:30.290+0000 7f1f6b0d2700 1 -- 172.21.15.98:0/808591687 >> 172.21.15.98:0/808591687 conn(0x7f1f640fa720 msgr2=0x7f1f640fcb30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:29:31.033 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:30.290+0000 7f1f6b0d2700 1 -- 172.21.15.98:0/808591687 shutdown_connections 2023-07-14T19:29:31.033 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:30.290+0000 7f1f6b0d2700 1 -- 172.21.15.98:0/808591687 wait complete. 2023-07-14T19:29:31.033 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:30.291+0000 7f1f6b0d2700 1 Processor -- start 2023-07-14T19:29:31.033 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:30.291+0000 7f1f6b0d2700 1 -- start start 2023-07-14T19:29:31.033 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:30.291+0000 7f1f6b0d2700 1 --2- >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f1f64100f80 0x7f1f64195e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:29:31.034 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:30.291+0000 7f1f6b0d2700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1f64101890 0x7f1f64196380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:29:31.034 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:30.291+0000 7f1f6b0d2700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f1f6410a2d0 0x7f1f6419a7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:29:31.034 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:30.291+0000 7f1f6b0d2700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1f6419ae00 con 0x7f1f64101890 2023-07-14T19:29:31.034 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:30.291+0000 7f1f6b0d2700 1 -- --> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1f6419af70 con 0x7f1f64100f80 2023-07-14T19:29:31.034 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:30.291+0000 7f1f6b0d2700 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1f6419b130 con 0x7f1f6410a2d0 2023-07-14T19:29:31.034 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:30.292+0000 7f1f63fff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1f64101890 0x7f1f64196380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:29:31.034 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:30.292+0000 7f1f63fff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1f64101890 0x7f1f64196380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:46086/0 (socket says 172.21.15.98:46086) 2023-07-14T19:29:31.034 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:30.292+0000 7f1f63fff700 1 -- 172.21.15.98:0/1387765200 learned_addr learned my addr 172.21.15.98:0/1387765200 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-07-14T19:29:31.035 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:30.292+0000 7f1f68e6e700 1 --2- 172.21.15.98:0/1387765200 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f1f64100f80 0x7f1f64195e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:29:31.035 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:30.292+0000 7f1f6966f700 1 --2- 172.21.15.98:0/1387765200 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f1f6410a2d0 0x7f1f6419a7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:29:31.035 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:30.292+0000 7f1f63fff700 1 -- 172.21.15.98:0/1387765200 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f1f6410a2d0 msgr2=0x7f1f6419a7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:29:31.035 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:30.292+0000 7f1f63fff700 1 --2- 172.21.15.98:0/1387765200 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f1f6410a2d0 0x7f1f6419a7f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:29:31.035 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:30.292+0000 7f1f63fff700 1 -- 172.21.15.98:0/1387765200 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f1f64100f80 msgr2=0x7f1f64195e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:29:31.035 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:30.292+0000 7f1f63fff700 1 --2- 172.21.15.98:0/1387765200 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f1f64100f80 0x7f1f64195e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:29:31.035 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:30.292+0000 7f1f63fff700 1 -- 172.21.15.98:0/1387765200 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1f580095f0 con 0x7f1f64101890 2023-07-14T19:29:31.036 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:30.292+0000 7f1f63fff700 1 --2- 172.21.15.98:0/1387765200 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1f64101890 0x7f1f64196380 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7f1f5000d950 tx=0x7f1f5000dc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:29:31.036 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:30.292+0000 7f1f61ffb700 1 -- 172.21.15.98:0/1387765200 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f5000f890 con 0x7f1f64101890 2023-07-14T19:29:31.036 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:30.292+0000 7f1f61ffb700 1 -- 172.21.15.98:0/1387765200 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f1f50010460 con 0x7f1f64101890 2023-07-14T19:29:31.036 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:30.292+0000 7f1f6b0d2700 1 -- 172.21.15.98:0/1387765200 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1f6419b410 con 0x7f1f64101890 2023-07-14T19:29:31.036 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:30.292+0000 7f1f61ffb700 1 -- 172.21.15.98:0/1387765200 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f5000e610 con 0x7f1f64101890 2023-07-14T19:29:31.036 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:30.292+0000 7f1f6b0d2700 1 -- 172.21.15.98:0/1387765200 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1f641a2c70 con 0x7f1f64101890 2023-07-14T19:29:31.036 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:30.293+0000 7f1f6b0d2700 1 -- 172.21.15.98:0/1387765200 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f1f6410fe50 con 0x7f1f64101890 2023-07-14T19:29:31.037 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:30.294+0000 7f1f61ffb700 1 -- 172.21.15.98:0/1387765200 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 17) v1 ==== 135766+0+0 (secure 0 0 0) 0x7f1f500108d0 con 0x7f1f64101890 2023-07-14T19:29:31.037 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:30.295+0000 7f1f61ffb700 1 --2- 172.21.15.98:0/1387765200 >> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] conn(0x7f1f540a11e0 0x7f1f540a3690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:29:31.037 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:30.295+0000 7f1f68e6e700 1 --2- 172.21.15.98:0/1387765200 >> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] conn(0x7f1f540a11e0 0x7f1f540a3690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:29:31.037 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:30.295+0000 7f1f61ffb700 1 -- 172.21.15.98:0/1387765200 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 7979+0+0 (secure 0 0 0) 0x7f1f500cc6d0 con 0x7f1f64101890 2023-07-14T19:29:31.037 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:30.295+0000 7f1f61ffb700 1 --2- 172.21.15.98:0/1387765200 >> [v2:172.21.15.139:6800/945038995,v1:172.21.15.139:6801/945038995] conn(0x7f1f540aa9b0 0x7f1f540acdc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:29:31.037 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:30.295+0000 7f1f61ffb700 1 -- 172.21.15.98:0/1387765200 --> [v2:172.21.15.139:6800/945038995,v1:172.21.15.139:6801/945038995] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f1f540ad470 con 0x7f1f540aa9b0 2023-07-14T19:29:31.037 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:30.295+0000 7f1f61ffb700 1 -- 172.21.15.98:0/1387765200 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_get_version_reply(handle=1 version=83) v2 ==== 24+0+0 (secure 0 0 0) 0x7f1f500cede0 con 0x7f1f64101890 2023-07-14T19:29:31.037 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:30.295+0000 7f1f68e6e700 1 --2- 172.21.15.98:0/1387765200 >> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] conn(0x7f1f540a11e0 0x7f1f540a3690 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f1f5800b5c0 tx=0x7f1f58005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:29:31.038 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:30.295+0000 7f1f6966f700 1 --2- 172.21.15.98:0/1387765200 >> [v2:172.21.15.139:6800/945038995,v1:172.21.15.139:6801/945038995] conn(0x7f1f540aa9b0 0x7f1f540acdc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:29:31.038 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:30.296+0000 7f1f6966f700 1 --2- 172.21.15.98:0/1387765200 >> [v2:172.21.15.139:6800/945038995,v1:172.21.15.139:6801/945038995] conn(0x7f1f540aa9b0 0x7f1f540acdc0 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:29:31.038 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:30.297+0000 7f1f61ffb700 1 -- 172.21.15.98:0/1387765200 <== osd.2 v2:172.21.15.139:6800/945038995 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25106 (crc 0 0 0) 0x7f1f540ad470 con 0x7f1f540aa9b0 2023-07-14T19:29:31.039 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:30.312+0000 7f1f6b0d2700 1 -- 172.21.15.98:0/1387765200 --> [v2:172.21.15.139:6800/945038995,v1:172.21.15.139:6801/945038995] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f1f640682d0 con 0x7f1f540aa9b0 2023-07-14T19:29:31.039 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:30.313+0000 7f1f61ffb700 1 -- 172.21.15.98:0/1387765200 <== osd.2 v2:172.21.15.139:6800/945038995 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f1f640682d0 con 0x7f1f540aa9b0 2023-07-14T19:29:31.039 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:30.313+0000 7f1f6b0d2700 1 -- 172.21.15.98:0/1387765200 >> [v2:172.21.15.139:6800/945038995,v1:172.21.15.139:6801/945038995] conn(0x7f1f540aa9b0 msgr2=0x7f1f540acdc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:29:31.039 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:30.313+0000 7f1f6b0d2700 1 --2- 172.21.15.98:0/1387765200 >> [v2:172.21.15.139:6800/945038995,v1:172.21.15.139:6801/945038995] conn(0x7f1f540aa9b0 0x7f1f540acdc0 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:29:31.039 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:30.313+0000 7f1f6b0d2700 1 -- 172.21.15.98:0/1387765200 >> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] conn(0x7f1f540a11e0 msgr2=0x7f1f540a3690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:29:31.039 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:30.313+0000 7f1f6b0d2700 1 --2- 172.21.15.98:0/1387765200 >> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] conn(0x7f1f540a11e0 0x7f1f540a3690 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f1f5800b5c0 tx=0x7f1f58005fd0 comp rx=0 tx=0).stop 2023-07-14T19:29:31.039 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:30.313+0000 7f1f6b0d2700 1 -- 172.21.15.98:0/1387765200 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1f64101890 msgr2=0x7f1f64196380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:29:31.040 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:30.313+0000 7f1f6b0d2700 1 --2- 172.21.15.98:0/1387765200 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1f64101890 0x7f1f64196380 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7f1f5000d950 tx=0x7f1f5000dc60 comp rx=0 tx=0).stop 2023-07-14T19:29:31.040 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:30.313+0000 7f1f6966f700 1 -- 172.21.15.98:0/1387765200 reap_dead start 2023-07-14T19:29:31.040 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:30.314+0000 7f1f6b0d2700 1 -- 172.21.15.98:0/1387765200 shutdown_connections 2023-07-14T19:29:31.040 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:30.314+0000 7f1f6b0d2700 1 -- 172.21.15.98:0/1387765200 >> 172.21.15.98:0/1387765200 conn(0x7f1f640fa720 msgr2=0x7f1f6410c4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:29:31.041 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:30.314+0000 7f1f6b0d2700 1 -- 172.21.15.98:0/1387765200 shutdown_connections 2023-07-14T19:29:31.041 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:30.314+0000 7f1f6b0d2700 1 -- 172.21.15.98:0/1387765200 wait complete. 2023-07-14T19:29:31.171 INFO:teuthology.orchestra.run.smithi098.stdout:55834574908 2023-07-14T19:29:31.172 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81c9387d2e7576da065747ec9c16d1c7d69016ec shell --fsid 9adea648-227b-11ee-9b33-001a4aab830c -- ceph osd last-stat-seq osd.1 2023-07-14T19:29:31.238 INFO:teuthology.orchestra.run.smithi098.stdout:244813135901 2023-07-14T19:29:31.238 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81c9387d2e7576da065747ec9c16d1c7d69016ec shell --fsid 9adea648-227b-11ee-9b33-001a4aab830c -- ceph osd last-stat-seq osd.8 2023-07-14T19:29:31.257 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:31.083+0000 7fa45d8d7700 1 -- 172.21.15.98:0/4174660347 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa458100f80 msgr2=0x7fa458101350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:29:31.258 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:31.083+0000 7fa45d8d7700 1 --2- 172.21.15.98:0/4174660347 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa458100f80 0x7fa458101350 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7fa448009990 tx=0x7fa448009ca0 comp rx=0 tx=0).stop 2023-07-14T19:29:31.258 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:31.083+0000 7fa45d8d7700 1 -- 172.21.15.98:0/4174660347 shutdown_connections 2023-07-14T19:29:31.258 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:31.083+0000 7fa45d8d7700 1 --2- 172.21.15.98:0/4174660347 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fa45810a2d0 0x7fa45810c730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:29:31.258 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:31.083+0000 7fa45d8d7700 1 --2- 172.21.15.98:0/4174660347 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa458100f80 0x7fa458101350 unknown :-1 s=CLOSED pgs=264 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:29:31.258 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:31.083+0000 7fa45d8d7700 1 --2- 172.21.15.98:0/4174660347 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7fa458101890 0x7fa458109d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:29:31.258 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:31.083+0000 7fa45d8d7700 1 -- 172.21.15.98:0/4174660347 >> 172.21.15.98:0/4174660347 conn(0x7fa4580fa720 msgr2=0x7fa4580fcb30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:29:31.259 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:31.083+0000 7fa45d8d7700 1 -- 172.21.15.98:0/4174660347 shutdown_connections 2023-07-14T19:29:31.259 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:31.083+0000 7fa45d8d7700 1 -- 172.21.15.98:0/4174660347 wait complete. 2023-07-14T19:29:31.259 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:31.084+0000 7fa45d8d7700 1 Processor -- start 2023-07-14T19:29:31.259 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:31.084+0000 7fa45d8d7700 1 -- start start 2023-07-14T19:29:31.259 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:31.084+0000 7fa45d8d7700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fa458100f80 0x7fa458195e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:29:31.259 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:31.084+0000 7fa45d8d7700 1 --2- >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7fa458101890 0x7fa458196380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:29:31.259 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:31.084+0000 7fa45d8d7700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa45810a2d0 0x7fa45819a7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:29:31.260 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:31.084+0000 7fa45d8d7700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa45819ae00 con 0x7fa45810a2d0 2023-07-14T19:29:31.260 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:31.084+0000 7fa45d8d7700 1 -- --> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa45819af70 con 0x7fa458101890 2023-07-14T19:29:31.260 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:31.084+0000 7fa45d8d7700 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa45819b130 con 0x7fa458100f80 2023-07-14T19:29:31.260 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:31.084+0000 7fa4577fe700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa45810a2d0 0x7fa45819a7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:29:31.260 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:31.084+0000 7fa4577fe700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa45810a2d0 0x7fa45819a7f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:46104/0 (socket says 172.21.15.98:46104) 2023-07-14T19:29:31.260 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:31.084+0000 7fa4577fe700 1 -- 172.21.15.98:0/463178686 learned_addr learned my addr 172.21.15.98:0/463178686 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-07-14T19:29:31.260 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:31.085+0000 7fa456ffd700 1 --2- 172.21.15.98:0/463178686 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fa458100f80 0x7fa458195e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:29:31.261 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:31.085+0000 7fa4567fc700 1 --2- 172.21.15.98:0/463178686 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7fa458101890 0x7fa458196380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:29:31.261 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:31.085+0000 7fa4577fe700 1 -- 172.21.15.98:0/463178686 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fa458100f80 msgr2=0x7fa458195e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:29:31.261 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:31.085+0000 7fa4577fe700 1 --2- 172.21.15.98:0/463178686 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fa458100f80 0x7fa458195e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:29:31.261 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:31.085+0000 7fa4577fe700 1 -- 172.21.15.98:0/463178686 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7fa458101890 msgr2=0x7fa458196380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:29:31.261 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:31.085+0000 7fa4577fe700 1 --2- 172.21.15.98:0/463178686 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7fa458101890 0x7fa458196380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:29:31.261 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:31.085+0000 7fa4577fe700 1 -- 172.21.15.98:0/463178686 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa4480095f0 con 0x7fa45810a2d0 2023-07-14T19:29:31.261 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:31.085+0000 7fa4567fc700 1 --2- 172.21.15.98:0/463178686 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7fa458101890 0x7fa458196380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-07-14T19:29:31.261 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:31.085+0000 7fa4577fe700 1 --2- 172.21.15.98:0/463178686 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa45810a2d0 0x7fa45819a7f0 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7fa45000b970 tx=0x7fa45000bd30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:29:31.262 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:31.085+0000 7fa45c8d5700 1 -- 172.21.15.98:0/463178686 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa45000c9b0 con 0x7fa45810a2d0 2023-07-14T19:29:31.262 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:31.085+0000 7fa45d8d7700 1 -- 172.21.15.98:0/463178686 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa45819b410 con 0x7fa45810a2d0 2023-07-14T19:29:31.262 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:31.085+0000 7fa45c8d5700 1 -- 172.21.15.98:0/463178686 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fa450014440 con 0x7fa45810a2d0 2023-07-14T19:29:31.262 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:31.085+0000 7fa45c8d5700 1 -- 172.21.15.98:0/463178686 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa4500075d0 con 0x7fa45810a2d0 2023-07-14T19:29:31.262 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:31.085+0000 7fa45d8d7700 1 -- 172.21.15.98:0/463178686 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa4581a2c70 con 0x7fa45810a2d0 2023-07-14T19:29:31.262 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:31.086+0000 7fa45d8d7700 1 -- 172.21.15.98:0/463178686 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fa4580682d0 con 0x7fa45810a2d0 2023-07-14T19:29:31.262 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:31.087+0000 7fa45c8d5700 1 -- 172.21.15.98:0/463178686 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 17) v1 ==== 135766+0+0 (secure 0 0 0) 0x7fa4500145b0 con 0x7fa45810a2d0 2023-07-14T19:29:31.263 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:31.088+0000 7fa45c8d5700 1 --2- 172.21.15.98:0/463178686 >> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] conn(0x7fa4400a11e0 0x7fa4400a3690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:29:31.263 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:31.088+0000 7fa456ffd700 1 --2- 172.21.15.98:0/463178686 >> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] conn(0x7fa4400a11e0 0x7fa4400a3690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:29:31.263 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:31.088+0000 7fa45c8d5700 1 -- 172.21.15.98:0/463178686 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 7979+0+0 (secure 0 0 0) 0x7fa450006e40 con 0x7fa45810a2d0 2023-07-14T19:29:31.263 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:31.088+0000 7fa45c8d5700 1 --2- 172.21.15.98:0/463178686 >> [v2:172.21.15.139:6808/3532263042,v1:172.21.15.139:6809/3532263042] conn(0x7fa4400aa9b0 0x7fa4400acdc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:29:31.263 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:31.088+0000 7fa45c8d5700 1 -- 172.21.15.98:0/463178686 --> [v2:172.21.15.139:6808/3532263042,v1:172.21.15.139:6809/3532263042] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fa4400ad470 con 0x7fa4400aa9b0 2023-07-14T19:29:31.263 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:31.088+0000 7fa45c8d5700 1 -- 172.21.15.98:0/463178686 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_get_version_reply(handle=1 version=83) v2 ==== 24+0+0 (secure 0 0 0) 0x7fa4500cc610 con 0x7fa45810a2d0 2023-07-14T19:29:31.263 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:31.088+0000 7fa456ffd700 1 --2- 172.21.15.98:0/463178686 >> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] conn(0x7fa4400a11e0 0x7fa4400a3690 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7fa448006010 tx=0x7fa44800b540 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:29:31.264 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:31.088+0000 7fa4567fc700 1 --2- 172.21.15.98:0/463178686 >> [v2:172.21.15.139:6808/3532263042,v1:172.21.15.139:6809/3532263042] conn(0x7fa4400aa9b0 0x7fa4400acdc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:29:31.264 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:31.089+0000 7fa4567fc700 1 --2- 172.21.15.98:0/463178686 >> [v2:172.21.15.139:6808/3532263042,v1:172.21.15.139:6809/3532263042] conn(0x7fa4400aa9b0 0x7fa4400acdc0 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:29:31.264 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:31.091+0000 7fa45c8d5700 1 -- 172.21.15.98:0/463178686 <== osd.5 v2:172.21.15.139:6808/3532263042 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25106 (crc 0 0 0) 0x7fa4400ad470 con 0x7fa4400aa9b0 2023-07-14T19:29:31.264 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:31.107+0000 7fa45d8d7700 1 -- 172.21.15.98:0/463178686 --> [v2:172.21.15.139:6808/3532263042,v1:172.21.15.139:6809/3532263042] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7fa45810ffd0 con 0x7fa4400aa9b0 2023-07-14T19:29:31.264 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:31.108+0000 7fa45c8d5700 1 -- 172.21.15.98:0/463178686 <== osd.5 v2:172.21.15.139:6808/3532263042 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+12 (crc 0 0 0) 0x7fa44c0074d0 con 0x7fa4400aa9b0 2023-07-14T19:29:31.265 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:31.108+0000 7fa45d8d7700 1 -- 172.21.15.98:0/463178686 >> [v2:172.21.15.139:6808/3532263042,v1:172.21.15.139:6809/3532263042] conn(0x7fa4400aa9b0 msgr2=0x7fa4400acdc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:29:31.265 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:31.108+0000 7fa45d8d7700 1 --2- 172.21.15.98:0/463178686 >> [v2:172.21.15.139:6808/3532263042,v1:172.21.15.139:6809/3532263042] conn(0x7fa4400aa9b0 0x7fa4400acdc0 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:29:31.265 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:31.108+0000 7fa45d8d7700 1 -- 172.21.15.98:0/463178686 >> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] conn(0x7fa4400a11e0 msgr2=0x7fa4400a3690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:29:31.265 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:31.108+0000 7fa45d8d7700 1 --2- 172.21.15.98:0/463178686 >> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] conn(0x7fa4400a11e0 0x7fa4400a3690 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7fa448006010 tx=0x7fa44800b540 comp rx=0 tx=0).stop 2023-07-14T19:29:31.265 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:31.108+0000 7fa45d8d7700 1 -- 172.21.15.98:0/463178686 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa45810a2d0 msgr2=0x7fa45819a7f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:29:31.265 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:31.108+0000 7fa45d8d7700 1 --2- 172.21.15.98:0/463178686 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa45810a2d0 0x7fa45819a7f0 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7fa45000b970 tx=0x7fa45000bd30 comp rx=0 tx=0).stop 2023-07-14T19:29:31.265 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:31.108+0000 7fa4577fe700 1 -- 172.21.15.98:0/463178686 reap_dead start 2023-07-14T19:29:31.266 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:31.108+0000 7fa45d8d7700 1 -- 172.21.15.98:0/463178686 shutdown_connections 2023-07-14T19:29:31.266 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:31.108+0000 7fa45d8d7700 1 -- 172.21.15.98:0/463178686 >> 172.21.15.98:0/463178686 conn(0x7fa4580fa720 msgr2=0x7fa45810c490 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:29:31.266 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:31.108+0000 7fa45d8d7700 1 -- 172.21.15.98:0/463178686 shutdown_connections 2023-07-14T19:29:31.266 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:31.108+0000 7fa45d8d7700 1 -- 172.21.15.98:0/463178686 wait complete. 2023-07-14T19:29:31.638 INFO:teuthology.orchestra.run.smithi098.stdout:77309411383 2023-07-14T19:29:31.639 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81c9387d2e7576da065747ec9c16d1c7d69016ec shell --fsid 9adea648-227b-11ee-9b33-001a4aab830c -- ceph osd last-stat-seq osd.2 2023-07-14T19:29:32.282 INFO:teuthology.orchestra.run.smithi098.stdout:158913789994 2023-07-14T19:29:32.282 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81c9387d2e7576da065747ec9c16d1c7d69016ec shell --fsid 9adea648-227b-11ee-9b33-001a4aab830c -- ceph osd last-stat-seq osd.5 2023-07-14T19:29:32.344 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/9adea648-227b-11ee-9b33-001a4aab830c/mon.a/config 2023-07-14T19:29:32.514 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/9adea648-227b-11ee-9b33-001a4aab830c/mon.a/config 2023-07-14T19:29:32.531 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:29:32 smithi098 ceph-mon[127486]: pgmap v268: 97 pgs: 97 active+clean; 634 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 255 B/s wr, 6 op/s 2023-07-14T19:29:32.609 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/9adea648-227b-11ee-9b33-001a4aab830c/mon.a/config 2023-07-14T19:29:32.681 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/9adea648-227b-11ee-9b33-001a4aab830c/mon.a/config 2023-07-14T19:29:32.681 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/9adea648-227b-11ee-9b33-001a4aab830c/mon.a/config 2023-07-14T19:29:32.691 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:29:32 smithi101 ceph-mon[128807]: pgmap v268: 97 pgs: 97 active+clean; 634 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 255 B/s wr, 6 op/s 2023-07-14T19:29:32.711 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/9adea648-227b-11ee-9b33-001a4aab830c/mon.a/config 2023-07-14T19:29:32.720 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:29:32 smithi139 ceph-mon[129293]: pgmap v268: 97 pgs: 97 active+clean; 634 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 255 B/s wr, 6 op/s 2023-07-14T19:29:32.827 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/9adea648-227b-11ee-9b33-001a4aab830c/mon.a/config 2023-07-14T19:29:33.173 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/9adea648-227b-11ee-9b33-001a4aab830c/mon.a/config 2023-07-14T19:29:34.891 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/9adea648-227b-11ee-9b33-001a4aab830c/mon.a/config 2023-07-14T19:29:34.941 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:29:34 smithi101 ceph-mon[128807]: pgmap v269: 97 pgs: 97 active+clean; 634 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 255 B/s wr, 5 op/s 2023-07-14T19:29:34.956 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/9adea648-227b-11ee-9b33-001a4aab830c/mon.a/config 2023-07-14T19:29:34.970 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:29:34 smithi139 ceph-mon[129293]: pgmap v269: 97 pgs: 97 active+clean; 634 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 255 B/s wr, 5 op/s 2023-07-14T19:29:35.008 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:29:34 smithi098 ceph-mon[127486]: pgmap v269: 97 pgs: 97 active+clean; 634 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 255 B/s wr, 5 op/s 2023-07-14T19:29:36.177 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/9adea648-227b-11ee-9b33-001a4aab830c/mon.a/config 2023-07-14T19:29:36.198 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:36.020+0000 7f67644fb700 1 -- 172.21.15.98:0/10721148 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f675c100f20 msgr2=0x7f675c1012f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:29:36.198 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:36.020+0000 7f67644fb700 1 --2- 172.21.15.98:0/10721148 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f675c100f20 0x7f675c1012f0 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7f674c009990 tx=0x7f674c009ca0 comp rx=0 tx=0).stop 2023-07-14T19:29:36.198 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:36.020+0000 7f67644fb700 1 -- 172.21.15.98:0/10721148 shutdown_connections 2023-07-14T19:29:36.198 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:36.020+0000 7f67644fb700 1 --2- 172.21.15.98:0/10721148 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f675c10a270 0x7f675c10c6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:29:36.199 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:36.020+0000 7f67644fb700 1 --2- 172.21.15.98:0/10721148 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f675c100f20 0x7f675c1012f0 unknown :-1 s=CLOSED pgs=266 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:29:36.199 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:36.020+0000 7f67644fb700 1 --2- 172.21.15.98:0/10721148 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f675c101830 0x7f675c109d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:29:36.199 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:36.020+0000 7f67644fb700 1 -- 172.21.15.98:0/10721148 >> 172.21.15.98:0/10721148 conn(0x7f675c075ca0 msgr2=0x7f675c0760a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:29:36.199 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:36.020+0000 7f67644fb700 1 -- 172.21.15.98:0/10721148 shutdown_connections 2023-07-14T19:29:36.199 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:36.020+0000 7f67644fb700 1 -- 172.21.15.98:0/10721148 wait complete. 2023-07-14T19:29:36.199 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:36.021+0000 7f67644fb700 1 Processor -- start 2023-07-14T19:29:36.199 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:36.021+0000 7f67644fb700 1 -- start start 2023-07-14T19:29:36.200 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:36.021+0000 7f67644fb700 1 --2- >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f675c100f20 0x7f675c19ba60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:29:36.200 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:36.021+0000 7f67644fb700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f675c101830 0x7f675c19bfa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:29:36.200 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:36.021+0000 7f67644fb700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f675c10a270 0x7f675c195ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:29:36.200 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:36.021+0000 7f67644fb700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f675c196150 con 0x7f675c10a270 2023-07-14T19:29:36.200 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:36.021+0000 7f67644fb700 1 -- --> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f675c1962c0 con 0x7f675c100f20 2023-07-14T19:29:36.200 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:36.021+0000 7f67644fb700 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f675c196480 con 0x7f675c101830 2023-07-14T19:29:36.200 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:36.021+0000 7f6762a98700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f675c10a270 0x7f675c195ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:29:36.201 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:36.021+0000 7f6762a98700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f675c10a270 0x7f675c195ae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:46118/0 (socket says 172.21.15.98:46118) 2023-07-14T19:29:36.201 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:36.021+0000 7f6762a98700 1 -- 172.21.15.98:0/1584866165 learned_addr learned my addr 172.21.15.98:0/1584866165 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-07-14T19:29:36.201 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:36.021+0000 7f6762297700 1 --2- 172.21.15.98:0/1584866165 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f675c100f20 0x7f675c19ba60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:29:36.201 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:36.021+0000 7f6761a96700 1 --2- 172.21.15.98:0/1584866165 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f675c101830 0x7f675c19bfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:29:36.201 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:36.021+0000 7f6762a98700 1 -- 172.21.15.98:0/1584866165 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f675c101830 msgr2=0x7f675c19bfa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:29:36.201 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:36.021+0000 7f6762a98700 1 --2- 172.21.15.98:0/1584866165 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f675c101830 0x7f675c19bfa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:29:36.202 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:36.021+0000 7f6762a98700 1 -- 172.21.15.98:0/1584866165 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f675c100f20 msgr2=0x7f675c19ba60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:29:36.202 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:36.021+0000 7f6762a98700 1 --2- 172.21.15.98:0/1584866165 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f675c100f20 0x7f675c19ba60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:29:36.202 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:36.021+0000 7f6762a98700 1 -- 172.21.15.98:0/1584866165 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f674c0095f0 con 0x7f675c10a270 2023-07-14T19:29:36.202 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:36.022+0000 7f6762a98700 1 --2- 172.21.15.98:0/1584866165 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f675c10a270 0x7f675c195ae0 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7f6754007f10 tx=0x7f675400d3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:29:36.202 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:36.022+0000 7f67537fe700 1 -- 172.21.15.98:0/1584866165 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6754017070 con 0x7f675c10a270 2023-07-14T19:29:36.202 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:36.022+0000 7f67537fe700 1 -- 172.21.15.98:0/1584866165 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f675400f040 con 0x7f675c10a270 2023-07-14T19:29:36.202 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:36.022+0000 7f67537fe700 1 -- 172.21.15.98:0/1584866165 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f67540054b0 con 0x7f675c10a270 2023-07-14T19:29:36.202 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:36.022+0000 7f67644fb700 1 -- 172.21.15.98:0/1584866165 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f675c196760 con 0x7f675c10a270 2023-07-14T19:29:36.203 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:36.022+0000 7f67644fb700 1 -- 172.21.15.98:0/1584866165 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f675c1a2a50 con 0x7f675c10a270 2023-07-14T19:29:36.203 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:36.023+0000 7f67644fb700 1 -- 172.21.15.98:0/1584866165 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f675c04e340 con 0x7f675c10a270 2023-07-14T19:29:36.203 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:36.024+0000 7f67537fe700 1 -- 172.21.15.98:0/1584866165 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 17) v1 ==== 135766+0+0 (secure 0 0 0) 0x7f6754004ad0 con 0x7f675c10a270 2023-07-14T19:29:36.203 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:36.024+0000 7f67537fe700 1 --2- 172.21.15.98:0/1584866165 >> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] conn(0x7f67480a1260 0x7f67480a3710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:29:36.203 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:36.025+0000 7f6762297700 1 --2- 172.21.15.98:0/1584866165 >> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] conn(0x7f67480a1260 0x7f67480a3710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:29:36.203 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:36.025+0000 7f67537fe700 1 -- 172.21.15.98:0/1584866165 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 7979+0+0 (secure 0 0 0) 0x7f67540cc5f0 con 0x7f675c10a270 2023-07-14T19:29:36.203 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:36.025+0000 7f6762297700 1 --2- 172.21.15.98:0/1584866165 >> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] conn(0x7f67480a1260 0x7f67480a3710 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f674c00b5c0 tx=0x7f674c005960 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:29:36.204 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:36.028+0000 7f67537fe700 1 -- 172.21.15.98:0/1584866165 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f6754095c90 con 0x7f675c10a270 2023-07-14T19:29:36.229 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:36.228+0000 7f67644fb700 1 -- 172.21.15.98:0/1584866165 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 11} v 0) v1 -- 0x7f675c196ff0 con 0x7f675c10a270 2023-07-14T19:29:36.230 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:36.228+0000 7f67537fe700 1 -- 172.21.15.98:0/1584866165 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 11}]=0 v0) v1 ==== 75+0+13 (secure 0 0 0) 0x7f67540992b0 con 0x7f675c10a270 2023-07-14T19:29:36.230 INFO:teuthology.orchestra.run.smithi098.stdout:335007449105 2023-07-14T19:29:36.232 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:36.230+0000 7f67644fb700 1 -- 172.21.15.98:0/1584866165 >> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] conn(0x7f67480a1260 msgr2=0x7f67480a3710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:29:36.232 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:36.230+0000 7f67644fb700 1 --2- 172.21.15.98:0/1584866165 >> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] conn(0x7f67480a1260 0x7f67480a3710 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f674c00b5c0 tx=0x7f674c005960 comp rx=0 tx=0).stop 2023-07-14T19:29:36.232 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:36.230+0000 7f67644fb700 1 -- 172.21.15.98:0/1584866165 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f675c10a270 msgr2=0x7f675c195ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:29:36.232 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:36.230+0000 7f67644fb700 1 --2- 172.21.15.98:0/1584866165 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f675c10a270 0x7f675c195ae0 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7f6754007f10 tx=0x7f675400d3b0 comp rx=0 tx=0).stop 2023-07-14T19:29:36.232 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:36.230+0000 7f67644fb700 1 -- 172.21.15.98:0/1584866165 shutdown_connections 2023-07-14T19:29:36.233 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:36.230+0000 7f67644fb700 1 --2- 172.21.15.98:0/1584866165 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f675c10a270 0x7f675c195ae0 unknown :-1 s=CLOSED pgs=267 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:29:36.233 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:36.230+0000 7f67644fb700 1 --2- 172.21.15.98:0/1584866165 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f675c100f20 0x7f675c19ba60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:29:36.233 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:36.230+0000 7f67644fb700 1 --2- 172.21.15.98:0/1584866165 >> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] conn(0x7f67480a1260 0x7f67480a3710 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:29:36.233 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:36.230+0000 7f67644fb700 1 --2- 172.21.15.98:0/1584866165 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f675c101830 0x7f675c19bfa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:29:36.233 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:36.230+0000 7f67644fb700 1 -- 172.21.15.98:0/1584866165 >> 172.21.15.98:0/1584866165 conn(0x7f675c075ca0 msgr2=0x7f675c0fd4b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:29:36.233 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:36.230+0000 7f67644fb700 1 -- 172.21.15.98:0/1584866165 shutdown_connections 2023-07-14T19:29:36.233 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:36.230+0000 7f67644fb700 1 -- 172.21.15.98:0/1584866165 wait complete. 2023-07-14T19:29:36.728 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:29:36 smithi098 ceph-mon[127486]: pgmap v270: 97 pgs: 97 active+clean; 634 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 255 B/s wr, 6 op/s 2023-07-14T19:29:36.728 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:29:36 smithi098 ceph-mon[127486]: from='client.? 172.21.15.98:0/1584866165' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 11}]: dispatch 2023-07-14T19:29:36.734 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/9adea648-227b-11ee-9b33-001a4aab830c/mon.a/config 2023-07-14T19:29:36.903 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:36.901+0000 7fa17c75a700 1 -- 172.21.15.98:0/2459314870 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa174103f10 msgr2=0x7fa17410ba90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:29:36.903 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:36.901+0000 7fa17c75a700 1 --2- 172.21.15.98:0/2459314870 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa174103f10 0x7fa17410ba90 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7fa16400b0d0 tx=0x7fa16400b3e0 comp rx=0 tx=0).stop 2023-07-14T19:29:36.903 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:36.901+0000 7fa17c75a700 1 -- 172.21.15.98:0/2459314870 shutdown_connections 2023-07-14T19:29:36.903 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:36.901+0000 7fa17c75a700 1 --2- 172.21.15.98:0/2459314870 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fa1740fee70 0x7fa1740ff240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:29:36.903 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:36.901+0000 7fa17c75a700 1 --2- 172.21.15.98:0/2459314870 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa174103f10 0x7fa17410ba90 unknown :-1 s=CLOSED pgs=268 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:29:36.904 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:36.901+0000 7fa17c75a700 1 --2- 172.21.15.98:0/2459314870 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7fa1740ff810 0x7fa1741037e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:29:36.904 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:36.901+0000 7fa17c75a700 1 -- 172.21.15.98:0/2459314870 >> 172.21.15.98:0/2459314870 conn(0x7fa174075ae0 msgr2=0x7fa174075ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:29:36.904 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:36.901+0000 7fa17c75a700 1 -- 172.21.15.98:0/2459314870 shutdown_connections 2023-07-14T19:29:36.904 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:36.901+0000 7fa17c75a700 1 -- 172.21.15.98:0/2459314870 wait complete. 2023-07-14T19:29:36.905 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:36.902+0000 7fa17c75a700 1 Processor -- start 2023-07-14T19:29:36.906 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:36.902+0000 7fa17c75a700 1 -- start start 2023-07-14T19:29:36.906 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:36.902+0000 7fa17c75a700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fa1740fee70 0x7fa174195ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:29:36.906 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:36.902+0000 7fa17c75a700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa1740ff810 0x7fa174196430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:29:36.906 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:36.902+0000 7fa17c75a700 1 --2- >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7fa174103f10 0x7fa17419a8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:29:36.906 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:36.902+0000 7fa17c75a700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa17419aeb0 con 0x7fa1740ff810 2023-07-14T19:29:36.906 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:36.902+0000 7fa17c75a700 1 -- --> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa17419b020 con 0x7fa174103f10 2023-07-14T19:29:36.906 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:36.902+0000 7fa17c75a700 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa17419b1e0 con 0x7fa1740fee70 2023-07-14T19:29:36.907 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:36.902+0000 7fa179cf5700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa1740ff810 0x7fa174196430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:29:36.907 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:36.902+0000 7fa179cf5700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa1740ff810 0x7fa174196430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:46128/0 (socket says 172.21.15.98:46128) 2023-07-14T19:29:36.907 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:36.902+0000 7fa179cf5700 1 -- 172.21.15.98:0/888473501 learned_addr learned my addr 172.21.15.98:0/888473501 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-07-14T19:29:36.907 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:36.902+0000 7fa17acf7700 1 --2- 172.21.15.98:0/888473501 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7fa174103f10 0x7fa17419a8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:29:36.907 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:36.902+0000 7fa17a4f6700 1 --2- 172.21.15.98:0/888473501 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fa1740fee70 0x7fa174195ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:29:36.907 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:36.902+0000 7fa179cf5700 1 -- 172.21.15.98:0/888473501 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fa1740fee70 msgr2=0x7fa174195ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:29:36.907 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:36.902+0000 7fa179cf5700 1 --2- 172.21.15.98:0/888473501 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fa1740fee70 0x7fa174195ef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:29:36.907 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:36.902+0000 7fa179cf5700 1 -- 172.21.15.98:0/888473501 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7fa174103f10 msgr2=0x7fa17419a8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:29:36.908 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:36.902+0000 7fa179cf5700 1 --2- 172.21.15.98:0/888473501 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7fa174103f10 0x7fa17419a8a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:29:36.908 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:36.902+0000 7fa179cf5700 1 -- 172.21.15.98:0/888473501 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa164009d00 con 0x7fa1740ff810 2023-07-14T19:29:36.908 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:36.902+0000 7fa179cf5700 1 --2- 172.21.15.98:0/888473501 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa1740ff810 0x7fa174196430 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7fa16800c8f0 tx=0x7fa16800ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:29:36.908 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:36.903+0000 7fa1737fe700 1 -- 172.21.15.98:0/888473501 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa168007b00 con 0x7fa1740ff810 2023-07-14T19:29:36.908 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:36.903+0000 7fa1737fe700 1 -- 172.21.15.98:0/888473501 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fa168007c60 con 0x7fa1740ff810 2023-07-14T19:29:36.908 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:36.903+0000 7fa17c75a700 1 -- 172.21.15.98:0/888473501 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa17419b4c0 con 0x7fa1740ff810 2023-07-14T19:29:36.908 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:36.903+0000 7fa1737fe700 1 -- 172.21.15.98:0/888473501 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa168018660 con 0x7fa1740ff810 2023-07-14T19:29:36.909 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:36.903+0000 7fa17c75a700 1 -- 172.21.15.98:0/888473501 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa1741a2d20 con 0x7fa1740ff810 2023-07-14T19:29:36.909 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:36.904+0000 7fa17c75a700 1 -- 172.21.15.98:0/888473501 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa17404e340 con 0x7fa1740ff810 2023-07-14T19:29:36.909 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:36.905+0000 7fa1737fe700 1 -- 172.21.15.98:0/888473501 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 17) v1 ==== 135766+0+0 (secure 0 0 0) 0x7fa16800f450 con 0x7fa1740ff810 2023-07-14T19:29:36.910 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:36.905+0000 7fa1737fe700 1 --2- 172.21.15.98:0/888473501 >> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] conn(0x7fa15c0a11e0 0x7fa15c0a3690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:29:36.910 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:36.905+0000 7fa1737fe700 1 -- 172.21.15.98:0/888473501 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 7979+0+0 (secure 0 0 0) 0x7fa1680cc8c0 con 0x7fa1740ff810 2023-07-14T19:29:36.910 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:36.905+0000 7fa17a4f6700 1 --2- 172.21.15.98:0/888473501 >> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] conn(0x7fa15c0a11e0 0x7fa15c0a3690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:29:36.910 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:36.906+0000 7fa17a4f6700 1 --2- 172.21.15.98:0/888473501 >> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] conn(0x7fa15c0a11e0 0x7fa15c0a3690 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7fa16c005950 tx=0x7fa16c005cf0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:29:36.910 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:36.909+0000 7fa1737fe700 1 -- 172.21.15.98:0/888473501 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7fa1680960f0 con 0x7fa1740ff810 2023-07-14T19:29:36.941 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:29:36 smithi101 ceph-mon[128807]: pgmap v270: 97 pgs: 97 active+clean; 634 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 255 B/s wr, 6 op/s 2023-07-14T19:29:36.941 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:29:36 smithi101 ceph-mon[128807]: from='client.? 172.21.15.98:0/1584866165' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 11}]: dispatch 2023-07-14T19:29:36.970 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:29:36 smithi139 ceph-mon[129293]: pgmap v270: 97 pgs: 97 active+clean; 634 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 255 B/s wr, 6 op/s 2023-07-14T19:29:36.970 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:29:36 smithi139 ceph-mon[129293]: from='client.? 172.21.15.98:0/1584866165' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 11}]: dispatch 2023-07-14T19:29:37.109 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:37.107+0000 7fa17c75a700 1 -- 172.21.15.98:0/888473501 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 4} v 0) v1 -- 0x7fa1741a2ff0 con 0x7fa1740ff810 2023-07-14T19:29:37.109 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:37.108+0000 7fa1737fe700 1 -- 172.21.15.98:0/888473501 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 4}]=0 v0) v1 ==== 74+0+13 (secure 0 0 0) 0x7fa168099710 con 0x7fa1740ff810 2023-07-14T19:29:37.109 INFO:teuthology.orchestra.run.smithi098.stdout:137438953519 2023-07-14T19:29:37.111 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:37.109+0000 7fa17c75a700 1 -- 172.21.15.98:0/888473501 >> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] conn(0x7fa15c0a11e0 msgr2=0x7fa15c0a3690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:29:37.111 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:37.109+0000 7fa17c75a700 1 --2- 172.21.15.98:0/888473501 >> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] conn(0x7fa15c0a11e0 0x7fa15c0a3690 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7fa16c005950 tx=0x7fa16c005cf0 comp rx=0 tx=0).stop 2023-07-14T19:29:37.111 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:37.109+0000 7fa17c75a700 1 -- 172.21.15.98:0/888473501 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa1740ff810 msgr2=0x7fa174196430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:29:37.112 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:37.109+0000 7fa17c75a700 1 --2- 172.21.15.98:0/888473501 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa1740ff810 0x7fa174196430 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7fa16800c8f0 tx=0x7fa16800ccb0 comp rx=0 tx=0).stop 2023-07-14T19:29:37.112 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:37.110+0000 7fa17c75a700 1 -- 172.21.15.98:0/888473501 shutdown_connections 2023-07-14T19:29:37.112 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:37.110+0000 7fa17c75a700 1 --2- 172.21.15.98:0/888473501 >> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] conn(0x7fa15c0a11e0 0x7fa15c0a3690 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:29:37.112 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:37.110+0000 7fa17c75a700 1 --2- 172.21.15.98:0/888473501 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fa1740fee70 0x7fa174195ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:29:37.112 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:37.110+0000 7fa17c75a700 1 --2- 172.21.15.98:0/888473501 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7fa174103f10 0x7fa17419a8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:29:37.112 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:37.110+0000 7fa17c75a700 1 --2- 172.21.15.98:0/888473501 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa1740ff810 0x7fa174196430 unknown :-1 s=CLOSED pgs=269 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:29:37.112 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:37.110+0000 7fa17c75a700 1 -- 172.21.15.98:0/888473501 >> 172.21.15.98:0/888473501 conn(0x7fa174075ae0 msgr2=0x7fa174068aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:29:37.113 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:37.110+0000 7fa17c75a700 1 -- 172.21.15.98:0/888473501 shutdown_connections 2023-07-14T19:29:37.113 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:37.110+0000 7fa17c75a700 1 -- 172.21.15.98:0/888473501 wait complete. 2023-07-14T19:29:37.870 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:37.868+0000 7f301c946700 1 -- 172.21.15.98:0/4039087075 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f30180ff7e0 msgr2=0x7f30181037b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:29:37.870 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:37.868+0000 7f301c946700 1 --2- 172.21.15.98:0/4039087075 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f30180ff7e0 0x7f30181037b0 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7f300c0099c0 tx=0x7f300c009cd0 comp rx=0 tx=0).stop 2023-07-14T19:29:37.870 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:37.868+0000 7f301c946700 1 -- 172.21.15.98:0/4039087075 shutdown_connections 2023-07-14T19:29:37.870 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:37.868+0000 7f301c946700 1 --2- 172.21.15.98:0/4039087075 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f30180fee40 0x7f30180ff210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:29:37.871 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:37.868+0000 7f301c946700 1 --2- 172.21.15.98:0/4039087075 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f3018103ee0 0x7f301810ba60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:29:37.871 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:37.868+0000 7f301c946700 1 --2- 172.21.15.98:0/4039087075 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f30180ff7e0 0x7f30181037b0 unknown :-1 s=CLOSED pgs=270 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:29:37.871 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:37.868+0000 7f301c946700 1 -- 172.21.15.98:0/4039087075 >> 172.21.15.98:0/4039087075 conn(0x7f3018075ca0 msgr2=0x7f30180760a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:29:37.871 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:37.868+0000 7f301c946700 1 -- 172.21.15.98:0/4039087075 shutdown_connections 2023-07-14T19:29:37.871 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:37.868+0000 7f301c946700 1 -- 172.21.15.98:0/4039087075 wait complete. 2023-07-14T19:29:37.873 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:37.869+0000 7f301c946700 1 Processor -- start 2023-07-14T19:29:37.873 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:37.869+0000 7f301c946700 1 -- start start 2023-07-14T19:29:37.873 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:37.869+0000 7f301c946700 1 --2- >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f30180fee40 0x7f3018196ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:29:37.874 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:37.869+0000 7f301c946700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f30180ff7e0 0x7f30181970e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:29:37.874 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:37.869+0000 7f301c946700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3018103ee0 0x7f30181a1c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:29:37.874 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:37.869+0000 7f301c946700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f30181a2160 con 0x7f3018103ee0 2023-07-14T19:29:37.874 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:37.869+0000 7f301c946700 1 -- --> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f30181a22a0 con 0x7f30180fee40 2023-07-14T19:29:37.874 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:37.869+0000 7f301c946700 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f30181a23e0 con 0x7f30180ff7e0 2023-07-14T19:29:37.874 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:37.869+0000 7f3016d9d700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3018103ee0 0x7f30181a1c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:29:37.874 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:37.869+0000 7f3016d9d700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3018103ee0 0x7f30181a1c20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:46142/0 (socket says 172.21.15.98:46142) 2023-07-14T19:29:37.874 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:37.869+0000 7f3016d9d700 1 -- 172.21.15.98:0/1410355709 learned_addr learned my addr 172.21.15.98:0/1410355709 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-07-14T19:29:37.875 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:37.869+0000 7f301659c700 1 --2- 172.21.15.98:0/1410355709 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f30180fee40 0x7f3018196ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:29:37.875 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:37.869+0000 7f3015d9b700 1 --2- 172.21.15.98:0/1410355709 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f30180ff7e0 0x7f30181970e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:29:37.875 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:37.870+0000 7f3016d9d700 1 -- 172.21.15.98:0/1410355709 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f30180ff7e0 msgr2=0x7f30181970e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:29:37.875 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:37.870+0000 7f3016d9d700 1 --2- 172.21.15.98:0/1410355709 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f30180ff7e0 0x7f30181970e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:29:37.875 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:37.870+0000 7f3016d9d700 1 -- 172.21.15.98:0/1410355709 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f30180fee40 msgr2=0x7f3018196ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:29:37.875 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:37.870+0000 7f3016d9d700 1 --2- 172.21.15.98:0/1410355709 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f30180fee40 0x7f3018196ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:29:37.875 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:37.870+0000 7f3016d9d700 1 -- 172.21.15.98:0/1410355709 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f300c0095f0 con 0x7f3018103ee0 2023-07-14T19:29:37.876 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:37.870+0000 7f3015d9b700 1 --2- 172.21.15.98:0/1410355709 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f30180ff7e0 0x7f30181970e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-07-14T19:29:37.876 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:37.870+0000 7f3016d9d700 1 --2- 172.21.15.98:0/1410355709 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3018103ee0 0x7f30181a1c20 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7f301000ab50 tx=0x7f301000ae60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:29:37.876 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:37.870+0000 7f30077fe700 1 -- 172.21.15.98:0/1410355709 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f301000f070 con 0x7f3018103ee0 2023-07-14T19:29:37.876 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:37.870+0000 7f301c946700 1 -- 172.21.15.98:0/1410355709 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f30181a2640 con 0x7f3018103ee0 2023-07-14T19:29:37.877 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:37.870+0000 7f30077fe700 1 -- 172.21.15.98:0/1410355709 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f3010007a80 con 0x7f3018103ee0 2023-07-14T19:29:37.877 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:37.870+0000 7f30077fe700 1 -- 172.21.15.98:0/1410355709 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3010005280 con 0x7f3018103ee0 2023-07-14T19:29:37.877 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:37.870+0000 7f301c946700 1 -- 172.21.15.98:0/1410355709 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f30181a2b10 con 0x7f3018103ee0 2023-07-14T19:29:37.877 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:37.871+0000 7f301c946700 1 -- 172.21.15.98:0/1410355709 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f301804e340 con 0x7f3018103ee0 2023-07-14T19:29:37.878 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:37.872+0000 7f30077fe700 1 -- 172.21.15.98:0/1410355709 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 17) v1 ==== 135766+0+0 (secure 0 0 0) 0x7f301000c620 con 0x7f3018103ee0 2023-07-14T19:29:37.878 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:37.873+0000 7f30077fe700 1 --2- 172.21.15.98:0/1410355709 >> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] conn(0x7f30000a11e0 0x7f30000a3690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:29:37.878 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:37.873+0000 7f301659c700 1 --2- 172.21.15.98:0/1410355709 >> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] conn(0x7f30000a11e0 0x7f30000a3690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:29:37.879 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:37.873+0000 7f30077fe700 1 -- 172.21.15.98:0/1410355709 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 7979+0+0 (secure 0 0 0) 0x7f30100cc3e0 con 0x7f3018103ee0 2023-07-14T19:29:37.879 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:37.873+0000 7f301659c700 1 --2- 172.21.15.98:0/1410355709 >> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] conn(0x7f30000a11e0 0x7f30000a3690 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f300800ac50 tx=0x7f300800a380 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:29:37.879 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:37.876+0000 7f30077fe700 1 -- 172.21.15.98:0/1410355709 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f3010095b10 con 0x7f3018103ee0 2023-07-14T19:29:37.941 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:29:37 smithi101 ceph-mon[128807]: from='client.? 172.21.15.98:0/888473501' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2023-07-14T19:29:37.949 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:29:37 smithi098 ceph-mon[127486]: from='client.? 172.21.15.98:0/888473501' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2023-07-14T19:29:37.970 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:29:37 smithi139 ceph-mon[129293]: from='client.? 172.21.15.98:0/888473501' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2023-07-14T19:29:38.070 INFO:tasks.cephadm.ceph_manager.ceph:need seq 335007449104 got 335007449105 for osd.11 2023-07-14T19:29:38.071 DEBUG:teuthology.parallel:result is None 2023-07-14T19:29:38.092 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:38.090+0000 7f301c946700 1 -- 172.21.15.98:0/1410355709 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 0} v 0) v1 -- 0x7f30181a2e00 con 0x7f3018103ee0 2023-07-14T19:29:38.092 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:38.091+0000 7f30077fe700 1 -- 172.21.15.98:0/1410355709 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 0}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f3010099130 con 0x7f3018103ee0 2023-07-14T19:29:38.093 INFO:teuthology.orchestra.run.smithi098.stdout:34359738433 2023-07-14T19:29:38.094 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:38.093+0000 7f301c946700 1 -- 172.21.15.98:0/1410355709 >> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] conn(0x7f30000a11e0 msgr2=0x7f30000a3690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:29:38.095 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:38.093+0000 7f301c946700 1 --2- 172.21.15.98:0/1410355709 >> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] conn(0x7f30000a11e0 0x7f30000a3690 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f300800ac50 tx=0x7f300800a380 comp rx=0 tx=0).stop 2023-07-14T19:29:38.095 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:38.093+0000 7f301c946700 1 -- 172.21.15.98:0/1410355709 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3018103ee0 msgr2=0x7f30181a1c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:29:38.095 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:38.093+0000 7f301c946700 1 --2- 172.21.15.98:0/1410355709 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3018103ee0 0x7f30181a1c20 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7f301000ab50 tx=0x7f301000ae60 comp rx=0 tx=0).stop 2023-07-14T19:29:38.095 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:38.093+0000 7f301c946700 1 -- 172.21.15.98:0/1410355709 shutdown_connections 2023-07-14T19:29:38.095 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:38.093+0000 7f301c946700 1 --2- 172.21.15.98:0/1410355709 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3018103ee0 0x7f30181a1c20 unknown :-1 s=CLOSED pgs=271 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:29:38.095 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:38.093+0000 7f301c946700 1 --2- 172.21.15.98:0/1410355709 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f30180fee40 0x7f3018196ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:29:38.096 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:38.093+0000 7f301c946700 1 --2- 172.21.15.98:0/1410355709 >> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] conn(0x7f30000a11e0 0x7f30000a3690 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:29:38.096 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:38.093+0000 7f301c946700 1 --2- 172.21.15.98:0/1410355709 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f30180ff7e0 0x7f30181970e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:29:38.096 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:38.093+0000 7f301c946700 1 -- 172.21.15.98:0/1410355709 >> 172.21.15.98:0/1410355709 conn(0x7f3018075ca0 msgr2=0x7f30180fc4f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:29:38.096 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:38.093+0000 7f301c946700 1 -- 172.21.15.98:0/1410355709 shutdown_connections 2023-07-14T19:29:38.096 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:38.093+0000 7f301c946700 1 -- 172.21.15.98:0/1410355709 wait complete. 2023-07-14T19:29:38.687 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:38.685+0000 7f9c24d37700 1 -- 172.21.15.98:0/4283157151 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9c201007a0 msgr2=0x7f9c20100c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:29:38.687 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:38.685+0000 7f9c24d37700 1 --2- 172.21.15.98:0/4283157151 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9c201007a0 0x7f9c20100c10 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7f9c140099c0 tx=0x7f9c14009cd0 comp rx=0 tx=0).stop 2023-07-14T19:29:38.688 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:38.685+0000 7f9c24d37700 1 -- 172.21.15.98:0/4283157151 shutdown_connections 2023-07-14T19:29:38.688 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:38.685+0000 7f9c24d37700 1 --2- 172.21.15.98:0/4283157151 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f9c20101150 0x7f9c2010bb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:29:38.688 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:38.685+0000 7f9c24d37700 1 --2- 172.21.15.98:0/4283157151 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f9c200fef40 0x7f9c200ff310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:29:38.688 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:38.685+0000 7f9c24d37700 1 --2- 172.21.15.98:0/4283157151 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9c201007a0 0x7f9c20100c10 unknown :-1 s=CLOSED pgs=274 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:29:38.688 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:38.685+0000 7f9c24d37700 1 -- 172.21.15.98:0/4283157151 >> 172.21.15.98:0/4283157151 conn(0x7f9c20075c90 msgr2=0x7f9c20076090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:29:38.688 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:38.685+0000 7f9c24d37700 1 -- 172.21.15.98:0/4283157151 shutdown_connections 2023-07-14T19:29:38.689 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:38.685+0000 7f9c24d37700 1 -- 172.21.15.98:0/4283157151 wait complete. 2023-07-14T19:29:38.690 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:38.686+0000 7f9c24d37700 1 Processor -- start 2023-07-14T19:29:38.690 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:38.686+0000 7f9c24d37700 1 -- start start 2023-07-14T19:29:38.691 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:38.686+0000 7f9c24d37700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9c200fef40 0x7f9c20194d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:29:38.691 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:38.686+0000 7f9c24d37700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f9c201007a0 0x7f9c201952a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:29:38.691 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:38.686+0000 7f9c24d37700 1 --2- >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f9c20101150 0x7f9c20199710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:29:38.691 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:38.686+0000 7f9c24d37700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9c20199d20 con 0x7f9c200fef40 2023-07-14T19:29:38.691 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:38.686+0000 7f9c24d37700 1 -- --> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9c20199e90 con 0x7f9c20101150 2023-07-14T19:29:38.691 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:38.686+0000 7f9c24d37700 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9c2019a050 con 0x7f9c201007a0 2023-07-14T19:29:38.691 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:38.686+0000 7f9c1e59c700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9c200fef40 0x7f9c20194d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:29:38.692 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:38.686+0000 7f9c1e59c700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9c200fef40 0x7f9c20194d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:46166/0 (socket says 172.21.15.98:46166) 2023-07-14T19:29:38.692 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:38.686+0000 7f9c1e59c700 1 -- 172.21.15.98:0/1726745075 learned_addr learned my addr 172.21.15.98:0/1726745075 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-07-14T19:29:38.692 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:38.686+0000 7f9c1dd9b700 1 --2- 172.21.15.98:0/1726745075 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f9c201007a0 0x7f9c201952a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:29:38.692 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:38.686+0000 7f9c1ed9d700 1 --2- 172.21.15.98:0/1726745075 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f9c20101150 0x7f9c20199710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:29:38.692 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:38.687+0000 7f9c1e59c700 1 -- 172.21.15.98:0/1726745075 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f9c201007a0 msgr2=0x7f9c201952a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:29:38.692 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:38.687+0000 7f9c1e59c700 1 --2- 172.21.15.98:0/1726745075 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f9c201007a0 0x7f9c201952a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:29:38.693 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:38.687+0000 7f9c1e59c700 1 -- 172.21.15.98:0/1726745075 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f9c20101150 msgr2=0x7f9c20199710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:29:38.693 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:38.687+0000 7f9c1e59c700 1 --2- 172.21.15.98:0/1726745075 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f9c20101150 0x7f9c20199710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:29:38.693 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:38.687+0000 7f9c1e59c700 1 -- 172.21.15.98:0/1726745075 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9c140095f0 con 0x7f9c200fef40 2023-07-14T19:29:38.693 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:38.687+0000 7f9c1e59c700 1 --2- 172.21.15.98:0/1726745075 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9c200fef40 0x7f9c20194d60 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7f9c1000d980 tx=0x7f9c1000dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:29:38.693 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:38.687+0000 7f9c0f7fe700 1 -- 172.21.15.98:0/1726745075 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9c10004a30 con 0x7f9c200fef40 2023-07-14T19:29:38.693 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:38.687+0000 7f9c0f7fe700 1 -- 172.21.15.98:0/1726745075 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f9c100054e0 con 0x7f9c200fef40 2023-07-14T19:29:38.693 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:38.687+0000 7f9c24d37700 1 -- 172.21.15.98:0/1726745075 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9c2019a330 con 0x7f9c200fef40 2023-07-14T19:29:38.694 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:38.687+0000 7f9c0f7fe700 1 -- 172.21.15.98:0/1726745075 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9c10009c90 con 0x7f9c200fef40 2023-07-14T19:29:38.694 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:38.687+0000 7f9c24d37700 1 -- 172.21.15.98:0/1726745075 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9c201a1b90 con 0x7f9c200fef40 2023-07-14T19:29:38.694 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:38.688+0000 7f9c24d37700 1 -- 172.21.15.98:0/1726745075 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9c2004e340 con 0x7f9c200fef40 2023-07-14T19:29:38.694 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:38.689+0000 7f9c0f7fe700 1 -- 172.21.15.98:0/1726745075 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 17) v1 ==== 135766+0+0 (secure 0 0 0) 0x7f9c10004c20 con 0x7f9c200fef40 2023-07-14T19:29:38.695 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:38.689+0000 7f9c0f7fe700 1 --2- 172.21.15.98:0/1726745075 >> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] conn(0x7f9c080a11e0 0x7f9c080a3690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:29:38.695 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:38.690+0000 7f9c1dd9b700 1 --2- 172.21.15.98:0/1726745075 >> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] conn(0x7f9c080a11e0 0x7f9c080a3690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:29:38.695 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:38.690+0000 7f9c0f7fe700 1 -- 172.21.15.98:0/1726745075 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 7979+0+0 (secure 0 0 0) 0x7f9c100ccc10 con 0x7f9c200fef40 2023-07-14T19:29:38.695 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:38.690+0000 7f9c1dd9b700 1 --2- 172.21.15.98:0/1726745075 >> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] conn(0x7f9c080a11e0 0x7f9c080a3690 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f9c14000c00 tx=0x7f9c14003680 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:29:38.696 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:38.694+0000 7f9c0f7fe700 1 -- 172.21.15.98:0/1726745075 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f9c100963b0 con 0x7f9c200fef40 2023-07-14T19:29:38.892 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:38.890+0000 7f9c24d37700 1 -- 172.21.15.98:0/1726745075 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 9} v 0) v1 -- 0x7f9c201a1d20 con 0x7f9c200fef40 2023-07-14T19:29:38.892 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:38.890+0000 7f9c0f7fe700 1 -- 172.21.15.98:0/1726745075 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 9}]=0 v0) v1 ==== 74+0+13 (secure 0 0 0) 0x7f9c10017070 con 0x7f9c200fef40 2023-07-14T19:29:38.893 INFO:teuthology.orchestra.run.smithi098.stdout:279172874267 2023-07-14T19:29:38.894 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:38.892+0000 7f9c24d37700 1 -- 172.21.15.98:0/1726745075 >> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] conn(0x7f9c080a11e0 msgr2=0x7f9c080a3690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:29:38.894 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:38.892+0000 7f9c24d37700 1 --2- 172.21.15.98:0/1726745075 >> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] conn(0x7f9c080a11e0 0x7f9c080a3690 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f9c14000c00 tx=0x7f9c14003680 comp rx=0 tx=0).stop 2023-07-14T19:29:38.894 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:38.892+0000 7f9c24d37700 1 -- 172.21.15.98:0/1726745075 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9c200fef40 msgr2=0x7f9c20194d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:29:38.895 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:38.892+0000 7f9c24d37700 1 --2- 172.21.15.98:0/1726745075 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9c200fef40 0x7f9c20194d60 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7f9c1000d980 tx=0x7f9c1000dc90 comp rx=0 tx=0).stop 2023-07-14T19:29:38.895 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:38.893+0000 7f9c24d37700 1 -- 172.21.15.98:0/1726745075 shutdown_connections 2023-07-14T19:29:38.895 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:38.893+0000 7f9c24d37700 1 --2- 172.21.15.98:0/1726745075 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f9c20101150 0x7f9c20199710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:29:38.895 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:38.893+0000 7f9c24d37700 1 --2- 172.21.15.98:0/1726745075 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9c200fef40 0x7f9c20194d60 unknown :-1 s=CLOSED pgs=275 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:29:38.895 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:38.893+0000 7f9c24d37700 1 --2- 172.21.15.98:0/1726745075 >> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] conn(0x7f9c080a11e0 0x7f9c080a3690 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:29:38.895 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:38.893+0000 7f9c24d37700 1 --2- 172.21.15.98:0/1726745075 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f9c201007a0 0x7f9c201952a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:29:38.895 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:38.893+0000 7f9c24d37700 1 -- 172.21.15.98:0/1726745075 >> 172.21.15.98:0/1726745075 conn(0x7f9c20075c90 msgr2=0x7f9c200fc4f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:29:38.896 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:38.893+0000 7f9c24d37700 1 -- 172.21.15.98:0/1726745075 shutdown_connections 2023-07-14T19:29:38.896 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:38.893+0000 7f9c24d37700 1 -- 172.21.15.98:0/1726745075 wait complete. 2023-07-14T19:29:38.919 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:29:38 smithi098 ceph-mon[127486]: pgmap v271: 97 pgs: 97 active+clean; 634 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 255 B/s wr, 5 op/s 2023-07-14T19:29:38.919 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:29:38 smithi098 ceph-mon[127486]: from='client.? 172.21.15.98:0/1410355709' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2023-07-14T19:29:38.919 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:29:38 smithi098 ceph-mon[127486]: from='client.? 172.21.15.98:0/350613141' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2023-07-14T19:29:38.930 INFO:tasks.cephadm.ceph_manager.ceph:need seq 137438953517 got 137438953519 for osd.4 2023-07-14T19:29:38.931 DEBUG:teuthology.parallel:result is None 2023-07-14T19:29:38.941 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:29:38 smithi101 ceph-mon[128807]: pgmap v271: 97 pgs: 97 active+clean; 634 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 255 B/s wr, 5 op/s 2023-07-14T19:29:38.941 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:29:38 smithi101 ceph-mon[128807]: from='client.? 172.21.15.98:0/1410355709' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2023-07-14T19:29:38.941 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:29:38 smithi101 ceph-mon[128807]: from='client.? 172.21.15.98:0/350613141' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2023-07-14T19:29:38.970 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:29:38 smithi139 ceph-mon[129293]: pgmap v271: 97 pgs: 97 active+clean; 634 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 255 B/s wr, 5 op/s 2023-07-14T19:29:38.970 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:29:38 smithi139 ceph-mon[129293]: from='client.? 172.21.15.98:0/1410355709' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2023-07-14T19:29:38.971 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:29:38 smithi139 ceph-mon[129293]: from='client.? 172.21.15.98:0/350613141' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2023-07-14T19:29:39.014 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:38.189+0000 7f7bf40b5700 1 -- 172.21.15.98:0/1983086305 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7bec0ff7e0 msgr2=0x7f7bec1037b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:29:39.014 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:38.189+0000 7f7bf40b5700 1 --2- 172.21.15.98:0/1983086305 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7bec0ff7e0 0x7f7bec1037b0 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7f7bdc0099c0 tx=0x7f7bdc009cd0 comp rx=0 tx=0).stop 2023-07-14T19:29:39.015 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:38.189+0000 7f7bf40b5700 1 -- 172.21.15.98:0/1983086305 shutdown_connections 2023-07-14T19:29:39.015 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:38.189+0000 7f7bf40b5700 1 --2- 172.21.15.98:0/1983086305 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f7bec103ee0 0x7f7bec10ba60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:29:39.015 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:38.189+0000 7f7bf40b5700 1 --2- 172.21.15.98:0/1983086305 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f7bec0fee40 0x7f7bec0ff210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:29:39.015 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:38.189+0000 7f7bf40b5700 1 --2- 172.21.15.98:0/1983086305 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7bec0ff7e0 0x7f7bec1037b0 unknown :-1 s=CLOSED pgs=272 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:29:39.015 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:38.189+0000 7f7bf40b5700 1 -- 172.21.15.98:0/1983086305 >> 172.21.15.98:0/1983086305 conn(0x7f7bec075ca0 msgr2=0x7f7bec0760a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:29:39.015 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:38.189+0000 7f7bf40b5700 1 -- 172.21.15.98:0/1983086305 shutdown_connections 2023-07-14T19:29:39.016 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:38.190+0000 7f7bf40b5700 1 -- 172.21.15.98:0/1983086305 wait complete. 2023-07-14T19:29:39.016 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:38.190+0000 7f7bf40b5700 1 Processor -- start 2023-07-14T19:29:39.016 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:38.190+0000 7f7bf40b5700 1 -- start start 2023-07-14T19:29:39.016 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:38.190+0000 7f7bf40b5700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f7bec0fee40 0x7f7bec194ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:29:39.016 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:38.190+0000 7f7bf40b5700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7bec0ff7e0 0x7f7bec1951e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:29:39.016 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:38.190+0000 7f7bf40b5700 1 --2- >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f7bec103ee0 0x7f7bec1985a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:29:39.016 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:38.190+0000 7f7bf40b5700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7bec198ba0 con 0x7f7bec0ff7e0 2023-07-14T19:29:39.017 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:38.190+0000 7f7bf40b5700 1 -- --> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7bec198d10 con 0x7f7bec103ee0 2023-07-14T19:29:39.017 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:38.190+0000 7f7bf40b5700 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7bec198ed0 con 0x7f7bec0fee40 2023-07-14T19:29:39.017 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:38.190+0000 7f7bf1650700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7bec0ff7e0 0x7f7bec1951e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:29:39.017 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:38.190+0000 7f7bf1650700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7bec0ff7e0 0x7f7bec1951e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:46154/0 (socket says 172.21.15.98:46154) 2023-07-14T19:29:39.017 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:38.190+0000 7f7bf1650700 1 -- 172.21.15.98:0/350613141 learned_addr learned my addr 172.21.15.98:0/350613141 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-07-14T19:29:39.017 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:38.191+0000 7f7bf1e51700 1 --2- 172.21.15.98:0/350613141 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f7bec0fee40 0x7f7bec194ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:29:39.018 INFO:teuthology.orchestra.run.smithi098.stdout:214748364835 2023-07-14T19:29:39.019 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:38.191+0000 7f7bf1650700 1 -- 172.21.15.98:0/350613141 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f7bec0fee40 msgr2=0x7f7bec194ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:29:39.019 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:38.191+0000 7f7bf1650700 1 --2- 172.21.15.98:0/350613141 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f7bec0fee40 0x7f7bec194ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:29:39.019 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:38.191+0000 7f7bf2652700 1 --2- 172.21.15.98:0/350613141 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f7bec103ee0 0x7f7bec1985a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:29:39.019 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:38.191+0000 7f7bf1650700 1 -- 172.21.15.98:0/350613141 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f7bec103ee0 msgr2=0x7f7bec1985a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:29:39.019 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:38.191+0000 7f7bf1650700 1 --2- 172.21.15.98:0/350613141 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f7bec103ee0 0x7f7bec1985a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:29:39.020 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:38.191+0000 7f7bf1650700 1 -- 172.21.15.98:0/350613141 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7bdc0095f0 con 0x7f7bec0ff7e0 2023-07-14T19:29:39.020 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:38.191+0000 7f7bf1650700 1 --2- 172.21.15.98:0/350613141 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7bec0ff7e0 0x7f7bec1951e0 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7f7bdc009990 tx=0x7f7bdc004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:29:39.020 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:38.191+0000 7f7be2ffd700 1 -- 172.21.15.98:0/350613141 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7bdc01d070 con 0x7f7bec0ff7e0 2023-07-14T19:29:39.020 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:38.191+0000 7f7be2ffd700 1 -- 172.21.15.98:0/350613141 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f7bdc0054f0 con 0x7f7bec0ff7e0 2023-07-14T19:29:39.020 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:38.191+0000 7f7be2ffd700 1 -- 172.21.15.98:0/350613141 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7bdc003ad0 con 0x7f7bec0ff7e0 2023-07-14T19:29:39.020 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:38.191+0000 7f7bf40b5700 1 -- 172.21.15.98:0/350613141 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7bec199150 con 0x7f7bec0ff7e0 2023-07-14T19:29:39.020 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:38.191+0000 7f7bf40b5700 1 -- 172.21.15.98:0/350613141 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7bec1a1930 con 0x7f7bec0ff7e0 2023-07-14T19:29:39.021 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:38.192+0000 7f7bf40b5700 1 -- 172.21.15.98:0/350613141 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7bec04e340 con 0x7f7bec0ff7e0 2023-07-14T19:29:39.021 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:38.193+0000 7f7be2ffd700 1 -- 172.21.15.98:0/350613141 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 17) v1 ==== 135766+0+0 (secure 0 0 0) 0x7f7bdc0047a0 con 0x7f7bec0ff7e0 2023-07-14T19:29:39.021 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:38.193+0000 7f7be2ffd700 1 --2- 172.21.15.98:0/350613141 >> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] conn(0x7f7bd80a11e0 0x7f7bd80a3690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:29:39.021 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:38.194+0000 7f7bf1e51700 1 --2- 172.21.15.98:0/350613141 >> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] conn(0x7f7bd80a11e0 0x7f7bd80a3690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:29:39.021 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:38.194+0000 7f7be2ffd700 1 -- 172.21.15.98:0/350613141 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 7979+0+0 (secure 0 0 0) 0x7f7bdc0cd840 con 0x7f7bec0ff7e0 2023-07-14T19:29:39.021 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:38.194+0000 7f7bf1e51700 1 --2- 172.21.15.98:0/350613141 >> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] conn(0x7f7bd80a11e0 0x7f7bd80a3690 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f7be8005950 tx=0x7f7be800a300 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:29:39.021 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:38.198+0000 7f7be2ffd700 1 -- 172.21.15.98:0/350613141 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f7bdc096f70 con 0x7f7bec0ff7e0 2023-07-14T19:29:39.022 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:38.402+0000 7f7bf40b5700 1 -- 172.21.15.98:0/350613141 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 7} v 0) v1 -- 0x7f7bec195fc0 con 0x7f7bec0ff7e0 2023-07-14T19:29:39.022 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:38.403+0000 7f7be2ffd700 1 -- 172.21.15.98:0/350613141 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 7}]=0 v0) v1 ==== 74+0+13 (secure 0 0 0) 0x7f7bdc09a590 con 0x7f7bec0ff7e0 2023-07-14T19:29:39.022 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:38.405+0000 7f7bf40b5700 1 -- 172.21.15.98:0/350613141 >> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] conn(0x7f7bd80a11e0 msgr2=0x7f7bd80a3690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:29:39.022 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:38.405+0000 7f7bf40b5700 1 --2- 172.21.15.98:0/350613141 >> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] conn(0x7f7bd80a11e0 0x7f7bd80a3690 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f7be8005950 tx=0x7f7be800a300 comp rx=0 tx=0).stop 2023-07-14T19:29:39.022 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:38.405+0000 7f7bf40b5700 1 -- 172.21.15.98:0/350613141 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7bec0ff7e0 msgr2=0x7f7bec1951e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:29:39.022 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:38.405+0000 7f7bf40b5700 1 --2- 172.21.15.98:0/350613141 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7bec0ff7e0 0x7f7bec1951e0 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7f7bdc009990 tx=0x7f7bdc004730 comp rx=0 tx=0).stop 2023-07-14T19:29:39.022 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:38.405+0000 7f7bf40b5700 1 -- 172.21.15.98:0/350613141 shutdown_connections 2023-07-14T19:29:39.023 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:38.405+0000 7f7bf40b5700 1 --2- 172.21.15.98:0/350613141 >> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] conn(0x7f7bd80a11e0 0x7f7bd80a3690 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:29:39.023 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:38.405+0000 7f7bf40b5700 1 --2- 172.21.15.98:0/350613141 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f7bec0fee40 0x7f7bec194ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:29:39.023 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:38.405+0000 7f7bf40b5700 1 --2- 172.21.15.98:0/350613141 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f7bec103ee0 0x7f7bec1985a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:29:39.023 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:38.405+0000 7f7bf40b5700 1 --2- 172.21.15.98:0/350613141 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7bec0ff7e0 0x7f7bec1951e0 unknown :-1 s=CLOSED pgs=273 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:29:39.023 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:38.405+0000 7f7bf40b5700 1 -- 172.21.15.98:0/350613141 >> 172.21.15.98:0/350613141 conn(0x7f7bec075ca0 msgr2=0x7f7bec0fc4f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:29:39.023 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:38.405+0000 7f7bf40b5700 1 -- 172.21.15.98:0/350613141 shutdown_connections 2023-07-14T19:29:39.023 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:38.405+0000 7f7bf40b5700 1 -- 172.21.15.98:0/350613141 wait complete. 2023-07-14T19:29:39.560 INFO:tasks.cephadm.ceph_manager.ceph:need seq 214748364832 got 214748364835 for osd.7 2023-07-14T19:29:39.560 DEBUG:teuthology.parallel:result is None 2023-07-14T19:29:39.564 INFO:tasks.cephadm.ceph_manager.ceph:need seq 34359738430 got 34359738433 for osd.0 2023-07-14T19:29:39.564 DEBUG:teuthology.parallel:result is None 2023-07-14T19:29:39.764 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:29:39 smithi098 ceph-mon[127486]: from='client.? 172.21.15.98:0/1726745075' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 9}]: dispatch 2023-07-14T19:29:39.856 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:39.816+0000 7faaff0da700 1 -- 172.21.15.98:0/1544567755 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7faaf80fef40 msgr2=0x7faaf80ff310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:29:39.856 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:39.816+0000 7faaff0da700 1 --2- 172.21.15.98:0/1544567755 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7faaf80fef40 0x7faaf80ff310 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7faaec0099c0 tx=0x7faaec009cd0 comp rx=0 tx=0).stop 2023-07-14T19:29:39.856 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:39.816+0000 7faaff0da700 1 -- 172.21.15.98:0/1544567755 shutdown_connections 2023-07-14T19:29:39.857 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:39.816+0000 7faaff0da700 1 --2- 172.21.15.98:0/1544567755 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7faaf8101150 0x7faaf810bb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:29:39.857 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:39.816+0000 7faaff0da700 1 --2- 172.21.15.98:0/1544567755 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7faaf80fef40 0x7faaf80ff310 unknown :-1 s=CLOSED pgs=276 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:29:39.857 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:39.816+0000 7faaff0da700 1 --2- 172.21.15.98:0/1544567755 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7faaf81007a0 0x7faaf8100c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:29:39.857 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:39.816+0000 7faaff0da700 1 -- 172.21.15.98:0/1544567755 >> 172.21.15.98:0/1544567755 conn(0x7faaf8075c90 msgr2=0x7faaf8076090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:29:39.858 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:39.816+0000 7faaff0da700 1 -- 172.21.15.98:0/1544567755 shutdown_connections 2023-07-14T19:29:39.858 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:39.816+0000 7faaff0da700 1 -- 172.21.15.98:0/1544567755 wait complete. 2023-07-14T19:29:39.858 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:39.817+0000 7faaff0da700 1 Processor -- start 2023-07-14T19:29:39.859 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:39.817+0000 7faaff0da700 1 -- start start 2023-07-14T19:29:39.859 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:39.817+0000 7faaff0da700 1 --2- >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7faaf80fef40 0x7faaf8194ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:29:39.859 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:39.817+0000 7faaff0da700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7faaf81007a0 0x7faaf8195410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:29:39.860 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:39.817+0000 7faaff0da700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7faaf8101150 0x7faaf8199880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:29:39.860 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:39.817+0000 7faaff0da700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faaf8199e90 con 0x7faaf8101150 2023-07-14T19:29:39.860 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:39.817+0000 7faaff0da700 1 -- --> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faaf819a000 con 0x7faaf80fef40 2023-07-14T19:29:39.860 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:39.817+0000 7faaff0da700 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faaf819a1c0 con 0x7faaf81007a0 2023-07-14T19:29:39.861 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:39.817+0000 7faafd677700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7faaf8101150 0x7faaf8199880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:29:39.861 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:39.817+0000 7faafd677700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7faaf8101150 0x7faaf8199880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:46184/0 (socket says 172.21.15.98:46184) 2023-07-14T19:29:39.861 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:39.817+0000 7faafd677700 1 -- 172.21.15.98:0/3940104887 learned_addr learned my addr 172.21.15.98:0/3940104887 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-07-14T19:29:39.861 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:39.817+0000 7faafce76700 1 --2- 172.21.15.98:0/3940104887 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7faaf80fef40 0x7faaf8194ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:29:39.861 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:39.817+0000 7faaf7fff700 1 --2- 172.21.15.98:0/3940104887 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7faaf81007a0 0x7faaf8195410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:29:39.862 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:39.818+0000 7faafd677700 1 -- 172.21.15.98:0/3940104887 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7faaf81007a0 msgr2=0x7faaf8195410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:29:39.862 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:39.818+0000 7faafd677700 1 --2- 172.21.15.98:0/3940104887 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7faaf81007a0 0x7faaf8195410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:29:39.862 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:39.818+0000 7faafd677700 1 -- 172.21.15.98:0/3940104887 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7faaf80fef40 msgr2=0x7faaf8194ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:29:39.862 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:39.818+0000 7faafd677700 1 --2- 172.21.15.98:0/3940104887 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7faaf80fef40 0x7faaf8194ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:29:39.863 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:39.818+0000 7faafd677700 1 -- 172.21.15.98:0/3940104887 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faaec0095f0 con 0x7faaf8101150 2023-07-14T19:29:39.863 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:39.818+0000 7faaf7fff700 1 --2- 172.21.15.98:0/3940104887 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7faaf81007a0 0x7faaf8195410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-07-14T19:29:39.863 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:39.818+0000 7faafd677700 1 --2- 172.21.15.98:0/3940104887 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7faaf8101150 0x7faaf8199880 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7faaf0007ee0 tx=0x7faaf000d3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:29:39.863 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:39.818+0000 7faaf5ffb700 1 -- 172.21.15.98:0/3940104887 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faaf0017070 con 0x7faaf8101150 2023-07-14T19:29:39.863 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:39.818+0000 7faaff0da700 1 -- 172.21.15.98:0/3940104887 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faaf819a4a0 con 0x7faaf8101150 2023-07-14T19:29:39.864 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:39.818+0000 7faaf5ffb700 1 -- 172.21.15.98:0/3940104887 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7faaf000f040 con 0x7faaf8101150 2023-07-14T19:29:39.864 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:39.818+0000 7faaf5ffb700 1 -- 172.21.15.98:0/3940104887 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faaf0005740 con 0x7faaf8101150 2023-07-14T19:29:39.864 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:39.818+0000 7faaff0da700 1 -- 172.21.15.98:0/3940104887 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faaf8068b60 con 0x7faaf8101150 2023-07-14T19:29:39.864 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:39.819+0000 7faaff0da700 1 -- 172.21.15.98:0/3940104887 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faaf804e340 con 0x7faaf8101150 2023-07-14T19:29:39.865 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:39.820+0000 7faaf5ffb700 1 -- 172.21.15.98:0/3940104887 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 17) v1 ==== 135766+0+0 (secure 0 0 0) 0x7faaf000dc60 con 0x7faaf8101150 2023-07-14T19:29:39.865 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:39.820+0000 7faaf5ffb700 1 --2- 172.21.15.98:0/3940104887 >> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] conn(0x7faae80a11e0 0x7faae80a3690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:29:39.865 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:39.820+0000 7faafce76700 1 --2- 172.21.15.98:0/3940104887 >> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] conn(0x7faae80a11e0 0x7faae80a3690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:29:39.865 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:39.821+0000 7faaf5ffb700 1 -- 172.21.15.98:0/3940104887 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 7979+0+0 (secure 0 0 0) 0x7faaf00cc0a0 con 0x7faaf8101150 2023-07-14T19:29:39.865 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:39.821+0000 7faafce76700 1 --2- 172.21.15.98:0/3940104887 >> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] conn(0x7faae80a11e0 0x7faae80a3690 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7faaec000c00 tx=0x7faaec003680 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:29:39.865 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:39.824+0000 7faaf5ffb700 1 -- 172.21.15.98:0/3940104887 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7faaf0095840 con 0x7faaf8101150 2023-07-14T19:29:39.941 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:29:39 smithi101 ceph-mon[128807]: from='client.? 172.21.15.98:0/1726745075' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 9}]: dispatch 2023-07-14T19:29:39.970 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:29:39 smithi139 ceph-mon[129293]: from='client.? 172.21.15.98:0/1726745075' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 9}]: dispatch 2023-07-14T19:29:40.032 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:40.030+0000 7faaff0da700 1 -- 172.21.15.98:0/3940104887 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 3} v 0) v1 -- 0x7faaf8068d80 con 0x7faaf8101150 2023-07-14T19:29:40.033 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:40.031+0000 7faaf5ffb700 1 -- 172.21.15.98:0/3940104887 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 3}]=0 v0) v1 ==== 74+0+13 (secure 0 0 0) 0x7faaf0098e60 con 0x7faaf8101150 2023-07-14T19:29:40.033 INFO:teuthology.orchestra.run.smithi098.stdout:107374182453 2023-07-14T19:29:40.037 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:40.035+0000 7faaff0da700 1 -- 172.21.15.98:0/3940104887 >> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] conn(0x7faae80a11e0 msgr2=0x7faae80a3690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:29:40.037 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:40.035+0000 7faaff0da700 1 --2- 172.21.15.98:0/3940104887 >> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] conn(0x7faae80a11e0 0x7faae80a3690 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7faaec000c00 tx=0x7faaec003680 comp rx=0 tx=0).stop 2023-07-14T19:29:40.037 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:40.035+0000 7faaff0da700 1 -- 172.21.15.98:0/3940104887 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7faaf8101150 msgr2=0x7faaf8199880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:29:40.038 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:40.035+0000 7faaff0da700 1 --2- 172.21.15.98:0/3940104887 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7faaf8101150 0x7faaf8199880 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7faaf0007ee0 tx=0x7faaf000d3b0 comp rx=0 tx=0).stop 2023-07-14T19:29:40.038 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:40.037+0000 7faaff0da700 1 -- 172.21.15.98:0/3940104887 shutdown_connections 2023-07-14T19:29:40.039 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:40.037+0000 7faaff0da700 1 --2- 172.21.15.98:0/3940104887 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7faaf8101150 0x7faaf8199880 unknown :-1 s=CLOSED pgs=277 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:29:40.039 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:40.037+0000 7faaff0da700 1 --2- 172.21.15.98:0/3940104887 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7faaf80fef40 0x7faaf8194ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:29:40.039 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:40.037+0000 7faaff0da700 1 --2- 172.21.15.98:0/3940104887 >> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] conn(0x7faae80a11e0 0x7faae80a3690 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:29:40.040 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:40.037+0000 7faaff0da700 1 --2- 172.21.15.98:0/3940104887 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7faaf81007a0 0x7faaf8195410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:29:40.040 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:40.037+0000 7faaff0da700 1 -- 172.21.15.98:0/3940104887 >> 172.21.15.98:0/3940104887 conn(0x7faaf8075c90 msgr2=0x7faaf80fc4f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:29:40.041 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:40.039+0000 7faaff0da700 1 -- 172.21.15.98:0/3940104887 shutdown_connections 2023-07-14T19:29:40.041 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:40.039+0000 7faaff0da700 1 -- 172.21.15.98:0/3940104887 wait complete. 2023-07-14T19:29:40.381 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:40.304+0000 7feb63c1c700 1 -- 172.21.15.98:0/2519947374 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7feb5c0ff580 msgr2=0x7feb5c0ff950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:29:40.381 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:40.304+0000 7feb63c1c700 1 --2- 172.21.15.98:0/2519947374 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7feb5c0ff580 0x7feb5c0ff950 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7feb5400b0d0 tx=0x7feb5400b3e0 comp rx=0 tx=0).stop 2023-07-14T19:29:40.381 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:40.304+0000 7feb63c1c700 1 -- 172.21.15.98:0/2519947374 shutdown_connections 2023-07-14T19:29:40.382 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:40.304+0000 7feb63c1c700 1 --2- 172.21.15.98:0/2519947374 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7feb5c103f50 0x7feb5c10bb10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:29:40.382 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:40.304+0000 7feb63c1c700 1 --2- 172.21.15.98:0/2519947374 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7feb5c0ff580 0x7feb5c0ff950 unknown :-1 s=CLOSED pgs=280 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:29:40.382 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:40.304+0000 7feb63c1c700 1 --2- 172.21.15.98:0/2519947374 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7feb5c0fff20 0x7feb5c103820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:29:40.382 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:40.304+0000 7feb63c1c700 1 -- 172.21.15.98:0/2519947374 >> 172.21.15.98:0/2519947374 conn(0x7feb5c075280 msgr2=0x7feb5c075680 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:29:40.382 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:40.304+0000 7feb63c1c700 1 -- 172.21.15.98:0/2519947374 shutdown_connections 2023-07-14T19:29:40.382 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:40.304+0000 7feb63c1c700 1 -- 172.21.15.98:0/2519947374 wait complete. 2023-07-14T19:29:40.382 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:40.304+0000 7feb63c1c700 1 Processor -- start 2023-07-14T19:29:40.383 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:40.304+0000 7feb63c1c700 1 -- start start 2023-07-14T19:29:40.383 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:40.305+0000 7feb63c1c700 1 --2- >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7feb5c0ff580 0x7feb5c195f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:29:40.383 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:40.305+0000 7feb63c1c700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7feb5c0fff20 0x7feb5c1964c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:29:40.383 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:40.305+0000 7feb63c1c700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7feb5c103f50 0x7feb5c19a930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:29:40.383 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:40.305+0000 7feb63c1c700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feb5c19af40 con 0x7feb5c103f50 2023-07-14T19:29:40.383 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:40.305+0000 7feb63c1c700 1 -- --> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feb5c19b0b0 con 0x7feb5c0ff580 2023-07-14T19:29:40.383 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:40.305+0000 7feb63c1c700 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feb5c19b270 con 0x7feb5c0fff20 2023-07-14T19:29:40.384 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:40.305+0000 7feb621b9700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7feb5c103f50 0x7feb5c19a930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:29:40.384 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:40.305+0000 7feb621b9700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7feb5c103f50 0x7feb5c19a930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:46212/0 (socket says 172.21.15.98:46212) 2023-07-14T19:29:40.384 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:40.305+0000 7feb621b9700 1 -- 172.21.15.98:0/288062076 learned_addr learned my addr 172.21.15.98:0/288062076 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-07-14T19:29:40.384 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:40.305+0000 7feb619b8700 1 --2- 172.21.15.98:0/288062076 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7feb5c0ff580 0x7feb5c195f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:29:40.384 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:40.305+0000 7feb611b7700 1 --2- 172.21.15.98:0/288062076 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7feb5c0fff20 0x7feb5c1964c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:29:40.384 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:40.305+0000 7feb621b9700 1 -- 172.21.15.98:0/288062076 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7feb5c0fff20 msgr2=0x7feb5c1964c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:29:40.385 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:40.305+0000 7feb621b9700 1 --2- 172.21.15.98:0/288062076 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7feb5c0fff20 0x7feb5c1964c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:29:40.385 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:40.305+0000 7feb621b9700 1 -- 172.21.15.98:0/288062076 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7feb5c0ff580 msgr2=0x7feb5c195f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:29:40.385 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:40.305+0000 7feb621b9700 1 --2- 172.21.15.98:0/288062076 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7feb5c0ff580 0x7feb5c195f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:29:40.385 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:40.305+0000 7feb621b9700 1 -- 172.21.15.98:0/288062076 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feb54009d00 con 0x7feb5c103f50 2023-07-14T19:29:40.385 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:40.305+0000 7feb621b9700 1 --2- 172.21.15.98:0/288062076 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7feb5c103f50 0x7feb5c19a930 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7feb5800b550 tx=0x7feb5800b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:29:40.385 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:40.305+0000 7feb52ffd700 1 -- 172.21.15.98:0/288062076 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb58011830 con 0x7feb5c103f50 2023-07-14T19:29:40.386 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:40.305+0000 7feb63c1c700 1 -- 172.21.15.98:0/288062076 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7feb5c19b550 con 0x7feb5c103f50 2023-07-14T19:29:40.386 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:40.305+0000 7feb52ffd700 1 -- 172.21.15.98:0/288062076 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7feb58011e50 con 0x7feb5c103f50 2023-07-14T19:29:40.386 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:40.305+0000 7feb52ffd700 1 -- 172.21.15.98:0/288062076 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb58010e70 con 0x7feb5c103f50 2023-07-14T19:29:40.386 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:40.305+0000 7feb63c1c700 1 -- 172.21.15.98:0/288062076 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7feb5c1a2db0 con 0x7feb5c103f50 2023-07-14T19:29:40.386 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:40.306+0000 7feb63c1c700 1 -- 172.21.15.98:0/288062076 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7feb5c04e340 con 0x7feb5c103f50 2023-07-14T19:29:40.386 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:40.307+0000 7feb52ffd700 1 -- 172.21.15.98:0/288062076 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 17) v1 ==== 135766+0+0 (secure 0 0 0) 0x7feb58010510 con 0x7feb5c103f50 2023-07-14T19:29:40.386 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:40.308+0000 7feb52ffd700 1 --2- 172.21.15.98:0/288062076 >> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] conn(0x7feb480a11e0 0x7feb480a3690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:29:40.386 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:40.308+0000 7feb52ffd700 1 -- 172.21.15.98:0/288062076 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 7979+0+0 (secure 0 0 0) 0x7feb580cc2e0 con 0x7feb5c103f50 2023-07-14T19:29:40.387 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:40.308+0000 7feb619b8700 1 --2- 172.21.15.98:0/288062076 >> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] conn(0x7feb480a11e0 0x7feb480a3690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:29:40.387 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:40.308+0000 7feb619b8700 1 --2- 172.21.15.98:0/288062076 >> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] conn(0x7feb480a11e0 0x7feb480a3690 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7feb54009fd0 tx=0x7feb5400b040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:29:40.387 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:40.312+0000 7feb52ffd700 1 -- 172.21.15.98:0/288062076 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7feb58095b90 con 0x7feb5c103f50 2023-07-14T19:29:40.428 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:40.426+0000 7f4afe041700 1 -- 172.21.15.98:0/4035347890 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4af8104fd0 msgr2=0x7f4af810cb50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:29:40.428 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:40.426+0000 7f4afe041700 1 --2- 172.21.15.98:0/4035347890 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4af8104fd0 0x7f4af810cb50 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7f4af00099c0 tx=0x7f4af0009cd0 comp rx=0 tx=0).stop 2023-07-14T19:29:40.429 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:40.426+0000 7f4afe041700 1 -- 172.21.15.98:0/4035347890 shutdown_connections 2023-07-14T19:29:40.429 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:40.426+0000 7f4afe041700 1 --2- 172.21.15.98:0/4035347890 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4af8104fd0 0x7f4af810cb50 unknown :-1 s=CLOSED pgs=282 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:29:40.429 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:40.426+0000 7f4afe041700 1 --2- 172.21.15.98:0/4035347890 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f4af80fff30 0x7f4af8100300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:29:40.429 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:40.426+0000 7f4afe041700 1 --2- 172.21.15.98:0/4035347890 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f4af81008d0 0x7f4af81048a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:29:40.429 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:40.426+0000 7f4afe041700 1 -- 172.21.15.98:0/4035347890 >> 172.21.15.98:0/4035347890 conn(0x7f4af80fb790 msgr2=0x7f4af80fdba0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:29:40.429 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:40.427+0000 7f4afe041700 1 -- 172.21.15.98:0/4035347890 shutdown_connections 2023-07-14T19:29:40.429 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:40.427+0000 7f4afe041700 1 -- 172.21.15.98:0/4035347890 wait complete. 2023-07-14T19:29:40.431 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:40.427+0000 7f4afe041700 1 Processor -- start 2023-07-14T19:29:40.431 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:40.427+0000 7f4afe041700 1 -- start start 2023-07-14T19:29:40.431 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:40.427+0000 7f4afe041700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f4af80fff30 0x7f4af8195d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:29:40.431 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:40.427+0000 7f4afe041700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4af81008d0 0x7f4af81962c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:29:40.431 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:40.427+0000 7f4afe041700 1 --2- >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f4af8104fd0 0x7f4af819a730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:29:40.431 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:40.427+0000 7f4afe041700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4af819ad40 con 0x7f4af81008d0 2023-07-14T19:29:40.432 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:40.427+0000 7f4afe041700 1 -- --> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4af819aeb0 con 0x7f4af8104fd0 2023-07-14T19:29:40.432 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:40.427+0000 7f4afe041700 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4af819b070 con 0x7f4af80fff30 2023-07-14T19:29:40.432 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:40.428+0000 7f4af6ffd700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4af81008d0 0x7f4af81962c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:29:40.432 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:40.428+0000 7f4af77fe700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f4af80fff30 0x7f4af8195d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:29:40.432 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:40.428+0000 7f4af6ffd700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4af81008d0 0x7f4af81962c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:46222/0 (socket says 172.21.15.98:46222) 2023-07-14T19:29:40.432 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:40.428+0000 7f4af6ffd700 1 -- 172.21.15.98:0/699395773 learned_addr learned my addr 172.21.15.98:0/699395773 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-07-14T19:29:40.433 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:40.428+0000 7f4af7fff700 1 --2- 172.21.15.98:0/699395773 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f4af8104fd0 0x7f4af819a730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:29:40.433 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:40.428+0000 7f4af77fe700 1 -- 172.21.15.98:0/699395773 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f4af8104fd0 msgr2=0x7f4af819a730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:29:40.433 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:40.428+0000 7f4af77fe700 1 --2- 172.21.15.98:0/699395773 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f4af8104fd0 0x7f4af819a730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:29:40.433 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:40.428+0000 7f4af77fe700 1 -- 172.21.15.98:0/699395773 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4af81008d0 msgr2=0x7f4af81962c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:29:40.433 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:40.428+0000 7f4af77fe700 1 --2- 172.21.15.98:0/699395773 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4af81008d0 0x7f4af81962c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:29:40.433 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:40.428+0000 7f4af77fe700 1 -- 172.21.15.98:0/699395773 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4af00095f0 con 0x7f4af80fff30 2023-07-14T19:29:40.433 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:40.428+0000 7f4af6ffd700 1 --2- 172.21.15.98:0/699395773 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4af81008d0 0x7f4af81962c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-07-14T19:29:40.433 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:40.428+0000 7f4af77fe700 1 --2- 172.21.15.98:0/699395773 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f4af80fff30 0x7f4af8195d80 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f4ae8007ec0 tx=0x7f4ae800d3b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:29:40.434 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:40.428+0000 7f4af4ff9700 1 -- 172.21.15.98:0/699395773 <== mon.1 v2:172.21.15.139:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ae8004db0 con 0x7f4af80fff30 2023-07-14T19:29:40.434 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:40.428+0000 7f4afe041700 1 -- 172.21.15.98:0/699395773 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4af819b350 con 0x7f4af80fff30 2023-07-14T19:29:40.434 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:40.428+0000 7f4afe041700 1 -- 172.21.15.98:0/699395773 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4af81a2bb0 con 0x7f4af80fff30 2023-07-14T19:29:40.434 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:40.428+0000 7f4af4ff9700 1 -- 172.21.15.98:0/699395773 <== mon.1 v2:172.21.15.139:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f4ae800f040 con 0x7f4af80fff30 2023-07-14T19:29:40.434 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:40.428+0000 7f4af4ff9700 1 -- 172.21.15.98:0/699395773 <== mon.1 v2:172.21.15.139:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ae8005850 con 0x7f4af80fff30 2023-07-14T19:29:40.434 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:40.429+0000 7f4afe041700 1 -- 172.21.15.98:0/699395773 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4af804e340 con 0x7f4af80fff30 2023-07-14T19:29:40.435 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:40.431+0000 7f4af4ff9700 1 -- 172.21.15.98:0/699395773 <== mon.1 v2:172.21.15.139:3300/0 4 ==== mgrmap(e 17) v1 ==== 135766+0+0 (secure 0 0 0) 0x7f4ae8005020 con 0x7f4af80fff30 2023-07-14T19:29:40.435 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:40.431+0000 7f4af4ff9700 1 --2- 172.21.15.98:0/699395773 >> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] conn(0x7f4ae40a11e0 0x7f4ae40a3690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:29:40.435 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:40.431+0000 7f4af6ffd700 1 --2- 172.21.15.98:0/699395773 >> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] conn(0x7f4ae40a11e0 0x7f4ae40a3690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:29:40.435 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:40.431+0000 7f4af4ff9700 1 -- 172.21.15.98:0/699395773 <== mon.1 v2:172.21.15.139:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 7979+0+0 (secure 0 0 0) 0x7f4ae80cc610 con 0x7f4af80fff30 2023-07-14T19:29:40.436 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:40.432+0000 7f4af6ffd700 1 --2- 172.21.15.98:0/699395773 >> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] conn(0x7f4ae40a11e0 0x7f4ae40a3690 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f4af81973a0 tx=0x7f4ae000b480 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:29:40.436 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:40.435+0000 7f4af4ff9700 1 -- 172.21.15.98:0/699395773 <== mon.1 v2:172.21.15.139:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f4ae8095cf0 con 0x7f4af80fff30 2023-07-14T19:29:40.522 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:40.520+0000 7feb63c1c700 1 -- 172.21.15.98:0/288062076 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 8} v 0) v1 -- 0x7feb5c1972e0 con 0x7feb5c103f50 2023-07-14T19:29:40.522 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:40.520+0000 7feb52ffd700 1 -- 172.21.15.98:0/288062076 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 8}]=0 v0) v1 ==== 74+0+13 (secure 0 0 0) 0x7feb580991b0 con 0x7feb5c103f50 2023-07-14T19:29:40.523 INFO:teuthology.orchestra.run.smithi098.stdout:244813135903 2023-07-14T19:29:40.525 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:40.523+0000 7feb63c1c700 1 -- 172.21.15.98:0/288062076 >> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] conn(0x7feb480a11e0 msgr2=0x7feb480a3690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:29:40.526 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:40.523+0000 7feb63c1c700 1 --2- 172.21.15.98:0/288062076 >> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] conn(0x7feb480a11e0 0x7feb480a3690 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7feb54009fd0 tx=0x7feb5400b040 comp rx=0 tx=0).stop 2023-07-14T19:29:40.526 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:40.523+0000 7feb63c1c700 1 -- 172.21.15.98:0/288062076 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7feb5c103f50 msgr2=0x7feb5c19a930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:29:40.526 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:40.523+0000 7feb63c1c700 1 --2- 172.21.15.98:0/288062076 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7feb5c103f50 0x7feb5c19a930 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7feb5800b550 tx=0x7feb5800b910 comp rx=0 tx=0).stop 2023-07-14T19:29:40.527 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:40.524+0000 7feb63c1c700 1 -- 172.21.15.98:0/288062076 shutdown_connections 2023-07-14T19:29:40.527 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:40.524+0000 7feb63c1c700 1 --2- 172.21.15.98:0/288062076 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7feb5c103f50 0x7feb5c19a930 unknown :-1 s=CLOSED pgs=281 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:29:40.527 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:40.524+0000 7feb63c1c700 1 --2- 172.21.15.98:0/288062076 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7feb5c0ff580 0x7feb5c195f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:29:40.527 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:40.524+0000 7feb63c1c700 1 --2- 172.21.15.98:0/288062076 >> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] conn(0x7feb480a11e0 0x7feb480a3690 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:29:40.528 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:40.524+0000 7feb63c1c700 1 --2- 172.21.15.98:0/288062076 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7feb5c0fff20 0x7feb5c1964c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:29:40.528 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:40.524+0000 7feb63c1c700 1 -- 172.21.15.98:0/288062076 >> 172.21.15.98:0/288062076 conn(0x7feb5c075280 msgr2=0x7feb5c102a00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:29:40.528 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:40.524+0000 7feb63c1c700 1 -- 172.21.15.98:0/288062076 shutdown_connections 2023-07-14T19:29:40.529 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:40.524+0000 7feb63c1c700 1 -- 172.21.15.98:0/288062076 wait complete. 2023-07-14T19:29:40.645 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:40.643+0000 7f4afe041700 1 -- 172.21.15.98:0/699395773 --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 10} v 0) v1 -- 0x7f4af81970e0 con 0x7f4af80fff30 2023-07-14T19:29:40.645 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:40.644+0000 7f4af4ff9700 1 -- 172.21.15.98:0/699395773 <== mon.1 v2:172.21.15.139:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 10}]=0 v0) v1 ==== 75+0+13 (secure 0 0 0) 0x7f4ae8023090 con 0x7f4af80fff30 2023-07-14T19:29:40.646 INFO:teuthology.orchestra.run.smithi098.stdout:300647710744 2023-07-14T19:29:40.647 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:40.646+0000 7f4afe041700 1 -- 172.21.15.98:0/699395773 >> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] conn(0x7f4ae40a11e0 msgr2=0x7f4ae40a3690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:29:40.647 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:40.646+0000 7f4afe041700 1 --2- 172.21.15.98:0/699395773 >> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] conn(0x7f4ae40a11e0 0x7f4ae40a3690 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f4af81973a0 tx=0x7f4ae000b480 comp rx=0 tx=0).stop 2023-07-14T19:29:40.647 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:40.646+0000 7f4afe041700 1 -- 172.21.15.98:0/699395773 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f4af80fff30 msgr2=0x7f4af8195d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:29:40.648 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:40.646+0000 7f4afe041700 1 --2- 172.21.15.98:0/699395773 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f4af80fff30 0x7f4af8195d80 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f4ae8007ec0 tx=0x7f4ae800d3b0 comp rx=0 tx=0).stop 2023-07-14T19:29:40.648 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:40.646+0000 7f4afe041700 1 -- 172.21.15.98:0/699395773 shutdown_connections 2023-07-14T19:29:40.648 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:40.646+0000 7f4afe041700 1 --2- 172.21.15.98:0/699395773 >> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] conn(0x7f4ae40a11e0 0x7f4ae40a3690 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:29:40.648 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:40.646+0000 7f4afe041700 1 --2- 172.21.15.98:0/699395773 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f4af80fff30 0x7f4af8195d80 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:29:40.648 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:40.646+0000 7f4afe041700 1 --2- 172.21.15.98:0/699395773 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f4af8104fd0 0x7f4af819a730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:29:40.648 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:40.646+0000 7f4afe041700 1 --2- 172.21.15.98:0/699395773 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4af81008d0 0x7f4af81962c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:29:40.648 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:40.646+0000 7f4afe041700 1 -- 172.21.15.98:0/699395773 >> 172.21.15.98:0/699395773 conn(0x7f4af80fb790 msgr2=0x7f4af80fc1b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:29:40.649 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:40.646+0000 7f4afe041700 1 -- 172.21.15.98:0/699395773 shutdown_connections 2023-07-14T19:29:40.649 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:40.646+0000 7f4afe041700 1 -- 172.21.15.98:0/699395773 wait complete. 2023-07-14T19:29:40.738 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:29:40 smithi098 ceph-mon[127486]: pgmap v272: 97 pgs: 97 active+clean; 634 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 255 B/s wr, 6 op/s 2023-07-14T19:29:40.738 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:29:40 smithi098 ceph-mon[127486]: from='client.? 172.21.15.98:0/3940104887' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2023-07-14T19:29:40.738 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:29:40 smithi098 ceph-mon[127486]: from='client.? 172.21.15.98:0/2252856940' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2023-07-14T19:29:40.738 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:29:40 smithi098 ceph-mon[127486]: from='client.? 172.21.15.98:0/288062076' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 8}]: dispatch 2023-07-14T19:29:40.738 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:29:40 smithi098 ceph-mon[127486]: from='client.? 172.21.15.98:0/699395773' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 10}]: dispatch 2023-07-14T19:29:40.766 INFO:tasks.cephadm.ceph_manager.ceph:need seq 279172874265 got 279172874267 for osd.9 2023-07-14T19:29:40.767 DEBUG:teuthology.parallel:result is None 2023-07-14T19:29:40.941 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:29:40 smithi101 ceph-mon[128807]: pgmap v272: 97 pgs: 97 active+clean; 634 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 255 B/s wr, 6 op/s 2023-07-14T19:29:40.941 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:29:40 smithi101 ceph-mon[128807]: from='client.? 172.21.15.98:0/3940104887' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2023-07-14T19:29:40.941 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:29:40 smithi101 ceph-mon[128807]: from='client.? 172.21.15.98:0/2252856940' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2023-07-14T19:29:40.942 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:29:40 smithi101 ceph-mon[128807]: from='client.? 172.21.15.98:0/288062076' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 8}]: dispatch 2023-07-14T19:29:40.942 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:29:40 smithi101 ceph-mon[128807]: from='client.? 172.21.15.98:0/699395773' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 10}]: dispatch 2023-07-14T19:29:40.970 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:29:40 smithi139 ceph-mon[129293]: pgmap v272: 97 pgs: 97 active+clean; 634 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 255 B/s wr, 6 op/s 2023-07-14T19:29:40.970 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:29:40 smithi139 ceph-mon[129293]: from='client.? 172.21.15.98:0/3940104887' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2023-07-14T19:29:40.970 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:29:40 smithi139 ceph-mon[129293]: from='client.? 172.21.15.98:0/2252856940' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2023-07-14T19:29:40.971 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:29:40 smithi139 ceph-mon[129293]: from='client.? 172.21.15.98:0/288062076' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 8}]: dispatch 2023-07-14T19:29:40.971 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:29:40 smithi139 ceph-mon[129293]: from='client.? 172.21.15.98:0/699395773' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 10}]: dispatch 2023-07-14T19:29:41.353 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:40.007+0000 7f70df053700 1 -- 172.21.15.98:0/457097027 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f70d8100f80 msgr2=0x7f70d8101350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:29:41.353 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:40.007+0000 7f70df053700 1 --2- 172.21.15.98:0/457097027 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f70d8100f80 0x7f70d8101350 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7f70cc009990 tx=0x7f70cc009ca0 comp rx=0 tx=0).stop 2023-07-14T19:29:41.353 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:40.007+0000 7f70df053700 1 -- 172.21.15.98:0/457097027 shutdown_connections 2023-07-14T19:29:41.353 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:40.007+0000 7f70df053700 1 --2- 172.21.15.98:0/457097027 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f70d810a2d0 0x7f70d810c730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:29:41.353 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:40.007+0000 7f70df053700 1 --2- 172.21.15.98:0/457097027 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f70d8100f80 0x7f70d8101350 unknown :-1 s=CLOSED pgs=278 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:29:41.354 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:40.007+0000 7f70df053700 1 --2- 172.21.15.98:0/457097027 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f70d8101890 0x7f70d8109d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:29:41.354 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:40.007+0000 7f70df053700 1 -- 172.21.15.98:0/457097027 >> 172.21.15.98:0/457097027 conn(0x7f70d80fa720 msgr2=0x7f70d80fcb30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:29:41.354 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:40.007+0000 7f70df053700 1 -- 172.21.15.98:0/457097027 shutdown_connections 2023-07-14T19:29:41.354 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:40.007+0000 7f70df053700 1 -- 172.21.15.98:0/457097027 wait complete. 2023-07-14T19:29:41.354 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:40.008+0000 7f70df053700 1 Processor -- start 2023-07-14T19:29:41.355 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:40.008+0000 7f70df053700 1 -- start start 2023-07-14T19:29:41.355 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:40.008+0000 7f70df053700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f70d8100f80 0x7f70d80febe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:29:41.355 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:40.008+0000 7f70df053700 1 --2- >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f70d8101890 0x7f70d80ff120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:29:41.355 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:40.008+0000 7f70df053700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f70d810a2d0 0x7f70d80ff6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:29:41.355 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:40.008+0000 7f70df053700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f70d81a4200 con 0x7f70d810a2d0 2023-07-14T19:29:41.355 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:40.008+0000 7f70df053700 1 -- --> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f70d81a4370 con 0x7f70d8101890 2023-07-14T19:29:41.355 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:40.008+0000 7f70df053700 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f70d81a4530 con 0x7f70d8100f80 2023-07-14T19:29:41.356 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:40.008+0000 7f70dd5f0700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f70d810a2d0 0x7f70d80ff6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:29:41.356 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:40.009+0000 7f70dd5f0700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f70d810a2d0 0x7f70d80ff6a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:46196/0 (socket says 172.21.15.98:46196) 2023-07-14T19:29:41.356 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:40.009+0000 7f70dd5f0700 1 -- 172.21.15.98:0/2252856940 learned_addr learned my addr 172.21.15.98:0/2252856940 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-07-14T19:29:41.356 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:40.009+0000 7f70d7fff700 1 --2- 172.21.15.98:0/2252856940 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f70d8101890 0x7f70d80ff120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:29:41.356 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:40.009+0000 7f70dcdef700 1 --2- 172.21.15.98:0/2252856940 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f70d8100f80 0x7f70d80febe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:29:41.356 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:40.009+0000 7f70dd5f0700 1 -- 172.21.15.98:0/2252856940 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f70d8100f80 msgr2=0x7f70d80febe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:29:41.356 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:40.009+0000 7f70dd5f0700 1 --2- 172.21.15.98:0/2252856940 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f70d8100f80 0x7f70d80febe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:29:41.356 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:40.009+0000 7f70dd5f0700 1 -- 172.21.15.98:0/2252856940 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f70d8101890 msgr2=0x7f70d80ff120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:29:41.357 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:40.009+0000 7f70dd5f0700 1 --2- 172.21.15.98:0/2252856940 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f70d8101890 0x7f70d80ff120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:29:41.357 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:40.009+0000 7f70dd5f0700 1 -- 172.21.15.98:0/2252856940 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f70cc0095f0 con 0x7f70d810a2d0 2023-07-14T19:29:41.357 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:40.009+0000 7f70dd5f0700 1 --2- 172.21.15.98:0/2252856940 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f70d810a2d0 0x7f70d80ff6a0 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7f70d000ef30 tx=0x7f70d000c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:29:41.357 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:40.009+0000 7f70d5ffb700 1 -- 172.21.15.98:0/2252856940 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f70d0018070 con 0x7f70d810a2d0 2023-07-14T19:29:41.357 INFO:teuthology.orchestra.run.smithi098.stdout:55834574910 2023-07-14T19:29:41.358 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:40.009+0000 7f70d5ffb700 1 -- 172.21.15.98:0/2252856940 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f70d000f040 con 0x7f70d810a2d0 2023-07-14T19:29:41.358 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:40.009+0000 7f70df053700 1 -- 172.21.15.98:0/2252856940 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f70d81a47b0 con 0x7f70d810a2d0 2023-07-14T19:29:41.358 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:40.009+0000 7f70d5ffb700 1 -- 172.21.15.98:0/2252856940 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f70d000cdb0 con 0x7f70d810a2d0 2023-07-14T19:29:41.358 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:40.009+0000 7f70df053700 1 -- 172.21.15.98:0/2252856940 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f70d81a4cb0 con 0x7f70d810a2d0 2023-07-14T19:29:41.359 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:40.010+0000 7f70df053700 1 -- 172.21.15.98:0/2252856940 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f70d804e340 con 0x7f70d810a2d0 2023-07-14T19:29:41.359 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:40.011+0000 7f70d5ffb700 1 -- 172.21.15.98:0/2252856940 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 17) v1 ==== 135766+0+0 (secure 0 0 0) 0x7f70d0003a00 con 0x7f70d810a2d0 2023-07-14T19:29:41.359 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:40.012+0000 7f70d5ffb700 1 --2- 172.21.15.98:0/2252856940 >> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] conn(0x7f70c80a1260 0x7f70c80a3710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:29:41.359 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:40.012+0000 7f70dcdef700 1 --2- 172.21.15.98:0/2252856940 >> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] conn(0x7f70c80a1260 0x7f70c80a3710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:29:41.359 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:40.012+0000 7f70d5ffb700 1 -- 172.21.15.98:0/2252856940 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 7979+0+0 (secure 0 0 0) 0x7f70d00cc420 con 0x7f70d810a2d0 2023-07-14T19:29:41.359 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:40.012+0000 7f70dcdef700 1 --2- 172.21.15.98:0/2252856940 >> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] conn(0x7f70c80a1260 0x7f70c80a3710 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f70cc006010 tx=0x7f70cc00b260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:29:41.359 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:40.021+0000 7f70d5ffb700 1 -- 172.21.15.98:0/2252856940 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f70d0095c50 con 0x7f70d810a2d0 2023-07-14T19:29:41.359 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:40.255+0000 7f70df053700 1 -- 172.21.15.98:0/2252856940 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) v1 -- 0x7f70d81a4fe0 con 0x7f70d810a2d0 2023-07-14T19:29:41.360 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:40.256+0000 7f70d5ffb700 1 -- 172.21.15.98:0/2252856940 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f70d0099270 con 0x7f70d810a2d0 2023-07-14T19:29:41.360 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:40.258+0000 7f70df053700 1 -- 172.21.15.98:0/2252856940 >> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] conn(0x7f70c80a1260 msgr2=0x7f70c80a3710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:29:41.360 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:40.258+0000 7f70df053700 1 --2- 172.21.15.98:0/2252856940 >> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] conn(0x7f70c80a1260 0x7f70c80a3710 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f70cc006010 tx=0x7f70cc00b260 comp rx=0 tx=0).stop 2023-07-14T19:29:41.360 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:40.258+0000 7f70df053700 1 -- 172.21.15.98:0/2252856940 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f70d810a2d0 msgr2=0x7f70d80ff6a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:29:41.360 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:40.258+0000 7f70df053700 1 --2- 172.21.15.98:0/2252856940 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f70d810a2d0 0x7f70d80ff6a0 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7f70d000ef30 tx=0x7f70d000c3a0 comp rx=0 tx=0).stop 2023-07-14T19:29:41.360 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:40.258+0000 7f70df053700 1 -- 172.21.15.98:0/2252856940 shutdown_connections 2023-07-14T19:29:41.360 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:40.258+0000 7f70df053700 1 --2- 172.21.15.98:0/2252856940 >> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] conn(0x7f70c80a1260 0x7f70c80a3710 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:29:41.361 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:40.258+0000 7f70df053700 1 --2- 172.21.15.98:0/2252856940 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f70d8100f80 0x7f70d80febe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:29:41.361 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:40.258+0000 7f70df053700 1 --2- 172.21.15.98:0/2252856940 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f70d810a2d0 0x7f70d80ff6a0 unknown :-1 s=CLOSED pgs=279 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:29:41.361 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:40.258+0000 7f70df053700 1 --2- 172.21.15.98:0/2252856940 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f70d8101890 0x7f70d80ff120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:29:41.361 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:40.258+0000 7f70df053700 1 -- 172.21.15.98:0/2252856940 >> 172.21.15.98:0/2252856940 conn(0x7f70d80fa720 msgr2=0x7f70d810c500 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:29:41.361 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:40.258+0000 7f70df053700 1 -- 172.21.15.98:0/2252856940 shutdown_connections 2023-07-14T19:29:41.361 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:40.258+0000 7f70df053700 1 -- 172.21.15.98:0/2252856940 wait complete. 2023-07-14T19:29:41.385 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:41.383+0000 7ff0f93f0700 1 -- 172.21.15.98:0/1922503957 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff0f4100980 msgr2=0x7ff0f41090e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:29:41.385 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:41.383+0000 7ff0f93f0700 1 --2- 172.21.15.98:0/1922503957 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff0f4100980 0x7ff0f41090e0 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7ff0e8009990 tx=0x7ff0e8009ca0 comp rx=0 tx=0).stop 2023-07-14T19:29:41.385 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:41.383+0000 7ff0f93f0700 1 -- 172.21.15.98:0/1922503957 shutdown_connections 2023-07-14T19:29:41.386 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:41.383+0000 7ff0f93f0700 1 --2- 172.21.15.98:0/1922503957 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7ff0f4109620 0x7ff0f410ba00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:29:41.386 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:41.383+0000 7ff0f93f0700 1 --2- 172.21.15.98:0/1922503957 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7ff0f4100070 0x7ff0f4100440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:29:41.386 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:41.383+0000 7ff0f93f0700 1 --2- 172.21.15.98:0/1922503957 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff0f4100980 0x7ff0f41090e0 unknown :-1 s=CLOSED pgs=283 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:29:41.386 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:41.383+0000 7ff0f93f0700 1 -- 172.21.15.98:0/1922503957 >> 172.21.15.98:0/1922503957 conn(0x7ff0f4074550 msgr2=0x7ff0f4074950 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:29:41.386 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:41.383+0000 7ff0f93f0700 1 -- 172.21.15.98:0/1922503957 shutdown_connections 2023-07-14T19:29:41.386 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:41.383+0000 7ff0f93f0700 1 -- 172.21.15.98:0/1922503957 wait complete. 2023-07-14T19:29:41.386 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:41.384+0000 7ff0f93f0700 1 Processor -- start 2023-07-14T19:29:41.388 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:41.384+0000 7ff0f93f0700 1 -- start start 2023-07-14T19:29:41.388 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:41.384+0000 7ff0f93f0700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7ff0f4100070 0x7ff0f4196bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:29:41.388 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:41.384+0000 7ff0f93f0700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff0f4100980 0x7ff0f4197130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:29:41.388 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:41.384+0000 7ff0f93f0700 1 --2- >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7ff0f4109620 0x7ff0f41a1c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:29:41.389 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:41.384+0000 7ff0f93f0700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff0f4197a40 con 0x7ff0f4100980 2023-07-14T19:29:41.389 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:41.384+0000 7ff0f93f0700 1 -- --> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff0f41a21b0 con 0x7ff0f4109620 2023-07-14T19:29:41.389 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:41.384+0000 7ff0f93f0700 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff0f41a2320 con 0x7ff0f4100070 2023-07-14T19:29:41.389 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:41.384+0000 7ff0f27fc700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff0f4100980 0x7ff0f4197130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:29:41.389 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:41.385+0000 7ff0f27fc700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff0f4100980 0x7ff0f4197130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:46236/0 (socket says 172.21.15.98:46236) 2023-07-14T19:29:41.389 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:41.385+0000 7ff0f27fc700 1 -- 172.21.15.98:0/2875482029 learned_addr learned my addr 172.21.15.98:0/2875482029 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-07-14T19:29:41.389 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:41.385+0000 7ff0f2ffd700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7ff0f4100070 0x7ff0f4196bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:29:41.389 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:41.385+0000 7ff0f37fe700 1 --2- 172.21.15.98:0/2875482029 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7ff0f4109620 0x7ff0f41a1c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:29:41.390 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:41.385+0000 7ff0f27fc700 1 -- 172.21.15.98:0/2875482029 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7ff0f4100070 msgr2=0x7ff0f4196bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:29:41.390 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:41.385+0000 7ff0f27fc700 1 --2- 172.21.15.98:0/2875482029 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7ff0f4100070 0x7ff0f4196bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:29:41.390 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:41.385+0000 7ff0f27fc700 1 -- 172.21.15.98:0/2875482029 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7ff0f4109620 msgr2=0x7ff0f41a1c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:29:41.390 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:41.385+0000 7ff0f27fc700 1 --2- 172.21.15.98:0/2875482029 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7ff0f4109620 0x7ff0f41a1c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:29:41.390 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:41.385+0000 7ff0f27fc700 1 -- 172.21.15.98:0/2875482029 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff0e80095f0 con 0x7ff0f4100980 2023-07-14T19:29:41.390 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:41.385+0000 7ff0f37fe700 1 --2- 172.21.15.98:0/2875482029 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7ff0f4109620 0x7ff0f41a1c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-07-14T19:29:41.390 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:41.385+0000 7ff0f27fc700 1 --2- 172.21.15.98:0/2875482029 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff0f4100980 0x7ff0f4197130 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7ff0e80095c0 tx=0x7ff0e800f710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:29:41.391 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:41.385+0000 7ff0dbfff700 1 -- 172.21.15.98:0/2875482029 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff0e801d070 con 0x7ff0f4100980 2023-07-14T19:29:41.391 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:41.385+0000 7ff0f93f0700 1 -- 172.21.15.98:0/2875482029 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff0f41a25a0 con 0x7ff0f4100980 2023-07-14T19:29:41.391 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:41.385+0000 7ff0dbfff700 1 -- 172.21.15.98:0/2875482029 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7ff0e800fb50 con 0x7ff0f4100980 2023-07-14T19:29:41.391 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:41.385+0000 7ff0dbfff700 1 -- 172.21.15.98:0/2875482029 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff0e80173f0 con 0x7ff0f4100980 2023-07-14T19:29:41.391 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:41.385+0000 7ff0f93f0700 1 -- 172.21.15.98:0/2875482029 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff0f41a2a90 con 0x7ff0f4100980 2023-07-14T19:29:41.392 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:41.386+0000 7ff0f93f0700 1 -- 172.21.15.98:0/2875482029 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff0f404e340 con 0x7ff0f4100980 2023-07-14T19:29:41.392 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:41.387+0000 7ff0dbfff700 1 -- 172.21.15.98:0/2875482029 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 17) v1 ==== 135766+0+0 (secure 0 0 0) 0x7ff0e800fcc0 con 0x7ff0f4100980 2023-07-14T19:29:41.392 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:41.388+0000 7ff0dbfff700 1 --2- 172.21.15.98:0/2875482029 >> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] conn(0x7ff0dc0a11e0 0x7ff0dc0a3690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:29:41.393 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:41.388+0000 7ff0f2ffd700 1 --2- 172.21.15.98:0/2875482029 >> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] conn(0x7ff0dc0a11e0 0x7ff0dc0a3690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:29:41.393 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:41.388+0000 7ff0dbfff700 1 -- 172.21.15.98:0/2875482029 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 7979+0+0 (secure 0 0 0) 0x7ff0e80cdb90 con 0x7ff0f4100980 2023-07-14T19:29:41.393 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:41.388+0000 7ff0f2ffd700 1 --2- 172.21.15.98:0/2875482029 >> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] conn(0x7ff0dc0a11e0 0x7ff0dc0a3690 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7ff0f4072b80 tx=0x7ff0e400b410 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:29:41.393 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:41.392+0000 7ff0dbfff700 1 -- 172.21.15.98:0/2875482029 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7ff0e80973c0 con 0x7ff0f4100980 2023-07-14T19:29:41.574 INFO:tasks.cephadm.ceph_manager.ceph:need seq 244813135901 got 244813135903 for osd.8 2023-07-14T19:29:41.575 DEBUG:teuthology.parallel:result is None 2023-07-14T19:29:41.591 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:41.590+0000 7ff0f93f0700 1 -- 172.21.15.98:0/2875482029 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 6} v 0) v1 -- 0x7ff0f41a2d50 con 0x7ff0f4100980 2023-07-14T19:29:41.592 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:41.590+0000 7ff0dbfff700 1 -- 172.21.15.98:0/2875482029 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 6}]=0 v0) v1 ==== 74+0+13 (secure 0 0 0) 0x7ff0e809a9e0 con 0x7ff0f4100980 2023-07-14T19:29:41.592 INFO:teuthology.orchestra.run.smithi098.stdout:193273528361 2023-07-14T19:29:41.594 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:41.592+0000 7ff0f93f0700 1 -- 172.21.15.98:0/2875482029 >> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] conn(0x7ff0dc0a11e0 msgr2=0x7ff0dc0a3690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:29:41.594 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:41.592+0000 7ff0f93f0700 1 --2- 172.21.15.98:0/2875482029 >> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] conn(0x7ff0dc0a11e0 0x7ff0dc0a3690 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7ff0f4072b80 tx=0x7ff0e400b410 comp rx=0 tx=0).stop 2023-07-14T19:29:41.594 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:41.592+0000 7ff0f93f0700 1 -- 172.21.15.98:0/2875482029 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff0f4100980 msgr2=0x7ff0f4197130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:29:41.594 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:41.592+0000 7ff0f93f0700 1 --2- 172.21.15.98:0/2875482029 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff0f4100980 0x7ff0f4197130 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7ff0e80095c0 tx=0x7ff0e800f710 comp rx=0 tx=0).stop 2023-07-14T19:29:41.594 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:41.592+0000 7ff0f93f0700 1 -- 172.21.15.98:0/2875482029 shutdown_connections 2023-07-14T19:29:41.594 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:41.592+0000 7ff0f93f0700 1 --2- 172.21.15.98:0/2875482029 >> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] conn(0x7ff0dc0a11e0 0x7ff0dc0a3690 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:29:41.595 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:41.592+0000 7ff0f93f0700 1 --2- 172.21.15.98:0/2875482029 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7ff0f4100070 0x7ff0f4196bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:29:41.595 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:41.592+0000 7ff0f93f0700 1 --2- 172.21.15.98:0/2875482029 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7ff0f4109620 0x7ff0f41a1c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:29:41.595 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:41.592+0000 7ff0f93f0700 1 --2- 172.21.15.98:0/2875482029 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff0f4100980 0x7ff0f4197130 unknown :-1 s=CLOSED pgs=284 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:29:41.595 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:41.592+0000 7ff0f93f0700 1 -- 172.21.15.98:0/2875482029 >> 172.21.15.98:0/2875482029 conn(0x7ff0f4074550 msgr2=0x7ff0f40fd030 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:29:41.595 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:41.592+0000 7ff0f93f0700 1 -- 172.21.15.98:0/2875482029 shutdown_connections 2023-07-14T19:29:41.595 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:41.592+0000 7ff0f93f0700 1 -- 172.21.15.98:0/2875482029 wait complete. 2023-07-14T19:29:41.678 INFO:tasks.cephadm.ceph_manager.ceph:need seq 107374182450 got 107374182453 for osd.3 2023-07-14T19:29:41.678 DEBUG:teuthology.parallel:result is None 2023-07-14T19:29:42.077 INFO:tasks.cephadm.ceph_manager.ceph:need seq 300647710741 got 300647710744 for osd.10 2023-07-14T19:29:42.077 DEBUG:teuthology.parallel:result is None 2023-07-14T19:29:42.433 INFO:tasks.cephadm.ceph_manager.ceph:need seq 193273528358 got 193273528361 for osd.6 2023-07-14T19:29:42.433 DEBUG:teuthology.parallel:result is None 2023-07-14T19:29:42.548 INFO:tasks.cephadm.ceph_manager.ceph:need seq 55834574908 got 55834574910 for osd.1 2023-07-14T19:29:42.549 DEBUG:teuthology.parallel:result is None 2023-07-14T19:29:42.578 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:29:42 smithi098 ceph-mon[127486]: from='client.? 172.21.15.98:0/2875482029' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2023-07-14T19:29:42.578 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:29:42 smithi098 ceph-mon[127486]: pgmap v273: 97 pgs: 97 active+clean; 634 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2023-07-14T19:29:42.648 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:42.646+0000 7f2558fcc700 1 -- 172.21.15.98:0/1629756822 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f25540feed0 msgr2=0x7f25540ff2a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:29:42.648 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:42.646+0000 7f2558fcc700 1 --2- 172.21.15.98:0/1629756822 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f25540feed0 0x7f25540ff2a0 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7f254400b0d0 tx=0x7f254400b3e0 comp rx=0 tx=0).stop 2023-07-14T19:29:42.649 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:42.647+0000 7f2558fcc700 1 -- 172.21.15.98:0/1629756822 shutdown_connections 2023-07-14T19:29:42.649 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:42.647+0000 7f2558fcc700 1 --2- 172.21.15.98:0/1629756822 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f2554103f20 0x7f255410baa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:29:42.649 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:42.647+0000 7f2558fcc700 1 --2- 172.21.15.98:0/1629756822 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f25540feed0 0x7f25540ff2a0 unknown :-1 s=CLOSED pgs=285 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:29:42.649 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:42.647+0000 7f2558fcc700 1 --2- 172.21.15.98:0/1629756822 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f25540ff870 0x7f25541037f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:29:42.649 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:42.647+0000 7f2558fcc700 1 -- 172.21.15.98:0/1629756822 >> 172.21.15.98:0/1629756822 conn(0x7f2554075d80 msgr2=0x7f2554076180 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:29:42.649 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:42.647+0000 7f2558fcc700 1 -- 172.21.15.98:0/1629756822 shutdown_connections 2023-07-14T19:29:42.649 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:42.647+0000 7f2558fcc700 1 -- 172.21.15.98:0/1629756822 wait complete. 2023-07-14T19:29:42.651 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:42.647+0000 7f2558fcc700 1 Processor -- start 2023-07-14T19:29:42.652 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:42.647+0000 7f2558fcc700 1 -- start start 2023-07-14T19:29:42.652 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:42.647+0000 7f2558fcc700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f25540feed0 0x7f2554196e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:29:42.652 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:42.648+0000 7f2558fcc700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f25540ff870 0x7f2554197390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:29:42.653 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:42.648+0000 7f2558fcc700 1 --2- >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f2554103f20 0x7f25541a1ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:29:42.653 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:42.648+0000 7f2558fcc700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2554197ca0 con 0x7f25540feed0 2023-07-14T19:29:42.653 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:42.648+0000 7f2558fcc700 1 -- --> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f25541a2410 con 0x7f2554103f20 2023-07-14T19:29:42.653 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:42.648+0000 7f2558fcc700 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f25541a2580 con 0x7f25540ff870 2023-07-14T19:29:42.653 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:42.648+0000 7f255259c700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f25540feed0 0x7f2554196e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:29:42.654 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:42.648+0000 7f255259c700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f25540feed0 0x7f2554196e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:46246/0 (socket says 172.21.15.98:46246) 2023-07-14T19:29:42.654 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:42.648+0000 7f255259c700 1 -- 172.21.15.98:0/822280173 learned_addr learned my addr 172.21.15.98:0/822280173 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-07-14T19:29:42.654 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:42.648+0000 7f2551d9b700 1 --2- 172.21.15.98:0/822280173 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f25540ff870 0x7f2554197390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:29:42.654 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:42.648+0000 7f2552d9d700 1 --2- 172.21.15.98:0/822280173 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f2554103f20 0x7f25541a1ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:29:42.654 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:42.648+0000 7f255259c700 1 -- 172.21.15.98:0/822280173 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f25540ff870 msgr2=0x7f2554197390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:29:42.654 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:42.648+0000 7f255259c700 1 --2- 172.21.15.98:0/822280173 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f25540ff870 0x7f2554197390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:29:42.655 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:42.648+0000 7f255259c700 1 -- 172.21.15.98:0/822280173 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f2554103f20 msgr2=0x7f25541a1ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:29:42.655 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:42.648+0000 7f255259c700 1 --2- 172.21.15.98:0/822280173 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f2554103f20 0x7f25541a1ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:29:42.655 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:42.648+0000 7f255259c700 1 -- 172.21.15.98:0/822280173 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2544009d00 con 0x7f25540feed0 2023-07-14T19:29:42.655 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:42.648+0000 7f255259c700 1 --2- 172.21.15.98:0/822280173 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f25540feed0 0x7f2554196e50 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7f25440078a0 tx=0x7f25440078d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:29:42.655 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:42.648+0000 7f254b7fe700 1 -- 172.21.15.98:0/822280173 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f254400e040 con 0x7f25540feed0 2023-07-14T19:29:42.656 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:42.648+0000 7f2558fcc700 1 -- 172.21.15.98:0/822280173 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f25541a2800 con 0x7f25540feed0 2023-07-14T19:29:42.656 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:42.648+0000 7f254b7fe700 1 -- 172.21.15.98:0/822280173 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f2544003e10 con 0x7f25540feed0 2023-07-14T19:29:42.656 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:42.648+0000 7f254b7fe700 1 -- 172.21.15.98:0/822280173 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f254401b940 con 0x7f25540feed0 2023-07-14T19:29:42.656 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:42.648+0000 7f2558fcc700 1 -- 172.21.15.98:0/822280173 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f25541a2c20 con 0x7f25540feed0 2023-07-14T19:29:42.656 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:42.649+0000 7f2558fcc700 1 -- 172.21.15.98:0/822280173 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f255404e340 con 0x7f25540feed0 2023-07-14T19:29:42.656 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:42.650+0000 7f254b7fe700 1 -- 172.21.15.98:0/822280173 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 17) v1 ==== 135766+0+0 (secure 0 0 0) 0x7f2544012470 con 0x7f25540feed0 2023-07-14T19:29:42.658 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:42.651+0000 7f254b7fe700 1 --2- 172.21.15.98:0/822280173 >> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] conn(0x7f25400a11e0 0x7f25400a3690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:29:42.658 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:42.651+0000 7f2551d9b700 1 --2- 172.21.15.98:0/822280173 >> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] conn(0x7f25400a11e0 0x7f25400a3690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:29:42.658 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:42.651+0000 7f254b7fe700 1 -- 172.21.15.98:0/822280173 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 7979+0+0 (secure 0 0 0) 0x7f25440cdea0 con 0x7f25540feed0 2023-07-14T19:29:42.658 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:42.651+0000 7f2551d9b700 1 --2- 172.21.15.98:0/822280173 >> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] conn(0x7f25400a11e0 0x7f25400a3690 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f2554069310 tx=0x7f253c009190 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:29:42.658 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:42.655+0000 7f254b7fe700 1 -- 172.21.15.98:0/822280173 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f25440976d0 con 0x7f25540feed0 2023-07-14T19:29:42.691 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:29:42 smithi101 ceph-mon[128807]: from='client.? 172.21.15.98:0/2875482029' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2023-07-14T19:29:42.691 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:29:42 smithi101 ceph-mon[128807]: pgmap v273: 97 pgs: 97 active+clean; 634 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2023-07-14T19:29:42.720 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:29:42 smithi139 ceph-mon[129293]: from='client.? 172.21.15.98:0/2875482029' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2023-07-14T19:29:42.720 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:29:42 smithi139 ceph-mon[129293]: pgmap v273: 97 pgs: 97 active+clean; 634 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2023-07-14T19:29:42.854 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:42.853+0000 7f2558fcc700 1 -- 172.21.15.98:0/822280173 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 2} v 0) v1 -- 0x7f25541a2e40 con 0x7f25540feed0 2023-07-14T19:29:42.854 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:42.853+0000 7f254b7fe700 1 -- 172.21.15.98:0/822280173 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 2}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f254409acf0 con 0x7f25540feed0 2023-07-14T19:29:42.855 INFO:teuthology.orchestra.run.smithi098.stdout:77309411385 2023-07-14T19:29:42.857 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:42.855+0000 7f2558fcc700 1 -- 172.21.15.98:0/822280173 >> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] conn(0x7f25400a11e0 msgr2=0x7f25400a3690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:29:42.857 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:42.855+0000 7f2558fcc700 1 --2- 172.21.15.98:0/822280173 >> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] conn(0x7f25400a11e0 0x7f25400a3690 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f2554069310 tx=0x7f253c009190 comp rx=0 tx=0).stop 2023-07-14T19:29:42.857 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:42.855+0000 7f2558fcc700 1 -- 172.21.15.98:0/822280173 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f25540feed0 msgr2=0x7f2554196e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:29:42.857 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:42.855+0000 7f2558fcc700 1 --2- 172.21.15.98:0/822280173 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f25540feed0 0x7f2554196e50 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7f25440078a0 tx=0x7f25440078d0 comp rx=0 tx=0).stop 2023-07-14T19:29:42.857 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:42.855+0000 7f2558fcc700 1 -- 172.21.15.98:0/822280173 shutdown_connections 2023-07-14T19:29:42.858 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:42.855+0000 7f2558fcc700 1 --2- 172.21.15.98:0/822280173 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f2554103f20 0x7f25541a1ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:29:42.858 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:42.855+0000 7f2558fcc700 1 --2- 172.21.15.98:0/822280173 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f25540feed0 0x7f2554196e50 unknown :-1 s=CLOSED pgs=286 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:29:42.858 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:42.855+0000 7f2558fcc700 1 --2- 172.21.15.98:0/822280173 >> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] conn(0x7f25400a11e0 0x7f25400a3690 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:29:42.858 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:42.855+0000 7f2558fcc700 1 --2- 172.21.15.98:0/822280173 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f25540ff870 0x7f2554197390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:29:42.858 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:42.855+0000 7f2558fcc700 1 -- 172.21.15.98:0/822280173 >> 172.21.15.98:0/822280173 conn(0x7f2554075d80 msgr2=0x7f25540fc4a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:29:42.858 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:42.855+0000 7f2558fcc700 1 -- 172.21.15.98:0/822280173 shutdown_connections 2023-07-14T19:29:42.858 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:42.855+0000 7f2558fcc700 1 -- 172.21.15.98:0/822280173 wait complete. 2023-07-14T19:29:42.983 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:42.981+0000 7f4c1c48e700 1 -- 172.21.15.98:0/141800590 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4c1410a3a0 msgr2=0x7f4c1410c7f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:29:42.984 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:42.981+0000 7f4c1c48e700 1 --2- 172.21.15.98:0/141800590 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4c1410a3a0 0x7f4c1410c7f0 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7f4c0c00b0a0 tx=0x7f4c0c00b3b0 comp rx=0 tx=0).stop 2023-07-14T19:29:42.984 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:42.982+0000 7f4c1c48e700 1 -- 172.21.15.98:0/141800590 shutdown_connections 2023-07-14T19:29:42.984 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:42.982+0000 7f4c1c48e700 1 --2- 172.21.15.98:0/141800590 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4c1410a3a0 0x7f4c1410c7f0 unknown :-1 s=CLOSED pgs=287 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:29:42.984 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:42.982+0000 7f4c1c48e700 1 --2- 172.21.15.98:0/141800590 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f4c14100f20 0x7f4c141012f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:29:42.984 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:42.982+0000 7f4c1c48e700 1 --2- 172.21.15.98:0/141800590 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f4c141018c0 0x7f4c14109d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:29:42.984 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:42.982+0000 7f4c1c48e700 1 -- 172.21.15.98:0/141800590 >> 172.21.15.98:0/141800590 conn(0x7f4c14075c90 msgr2=0x7f4c14076090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:29:42.984 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:42.982+0000 7f4c1c48e700 1 -- 172.21.15.98:0/141800590 shutdown_connections 2023-07-14T19:29:42.985 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:42.982+0000 7f4c1c48e700 1 -- 172.21.15.98:0/141800590 wait complete. 2023-07-14T19:29:42.986 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:42.982+0000 7f4c1c48e700 1 Processor -- start 2023-07-14T19:29:42.986 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:42.983+0000 7f4c1c48e700 1 -- start start 2023-07-14T19:29:42.986 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:42.983+0000 7f4c1c48e700 1 --2- >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f4c14100f20 0x7f4c14195f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:29:42.987 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:42.983+0000 7f4c1c48e700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4c141018c0 0x7f4c141964d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:29:42.987 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:42.983+0000 7f4c1c48e700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f4c1410a3a0 0x7f4c1419a940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:29:42.987 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:42.983+0000 7f4c1c48e700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4c1419af50 con 0x7f4c141018c0 2023-07-14T19:29:42.987 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:42.983+0000 7f4c1c48e700 1 -- --> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4c1419b0c0 con 0x7f4c14100f20 2023-07-14T19:29:42.987 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:42.983+0000 7f4c1c48e700 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4c1419b280 con 0x7f4c1410a3a0 2023-07-14T19:29:42.987 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:42.983+0000 7f4c19a29700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4c141018c0 0x7f4c141964d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:29:42.987 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:42.983+0000 7f4c19a29700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4c141018c0 0x7f4c141964d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:46262/0 (socket says 172.21.15.98:46262) 2023-07-14T19:29:42.988 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:42.983+0000 7f4c19a29700 1 -- 172.21.15.98:0/1586540498 learned_addr learned my addr 172.21.15.98:0/1586540498 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-07-14T19:29:42.988 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:42.983+0000 7f4c1a22a700 1 --2- 172.21.15.98:0/1586540498 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f4c14100f20 0x7f4c14195f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:29:42.988 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:42.983+0000 7f4c1aa2b700 1 --2- 172.21.15.98:0/1586540498 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f4c1410a3a0 0x7f4c1419a940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:29:42.988 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:42.983+0000 7f4c19a29700 1 -- 172.21.15.98:0/1586540498 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f4c1410a3a0 msgr2=0x7f4c1419a940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:29:42.988 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:42.983+0000 7f4c19a29700 1 --2- 172.21.15.98:0/1586540498 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f4c1410a3a0 0x7f4c1419a940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:29:42.988 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:42.983+0000 7f4c19a29700 1 -- 172.21.15.98:0/1586540498 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f4c14100f20 msgr2=0x7f4c14195f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:29:42.988 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:42.983+0000 7f4c19a29700 1 --2- 172.21.15.98:0/1586540498 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f4c14100f20 0x7f4c14195f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:29:42.989 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:42.983+0000 7f4c19a29700 1 -- 172.21.15.98:0/1586540498 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4c0c009d00 con 0x7f4c141018c0 2023-07-14T19:29:42.989 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:42.983+0000 7f4c1aa2b700 1 --2- 172.21.15.98:0/1586540498 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f4c1410a3a0 0x7f4c1419a940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-07-14T19:29:42.989 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:42.983+0000 7f4c19a29700 1 --2- 172.21.15.98:0/1586540498 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4c141018c0 0x7f4c141964d0 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7f4c0400e970 tx=0x7f4c0400ec80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:29:42.989 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:42.984+0000 7f4c0b7fe700 1 -- 172.21.15.98:0/1586540498 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c04009790 con 0x7f4c141018c0 2023-07-14T19:29:42.989 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:42.984+0000 7f4c1c48e700 1 -- 172.21.15.98:0/1586540498 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4c1419b560 con 0x7f4c141018c0 2023-07-14T19:29:42.989 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:42.984+0000 7f4c0b7fe700 1 -- 172.21.15.98:0/1586540498 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f4c0400eec0 con 0x7f4c141018c0 2023-07-14T19:29:42.989 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:42.984+0000 7f4c0b7fe700 1 -- 172.21.15.98:0/1586540498 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c04005590 con 0x7f4c141018c0 2023-07-14T19:29:42.989 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:42.984+0000 7f4c1c48e700 1 -- 172.21.15.98:0/1586540498 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4c141a2dc0 con 0x7f4c141018c0 2023-07-14T19:29:42.990 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:42.985+0000 7f4c1c48e700 1 -- 172.21.15.98:0/1586540498 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4c1404e340 con 0x7f4c141018c0 2023-07-14T19:29:42.990 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:42.986+0000 7f4c0b7fe700 1 -- 172.21.15.98:0/1586540498 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 17) v1 ==== 135766+0+0 (secure 0 0 0) 0x7f4c0400cbd0 con 0x7f4c141018c0 2023-07-14T19:29:42.990 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:42.986+0000 7f4c0b7fe700 1 --2- 172.21.15.98:0/1586540498 >> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] conn(0x7f4c000a11e0 0x7f4c000a3690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:29:42.991 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:42.986+0000 7f4c1a22a700 1 --2- 172.21.15.98:0/1586540498 >> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] conn(0x7f4c000a11e0 0x7f4c000a3690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:29:42.991 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:42.986+0000 7f4c0b7fe700 1 -- 172.21.15.98:0/1586540498 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 7979+0+0 (secure 0 0 0) 0x7f4c04014070 con 0x7f4c141018c0 2023-07-14T19:29:42.991 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:42.987+0000 7f4c1a22a700 1 --2- 172.21.15.98:0/1586540498 >> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] conn(0x7f4c000a11e0 0x7f4c000a3690 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f4c10005950 tx=0x7f4c1000b410 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:29:42.992 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:42.990+0000 7f4c0b7fe700 1 -- 172.21.15.98:0/1586540498 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f4c04096250 con 0x7f4c141018c0 2023-07-14T19:29:43.186 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:43.184+0000 7f4c1c48e700 1 -- 172.21.15.98:0/1586540498 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 5} v 0) v1 -- 0x7f4c141972f0 con 0x7f4c141018c0 2023-07-14T19:29:43.186 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:43.185+0000 7f4c0b7fe700 1 -- 172.21.15.98:0/1586540498 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 5}]=0 v0) v1 ==== 74+0+13 (secure 0 0 0) 0x7f4c04004ef0 con 0x7f4c141018c0 2023-07-14T19:29:43.187 INFO:teuthology.orchestra.run.smithi098.stdout:158913789996 2023-07-14T19:29:43.189 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:43.187+0000 7f4c1c48e700 1 -- 172.21.15.98:0/1586540498 >> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] conn(0x7f4c000a11e0 msgr2=0x7f4c000a3690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:29:43.189 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:43.187+0000 7f4c1c48e700 1 --2- 172.21.15.98:0/1586540498 >> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] conn(0x7f4c000a11e0 0x7f4c000a3690 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f4c10005950 tx=0x7f4c1000b410 comp rx=0 tx=0).stop 2023-07-14T19:29:43.189 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:43.187+0000 7f4c1c48e700 1 -- 172.21.15.98:0/1586540498 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4c141018c0 msgr2=0x7f4c141964d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:29:43.189 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:43.187+0000 7f4c1c48e700 1 --2- 172.21.15.98:0/1586540498 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4c141018c0 0x7f4c141964d0 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7f4c0400e970 tx=0x7f4c0400ec80 comp rx=0 tx=0).stop 2023-07-14T19:29:43.190 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:43.187+0000 7f4c1c48e700 1 -- 172.21.15.98:0/1586540498 shutdown_connections 2023-07-14T19:29:43.190 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:43.187+0000 7f4c1c48e700 1 --2- 172.21.15.98:0/1586540498 >> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] conn(0x7f4c000a11e0 0x7f4c000a3690 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:29:43.190 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:43.187+0000 7f4c1c48e700 1 --2- 172.21.15.98:0/1586540498 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f4c1410a3a0 0x7f4c1419a940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:29:43.190 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:43.187+0000 7f4c1c48e700 1 --2- 172.21.15.98:0/1586540498 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f4c14100f20 0x7f4c14195f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:29:43.190 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:43.187+0000 7f4c1c48e700 1 --2- 172.21.15.98:0/1586540498 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4c141018c0 0x7f4c141964d0 unknown :-1 s=CLOSED pgs=288 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:29:43.191 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:43.187+0000 7f4c1c48e700 1 -- 172.21.15.98:0/1586540498 >> 172.21.15.98:0/1586540498 conn(0x7f4c14075c90 msgr2=0x7f4c14103d40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:29:43.191 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:43.187+0000 7f4c1c48e700 1 -- 172.21.15.98:0/1586540498 shutdown_connections 2023-07-14T19:29:43.191 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:43.187+0000 7f4c1c48e700 1 -- 172.21.15.98:0/1586540498 wait complete. 2023-07-14T19:29:43.343 INFO:tasks.cephadm.ceph_manager.ceph:need seq 77309411383 got 77309411385 for osd.2 2023-07-14T19:29:43.343 DEBUG:teuthology.parallel:result is None 2023-07-14T19:29:43.531 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:29:43 smithi098 ceph-mon[127486]: from='client.? 172.21.15.98:0/822280173' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2023-07-14T19:29:43.531 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:29:43 smithi098 ceph-mon[127486]: from='client.? 172.21.15.98:0/1586540498' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2023-07-14T19:29:43.691 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:29:43 smithi101 ceph-mon[128807]: from='client.? 172.21.15.98:0/822280173' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2023-07-14T19:29:43.691 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:29:43 smithi101 ceph-mon[128807]: from='client.? 172.21.15.98:0/1586540498' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2023-07-14T19:29:43.695 INFO:tasks.cephadm.ceph_manager.ceph:need seq 158913789994 got 158913789996 for osd.5 2023-07-14T19:29:43.695 DEBUG:teuthology.parallel:result is None 2023-07-14T19:29:43.696 INFO:tasks.cephadm.ceph_manager.ceph:waiting for clean 2023-07-14T19:29:43.697 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81c9387d2e7576da065747ec9c16d1c7d69016ec shell --fsid 9adea648-227b-11ee-9b33-001a4aab830c -- ceph pg dump --format=json 2023-07-14T19:29:43.720 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:29:43 smithi139 ceph-mon[129293]: from='client.? 172.21.15.98:0/822280173' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2023-07-14T19:29:43.721 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:29:43 smithi139 ceph-mon[129293]: from='client.? 172.21.15.98:0/1586540498' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2023-07-14T19:29:44.249 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/9adea648-227b-11ee-9b33-001a4aab830c/mon.a/config 2023-07-14T19:29:44.608 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:29:44 smithi098 ceph-mon[127486]: pgmap v274: 97 pgs: 97 active+clean; 634 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2023-07-14T19:29:44.691 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:29:44 smithi101 ceph-mon[128807]: pgmap v274: 97 pgs: 97 active+clean; 634 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2023-07-14T19:29:44.720 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:29:44 smithi139 ceph-mon[129293]: pgmap v274: 97 pgs: 97 active+clean; 634 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2023-07-14T19:29:45.349 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:45.347+0000 7f8ef87f8700 1 -- 172.21.15.98:0/46639798 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8ef00fe4b0 msgr2=0x7f8ef00fe880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:29:45.350 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:45.347+0000 7f8ef87f8700 1 --2- 172.21.15.98:0/46639798 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8ef00fe4b0 0x7f8ef00fe880 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7f8ee40099c0 tx=0x7f8ee4009cd0 comp rx=0 tx=0).stop 2023-07-14T19:29:45.350 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:45.347+0000 7f8ef87f8700 1 -- 172.21.15.98:0/46639798 shutdown_connections 2023-07-14T19:29:45.350 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:45.347+0000 7f8ef87f8700 1 --2- 172.21.15.98:0/46639798 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f8ef0105010 0x7f8ef010cbc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:29:45.350 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:45.347+0000 7f8ef87f8700 1 --2- 172.21.15.98:0/46639798 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8ef00fe4b0 0x7f8ef00fe880 unknown :-1 s=CLOSED pgs=289 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:29:45.350 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:45.347+0000 7f8ef87f8700 1 --2- 172.21.15.98:0/46639798 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f8ef00fee50 0x7f8ef0104930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:29:45.350 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:45.347+0000 7f8ef87f8700 1 -- 172.21.15.98:0/46639798 >> 172.21.15.98:0/46639798 conn(0x7f8ef0075280 msgr2=0x7f8ef0075680 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:29:45.351 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:45.347+0000 7f8ef87f8700 1 -- 172.21.15.98:0/46639798 shutdown_connections 2023-07-14T19:29:45.351 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:45.347+0000 7f8ef87f8700 1 -- 172.21.15.98:0/46639798 wait complete. 2023-07-14T19:29:45.351 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:45.348+0000 7f8ef87f8700 1 Processor -- start 2023-07-14T19:29:45.351 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:45.348+0000 7f8ef87f8700 1 -- start start 2023-07-14T19:29:45.352 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:45.348+0000 7f8ef87f8700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8ef00fe4b0 0x7f8ef0195d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:29:45.353 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:45.348+0000 7f8ef87f8700 1 --2- >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f8ef00fee50 0x7f8ef0196250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:29:45.353 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:45.348+0000 7f8ef87f8700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f8ef0105010 0x7f8ef019a6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:29:45.353 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:45.348+0000 7f8ef87f8700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8ef019acd0 con 0x7f8ef00fe4b0 2023-07-14T19:29:45.353 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:45.348+0000 7f8ef87f8700 1 -- --> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8ef019ae40 con 0x7f8ef00fee50 2023-07-14T19:29:45.353 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:45.348+0000 7f8ef87f8700 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8ef019b000 con 0x7f8ef0105010 2023-07-14T19:29:45.353 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:45.348+0000 7f8ef6594700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8ef00fe4b0 0x7f8ef0195d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:29:45.354 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:45.348+0000 7f8ef6594700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8ef00fe4b0 0x7f8ef0195d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:46274/0 (socket says 172.21.15.98:46274) 2023-07-14T19:29:45.354 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:45.348+0000 7f8ef6594700 1 -- 172.21.15.98:0/3949857965 learned_addr learned my addr 172.21.15.98:0/3949857965 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-07-14T19:29:45.354 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:45.348+0000 7f8ef6d95700 1 --2- 172.21.15.98:0/3949857965 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f8ef0105010 0x7f8ef019a6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:29:45.354 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:45.349+0000 7f8ef5d93700 1 --2- 172.21.15.98:0/3949857965 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f8ef00fee50 0x7f8ef0196250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:29:45.354 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:45.349+0000 7f8ef6594700 1 -- 172.21.15.98:0/3949857965 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f8ef0105010 msgr2=0x7f8ef019a6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:29:45.354 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:45.349+0000 7f8ef6594700 1 --2- 172.21.15.98:0/3949857965 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f8ef0105010 0x7f8ef019a6c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:29:45.354 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:45.349+0000 7f8ef6594700 1 -- 172.21.15.98:0/3949857965 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f8ef00fee50 msgr2=0x7f8ef0196250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:29:45.355 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:45.349+0000 7f8ef6594700 1 --2- 172.21.15.98:0/3949857965 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f8ef00fee50 0x7f8ef0196250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:29:45.355 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:45.349+0000 7f8ef6594700 1 -- 172.21.15.98:0/3949857965 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8ee40095f0 con 0x7f8ef00fe4b0 2023-07-14T19:29:45.355 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:45.349+0000 7f8ef6d95700 1 --2- 172.21.15.98:0/3949857965 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f8ef0105010 0x7f8ef019a6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-07-14T19:29:45.355 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:45.349+0000 7f8ef6594700 1 --2- 172.21.15.98:0/3949857965 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8ef00fe4b0 0x7f8ef0195d10 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7f8ee40098e0 tx=0x7f8ee4004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:29:45.355 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:45.349+0000 7f8ee37fe700 1 -- 172.21.15.98:0/3949857965 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ee401d070 con 0x7f8ef00fe4b0 2023-07-14T19:29:45.355 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:45.349+0000 7f8ee37fe700 1 -- 172.21.15.98:0/3949857965 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f8ee4004160 con 0x7f8ef00fe4b0 2023-07-14T19:29:45.355 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:45.349+0000 7f8ef87f8700 1 -- 172.21.15.98:0/3949857965 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8ef019b280 con 0x7f8ef00fe4b0 2023-07-14T19:29:45.356 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:45.349+0000 7f8ee37fe700 1 -- 172.21.15.98:0/3949857965 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ee400f460 con 0x7f8ef00fe4b0 2023-07-14T19:29:45.356 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:45.349+0000 7f8ef87f8700 1 -- 172.21.15.98:0/3949857965 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8ef01a2b40 con 0x7f8ef00fe4b0 2023-07-14T19:29:45.356 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:45.350+0000 7f8ef87f8700 1 -- 172.21.15.98:0/3949857965 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8ef004e340 con 0x7f8ef00fe4b0 2023-07-14T19:29:45.356 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:45.351+0000 7f8ee37fe700 1 -- 172.21.15.98:0/3949857965 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 17) v1 ==== 135766+0+0 (secure 0 0 0) 0x7f8ee4022570 con 0x7f8ef00fe4b0 2023-07-14T19:29:45.357 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:45.351+0000 7f8ee37fe700 1 --2- 172.21.15.98:0/3949857965 >> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] conn(0x7f8edc0a11e0 0x7f8edc0a3690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:29:45.357 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:45.352+0000 7f8ef5d93700 1 --2- 172.21.15.98:0/3949857965 >> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] conn(0x7f8edc0a11e0 0x7f8edc0a3690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:29:45.357 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:45.352+0000 7f8ee37fe700 1 -- 172.21.15.98:0/3949857965 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 7979+0+0 (secure 0 0 0) 0x7f8ee40cde60 con 0x7f8ef00fe4b0 2023-07-14T19:29:45.357 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:45.352+0000 7f8ef5d93700 1 --2- 172.21.15.98:0/3949857965 >> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] conn(0x7f8edc0a11e0 0x7f8edc0a3690 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f8eec005950 tx=0x7f8eec0058e0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:29:45.357 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:45.356+0000 7f8ee37fe700 1 -- 172.21.15.98:0/3949857965 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f8ee40976a0 con 0x7f8ef00fe4b0 2023-07-14T19:29:45.543 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:45.542+0000 7f8ef87f8700 1 -- 172.21.15.98:0/3949857965 --> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f8ef00fd570 con 0x7f8edc0a11e0 2023-07-14T19:29:45.556 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:45.551+0000 7f8ee37fe700 1 -- 172.21.15.98:0/3949857965 <== mgr.14152 v2:172.21.15.98:6800/452042164 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+341177 (secure 0 0 0) 0x7f8ef00fd570 con 0x7f8edc0a11e0 2023-07-14T19:29:45.560 INFO:teuthology.orchestra.run.smithi098.stdout: 2023-07-14T19:29:45.562 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:45.555+0000 7f8ef87f8700 1 -- 172.21.15.98:0/3949857965 >> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] conn(0x7f8edc0a11e0 msgr2=0x7f8edc0a3690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:29:45.562 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:45.555+0000 7f8ef87f8700 1 --2- 172.21.15.98:0/3949857965 >> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] conn(0x7f8edc0a11e0 0x7f8edc0a3690 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f8eec005950 tx=0x7f8eec0058e0 comp rx=0 tx=0).stop 2023-07-14T19:29:45.562 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:45.555+0000 7f8ef87f8700 1 -- 172.21.15.98:0/3949857965 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8ef00fe4b0 msgr2=0x7f8ef0195d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:29:45.562 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:45.555+0000 7f8ef87f8700 1 --2- 172.21.15.98:0/3949857965 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8ef00fe4b0 0x7f8ef0195d10 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7f8ee40098e0 tx=0x7f8ee4004730 comp rx=0 tx=0).stop 2023-07-14T19:29:45.562 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:45.555+0000 7f8ef87f8700 1 -- 172.21.15.98:0/3949857965 shutdown_connections 2023-07-14T19:29:45.562 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:45.555+0000 7f8ef87f8700 1 --2- 172.21.15.98:0/3949857965 >> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] conn(0x7f8edc0a11e0 0x7f8edc0a3690 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:29:45.563 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:45.555+0000 7f8ef87f8700 1 --2- 172.21.15.98:0/3949857965 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f8ef0105010 0x7f8ef019a6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:29:45.563 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:45.555+0000 7f8ef87f8700 1 --2- 172.21.15.98:0/3949857965 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8ef00fe4b0 0x7f8ef0195d10 unknown :-1 s=CLOSED pgs=290 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:29:45.563 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:45.555+0000 7f8ef87f8700 1 --2- 172.21.15.98:0/3949857965 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f8ef00fee50 0x7f8ef0196250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:29:45.563 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:45.555+0000 7f8ef87f8700 1 -- 172.21.15.98:0/3949857965 >> 172.21.15.98:0/3949857965 conn(0x7f8ef0075280 msgr2=0x7f8ef00fce60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:29:45.563 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:45.555+0000 7f8ef87f8700 1 -- 172.21.15.98:0/3949857965 shutdown_connections 2023-07-14T19:29:45.563 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:45.555+0000 7f8ef87f8700 1 -- 172.21.15.98:0/3949857965 wait complete. 2023-07-14T19:29:45.564 INFO:teuthology.orchestra.run.smithi098.stderr:dumped all 2023-07-14T19:29:45.993 INFO:teuthology.orchestra.run.smithi098.stdout:{"pg_ready":true,"pg_map":{"version":274,"stamp":"2023-07-14T19:29:43.602016+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":649568,"num_objects":64,"num_object_clones":0,"num_object_copies":192,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":64,"num_whiteouts":0,"num_read":367,"num_read_kb":295,"num_write":332,"num_write_kb":3040,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":270,"ondisk_log_size":270,"up":291,"acting":291,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":291,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":336804,"kb_used_data":12612,"kb_used_omap":20,"kb_used_meta":324075,"kb_avail":1124605020,"statfs":{"total":1151940427776,"available":1151595540480,"internally_reserved":0,"allocated":12914688,"data_stored":7089245,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":21372,"internal_metadata":331852932},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":61,"num_read_kb":30,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002888"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":12,"reported_epoch":83,"state":"active+clean","last_fresh":"2023-07-14T19:29:02.573808+0000","last_change":"2023-07-14T19:29:01.313813+0000","last_active":"2023-07-14T19:29:02.573808+0000","last_peered":"2023-07-14T19:29:02.573808+0000","last_clean":"2023-07-14T19:29:02.573808+0000","last_became_active":"2023-07-14T19:29:01.313514+0000","last_became_peered":"2023-07-14T19:29:01.313514+0000","last_unstale":"2023-07-14T19:29:02.573808+0000","last_undegraded":"2023-07-14T19:29:02.573808+0000","last_fullsized":"2023-07-14T19:29:02.573808+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:29:00.604983+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:29:00.604983+0000","last_clean_scrub_stamp":"2023-07-14T19:29:00.604983+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T04:19:18.663334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":12,"reported_epoch":83,"state":"active+clean","last_fresh":"2023-07-14T19:29:02.603793+0000","last_change":"2023-07-14T19:29:01.316899+0000","last_active":"2023-07-14T19:29:02.603793+0000","last_peered":"2023-07-14T19:29:02.603793+0000","last_clean":"2023-07-14T19:29:02.603793+0000","last_became_active":"2023-07-14T19:29:01.316603+0000","last_became_peered":"2023-07-14T19:29:01.316603+0000","last_unstale":"2023-07-14T19:29:02.603793+0000","last_undegraded":"2023-07-14T19:29:02.603793+0000","last_fullsized":"2023-07-14T19:29:02.603793+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:29:00.604983+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:29:00.604983+0000","last_clean_scrub_stamp":"2023-07-14T19:29:00.604983+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T01:32:43.783686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":12,"reported_epoch":83,"state":"active+clean","last_fresh":"2023-07-14T19:29:02.603746+0000","last_change":"2023-07-14T19:29:01.316557+0000","last_active":"2023-07-14T19:29:02.603746+0000","last_peered":"2023-07-14T19:29:02.603746+0000","last_clean":"2023-07-14T19:29:02.603746+0000","last_became_active":"2023-07-14T19:29:01.316079+0000","last_became_peered":"2023-07-14T19:29:01.316079+0000","last_unstale":"2023-07-14T19:29:02.603746+0000","last_undegraded":"2023-07-14T19:29:02.603746+0000","last_fullsized":"2023-07-14T19:29:02.603746+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:29:00.604983+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:29:00.604983+0000","last_clean_scrub_stamp":"2023-07-14T19:29:00.604983+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T05:43:38.465282+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":12,"reported_epoch":83,"state":"active+clean","last_fresh":"2023-07-14T19:29:03.339617+0000","last_change":"2023-07-14T19:29:01.307992+0000","last_active":"2023-07-14T19:29:03.339617+0000","last_peered":"2023-07-14T19:29:03.339617+0000","last_clean":"2023-07-14T19:29:03.339617+0000","last_became_active":"2023-07-14T19:29:01.307687+0000","last_became_peered":"2023-07-14T19:29:01.307687+0000","last_unstale":"2023-07-14T19:29:03.339617+0000","last_undegraded":"2023-07-14T19:29:03.339617+0000","last_fullsized":"2023-07-14T19:29:03.339617+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:29:00.604983+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:29:00.604983+0000","last_clean_scrub_stamp":"2023-07-14T19:29:00.604983+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T03:04:56.480796+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":12,"reported_epoch":83,"state":"active+clean","last_fresh":"2023-07-14T19:29:02.660819+0000","last_change":"2023-07-14T19:29:01.311036+0000","last_active":"2023-07-14T19:29:02.660819+0000","last_peered":"2023-07-14T19:29:02.660819+0000","last_clean":"2023-07-14T19:29:02.660819+0000","last_became_active":"2023-07-14T19:29:01.310723+0000","last_became_peered":"2023-07-14T19:29:01.310723+0000","last_unstale":"2023-07-14T19:29:02.660819+0000","last_undegraded":"2023-07-14T19:29:02.660819+0000","last_fullsized":"2023-07-14T19:29:02.660819+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:29:00.604983+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:29:00.604983+0000","last_clean_scrub_stamp":"2023-07-14T19:29:00.604983+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T23:51:40.842901+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":12,"reported_epoch":83,"state":"active+clean","last_fresh":"2023-07-14T19:29:03.339712+0000","last_change":"2023-07-14T19:29:01.315246+0000","last_active":"2023-07-14T19:29:03.339712+0000","last_peered":"2023-07-14T19:29:03.339712+0000","last_clean":"2023-07-14T19:29:03.339712+0000","last_became_active":"2023-07-14T19:29:01.314885+0000","last_became_peered":"2023-07-14T19:29:01.314885+0000","last_unstale":"2023-07-14T19:29:03.339712+0000","last_undegraded":"2023-07-14T19:29:03.339712+0000","last_fullsized":"2023-07-14T19:29:03.339712+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:29:00.604983+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:29:00.604983+0000","last_clean_scrub_stamp":"2023-07-14T19:29:00.604983+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T04:22:25.924419+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":12,"reported_epoch":83,"state":"active+clean","last_fresh":"2023-07-14T19:29:03.338493+0000","last_change":"2023-07-14T19:29:01.306322+0000","last_active":"2023-07-14T19:29:03.338493+0000","last_peered":"2023-07-14T19:29:03.338493+0000","last_clean":"2023-07-14T19:29:03.338493+0000","last_became_active":"2023-07-14T19:29:01.306024+0000","last_became_peered":"2023-07-14T19:29:01.306024+0000","last_unstale":"2023-07-14T19:29:03.338493+0000","last_undegraded":"2023-07-14T19:29:03.338493+0000","last_fullsized":"2023-07-14T19:29:03.338493+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:29:00.604983+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:29:00.604983+0000","last_clean_scrub_stamp":"2023-07-14T19:29:00.604983+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T04:13:45.893986+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":12,"reported_epoch":83,"state":"active+clean","last_fresh":"2023-07-14T19:29:02.298956+0000","last_change":"2023-07-14T19:29:01.312514+0000","last_active":"2023-07-14T19:29:02.298956+0000","last_peered":"2023-07-14T19:29:02.298956+0000","last_clean":"2023-07-14T19:29:02.298956+0000","last_became_active":"2023-07-14T19:29:01.312260+0000","last_became_peered":"2023-07-14T19:29:01.312260+0000","last_unstale":"2023-07-14T19:29:02.298956+0000","last_undegraded":"2023-07-14T19:29:02.298956+0000","last_fullsized":"2023-07-14T19:29:02.298956+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:29:00.604983+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:29:00.604983+0000","last_clean_scrub_stamp":"2023-07-14T19:29:00.604983+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T21:31:03.757922+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":12,"reported_epoch":83,"state":"active+clean","last_fresh":"2023-07-14T19:29:02.300275+0000","last_change":"2023-07-14T19:29:01.312344+0000","last_active":"2023-07-14T19:29:02.300275+0000","last_peered":"2023-07-14T19:29:02.300275+0000","last_clean":"2023-07-14T19:29:02.300275+0000","last_became_active":"2023-07-14T19:29:01.311960+0000","last_became_peered":"2023-07-14T19:29:01.311960+0000","last_unstale":"2023-07-14T19:29:02.300275+0000","last_undegraded":"2023-07-14T19:29:02.300275+0000","last_fullsized":"2023-07-14T19:29:02.300275+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:29:00.604983+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:29:00.604983+0000","last_clean_scrub_stamp":"2023-07-14T19:29:00.604983+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T23:15:52.484949+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":12,"reported_epoch":83,"state":"active+clean","last_fresh":"2023-07-14T19:29:03.339513+0000","last_change":"2023-07-14T19:29:01.314868+0000","last_active":"2023-07-14T19:29:03.339513+0000","last_peered":"2023-07-14T19:29:03.339513+0000","last_clean":"2023-07-14T19:29:03.339513+0000","last_became_active":"2023-07-14T19:29:01.314617+0000","last_became_peered":"2023-07-14T19:29:01.314617+0000","last_unstale":"2023-07-14T19:29:03.339513+0000","last_undegraded":"2023-07-14T19:29:03.339513+0000","last_fullsized":"2023-07-14T19:29:03.339513+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:29:00.604983+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:29:00.604983+0000","last_clean_scrub_stamp":"2023-07-14T19:29:00.604983+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T20:33:35.726050+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":12,"reported_epoch":83,"state":"active+clean","last_fresh":"2023-07-14T19:29:03.339870+0000","last_change":"2023-07-14T19:29:01.319124+0000","last_active":"2023-07-14T19:29:03.339870+0000","last_peered":"2023-07-14T19:29:03.339870+0000","last_clean":"2023-07-14T19:29:03.339870+0000","last_became_active":"2023-07-14T19:29:01.318392+0000","last_became_peered":"2023-07-14T19:29:01.318392+0000","last_unstale":"2023-07-14T19:29:03.339870+0000","last_undegraded":"2023-07-14T19:29:03.339870+0000","last_fullsized":"2023-07-14T19:29:03.339870+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:29:00.604983+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:29:00.604983+0000","last_clean_scrub_stamp":"2023-07-14T19:29:00.604983+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T06:19:35.297680+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":12,"reported_epoch":83,"state":"active+clean","last_fresh":"2023-07-14T19:29:03.339129+0000","last_change":"2023-07-14T19:29:01.319003+0000","last_active":"2023-07-14T19:29:03.339129+0000","last_peered":"2023-07-14T19:29:03.339129+0000","last_clean":"2023-07-14T19:29:03.339129+0000","last_became_active":"2023-07-14T19:29:01.316550+0000","last_became_peered":"2023-07-14T19:29:01.316550+0000","last_unstale":"2023-07-14T19:29:03.339129+0000","last_undegraded":"2023-07-14T19:29:03.339129+0000","last_fullsized":"2023-07-14T19:29:03.339129+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:29:00.604983+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:29:00.604983+0000","last_clean_scrub_stamp":"2023-07-14T19:29:00.604983+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T22:06:18.517102+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":12,"reported_epoch":83,"state":"active+clean","last_fresh":"2023-07-14T19:29:02.444425+0000","last_change":"2023-07-14T19:29:01.318596+0000","last_active":"2023-07-14T19:29:02.444425+0000","last_peered":"2023-07-14T19:29:02.444425+0000","last_clean":"2023-07-14T19:29:02.444425+0000","last_became_active":"2023-07-14T19:29:01.318278+0000","last_became_peered":"2023-07-14T19:29:01.318278+0000","last_unstale":"2023-07-14T19:29:02.444425+0000","last_undegraded":"2023-07-14T19:29:02.444425+0000","last_fullsized":"2023-07-14T19:29:02.444425+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:29:00.604983+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:29:00.604983+0000","last_clean_scrub_stamp":"2023-07-14T19:29:00.604983+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T06:00:40.438416+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":12,"reported_epoch":83,"state":"active+clean","last_fresh":"2023-07-14T19:29:02.603553+0000","last_change":"2023-07-14T19:29:01.314344+0000","last_active":"2023-07-14T19:29:02.603553+0000","last_peered":"2023-07-14T19:29:02.603553+0000","last_clean":"2023-07-14T19:29:02.603553+0000","last_became_active":"2023-07-14T19:29:01.314054+0000","last_became_peered":"2023-07-14T19:29:01.314054+0000","last_unstale":"2023-07-14T19:29:02.603553+0000","last_undegraded":"2023-07-14T19:29:02.603553+0000","last_fullsized":"2023-07-14T19:29:02.603553+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:29:00.604983+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:29:00.604983+0000","last_clean_scrub_stamp":"2023-07-14T19:29:00.604983+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T23:03:20.122826+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":12,"reported_epoch":83,"state":"active+clean","last_fresh":"2023-07-14T19:29:02.603428+0000","last_change":"2023-07-14T19:29:01.314703+0000","last_active":"2023-07-14T19:29:02.603428+0000","last_peered":"2023-07-14T19:29:02.603428+0000","last_clean":"2023-07-14T19:29:02.603428+0000","last_became_active":"2023-07-14T19:29:01.314425+0000","last_became_peered":"2023-07-14T19:29:01.314425+0000","last_unstale":"2023-07-14T19:29:02.603428+0000","last_undegraded":"2023-07-14T19:29:02.603428+0000","last_fullsized":"2023-07-14T19:29:02.603428+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:29:00.604983+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:29:00.604983+0000","last_clean_scrub_stamp":"2023-07-14T19:29:00.604983+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T02:24:08.467559+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":12,"reported_epoch":83,"state":"active+clean","last_fresh":"2023-07-14T19:29:02.299763+0000","last_change":"2023-07-14T19:29:01.308769+0000","last_active":"2023-07-14T19:29:02.299763+0000","last_peered":"2023-07-14T19:29:02.299763+0000","last_clean":"2023-07-14T19:29:02.299763+0000","last_became_active":"2023-07-14T19:29:01.308450+0000","last_became_peered":"2023-07-14T19:29:01.308450+0000","last_unstale":"2023-07-14T19:29:02.299763+0000","last_undegraded":"2023-07-14T19:29:02.299763+0000","last_fullsized":"2023-07-14T19:29:02.299763+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:29:00.604983+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:29:00.604983+0000","last_clean_scrub_stamp":"2023-07-14T19:29:00.604983+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T06:36:10.772949+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":12,"reported_epoch":83,"state":"active+clean","last_fresh":"2023-07-14T19:29:02.573896+0000","last_change":"2023-07-14T19:29:01.313928+0000","last_active":"2023-07-14T19:29:02.573896+0000","last_peered":"2023-07-14T19:29:02.573896+0000","last_clean":"2023-07-14T19:29:02.573896+0000","last_became_active":"2023-07-14T19:29:01.313528+0000","last_became_peered":"2023-07-14T19:29:01.313528+0000","last_unstale":"2023-07-14T19:29:02.573896+0000","last_undegraded":"2023-07-14T19:29:02.573896+0000","last_fullsized":"2023-07-14T19:29:02.573896+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:29:00.604983+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:29:00.604983+0000","last_clean_scrub_stamp":"2023-07-14T19:29:00.604983+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T20:08:14.603380+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":12,"reported_epoch":83,"state":"active+clean","last_fresh":"2023-07-14T19:29:02.864370+0000","last_change":"2023-07-14T19:29:01.318903+0000","last_active":"2023-07-14T19:29:02.864370+0000","last_peered":"2023-07-14T19:29:02.864370+0000","last_clean":"2023-07-14T19:29:02.864370+0000","last_became_active":"2023-07-14T19:29:01.318610+0000","last_became_peered":"2023-07-14T19:29:01.318610+0000","last_unstale":"2023-07-14T19:29:02.864370+0000","last_undegraded":"2023-07-14T19:29:02.864370+0000","last_fullsized":"2023-07-14T19:29:02.864370+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:29:00.604983+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:29:00.604983+0000","last_clean_scrub_stamp":"2023-07-14T19:29:00.604983+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T20:01:45.469683+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":12,"reported_epoch":83,"state":"active+clean","last_fresh":"2023-07-14T19:29:03.222037+0000","last_change":"2023-07-14T19:29:01.315405+0000","last_active":"2023-07-14T19:29:03.222037+0000","last_peered":"2023-07-14T19:29:03.222037+0000","last_clean":"2023-07-14T19:29:03.222037+0000","last_became_active":"2023-07-14T19:29:01.315156+0000","last_became_peered":"2023-07-14T19:29:01.315156+0000","last_unstale":"2023-07-14T19:29:03.222037+0000","last_undegraded":"2023-07-14T19:29:03.222037+0000","last_fullsized":"2023-07-14T19:29:03.222037+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:29:00.604983+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:29:00.604983+0000","last_clean_scrub_stamp":"2023-07-14T19:29:00.604983+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T02:34:33.971469+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"83'9","reported_seq":24,"reported_epoch":83,"state":"active+clean","last_fresh":"2023-07-14T19:29:05.679207+0000","last_change":"2023-07-14T19:29:00.618385+0000","last_active":"2023-07-14T19:29:05.679207+0000","last_peered":"2023-07-14T19:29:05.679207+0000","last_clean":"2023-07-14T19:29:05.679207+0000","last_became_active":"2023-07-14T19:29:00.618081+0000","last_became_peered":"2023-07-14T19:29:00.618081+0000","last_unstale":"2023-07-14T19:29:05.679207+0000","last_undegraded":"2023-07-14T19:29:05.679207+0000","last_fullsized":"2023-07-14T19:29:05.679207+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:28:59.600128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:28:59.600128+0000","last_clean_scrub_stamp":"2023-07-14T19:28:59.600128+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T06:55:30.739187+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":12,"reported_epoch":83,"state":"active+clean","last_fresh":"2023-07-14T19:29:02.299334+0000","last_change":"2023-07-14T19:29:01.305197+0000","last_active":"2023-07-14T19:29:02.299334+0000","last_peered":"2023-07-14T19:29:02.299334+0000","last_clean":"2023-07-14T19:29:02.299334+0000","last_became_active":"2023-07-14T19:29:01.304902+0000","last_became_peered":"2023-07-14T19:29:01.304902+0000","last_unstale":"2023-07-14T19:29:02.299334+0000","last_undegraded":"2023-07-14T19:29:02.299334+0000","last_fullsized":"2023-07-14T19:29:02.299334+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:29:00.604983+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:29:00.604983+0000","last_clean_scrub_stamp":"2023-07-14T19:29:00.604983+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T20:54:29.407321+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1e","version":"83'2","reported_seq":16,"reported_epoch":83,"state":"active+clean","last_fresh":"2023-07-14T19:29:04.400749+0000","last_change":"2023-07-14T19:29:00.626316+0000","last_active":"2023-07-14T19:29:04.400749+0000","last_peered":"2023-07-14T19:29:04.400749+0000","last_clean":"2023-07-14T19:29:04.400749+0000","last_became_active":"2023-07-14T19:29:00.626035+0000","last_became_peered":"2023-07-14T19:29:00.626035+0000","last_unstale":"2023-07-14T19:29:04.400749+0000","last_undegraded":"2023-07-14T19:29:04.400749+0000","last_fullsized":"2023-07-14T19:29:04.400749+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:28:59.600128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:28:59.600128+0000","last_clean_scrub_stamp":"2023-07-14T19:28:59.600128+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T23:59:08.083433+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":12,"reported_epoch":83,"state":"active+clean","last_fresh":"2023-07-14T19:29:02.603883+0000","last_change":"2023-07-14T19:29:01.317118+0000","last_active":"2023-07-14T19:29:02.603883+0000","last_peered":"2023-07-14T19:29:02.603883+0000","last_clean":"2023-07-14T19:29:02.603883+0000","last_became_active":"2023-07-14T19:29:01.316805+0000","last_became_peered":"2023-07-14T19:29:01.316805+0000","last_unstale":"2023-07-14T19:29:02.603883+0000","last_undegraded":"2023-07-14T19:29:02.603883+0000","last_fullsized":"2023-07-14T19:29:02.603883+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:29:00.604983+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:29:00.604983+0000","last_clean_scrub_stamp":"2023-07-14T19:29:00.604983+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T04:47:32.634271+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1d","version":"83'4","reported_seq":18,"reported_epoch":83,"state":"active+clean","last_fresh":"2023-07-14T19:29:05.679368+0000","last_change":"2023-07-14T19:29:00.619435+0000","last_active":"2023-07-14T19:29:05.679368+0000","last_peered":"2023-07-14T19:29:05.679368+0000","last_clean":"2023-07-14T19:29:05.679368+0000","last_became_active":"2023-07-14T19:29:00.619131+0000","last_became_peered":"2023-07-14T19:29:00.619131+0000","last_unstale":"2023-07-14T19:29:05.679368+0000","last_undegraded":"2023-07-14T19:29:05.679368+0000","last_fullsized":"2023-07-14T19:29:05.679368+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:28:59.600128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:28:59.600128+0000","last_clean_scrub_stamp":"2023-07-14T19:28:59.600128+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T04:51:10.432113+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":12,"reported_epoch":83,"state":"active+clean","last_fresh":"2023-07-14T19:29:02.299884+0000","last_change":"2023-07-14T19:29:01.317947+0000","last_active":"2023-07-14T19:29:02.299884+0000","last_peered":"2023-07-14T19:29:02.299884+0000","last_clean":"2023-07-14T19:29:02.299884+0000","last_became_active":"2023-07-14T19:29:01.317633+0000","last_became_peered":"2023-07-14T19:29:01.317633+0000","last_unstale":"2023-07-14T19:29:02.299884+0000","last_undegraded":"2023-07-14T19:29:02.299884+0000","last_fullsized":"2023-07-14T19:29:02.299884+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:29:00.604983+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:29:00.604983+0000","last_clean_scrub_stamp":"2023-07-14T19:29:00.604983+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T00:25:43.200161+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1c","version":"82'2","reported_seq":16,"reported_epoch":83,"state":"active+clean","last_fresh":"2023-07-14T19:29:02.661063+0000","last_change":"2023-07-14T19:29:00.620401+0000","last_active":"2023-07-14T19:29:02.661063+0000","last_peered":"2023-07-14T19:29:02.661063+0000","last_clean":"2023-07-14T19:29:02.661063+0000","last_became_active":"2023-07-14T19:29:00.620134+0000","last_became_peered":"2023-07-14T19:29:00.620134+0000","last_unstale":"2023-07-14T19:29:02.661063+0000","last_undegraded":"2023-07-14T19:29:02.661063+0000","last_fullsized":"2023-07-14T19:29:02.661063+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:28:59.600128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:28:59.600128+0000","last_clean_scrub_stamp":"2023-07-14T19:28:59.600128+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T06:39:10.382383+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":12,"reported_epoch":83,"state":"active+clean","last_fresh":"2023-07-14T19:29:02.299819+0000","last_change":"2023-07-14T19:29:01.312174+0000","last_active":"2023-07-14T19:29:02.299819+0000","last_peered":"2023-07-14T19:29:02.299819+0000","last_clean":"2023-07-14T19:29:02.299819+0000","last_became_active":"2023-07-14T19:29:01.311891+0000","last_became_peered":"2023-07-14T19:29:01.311891+0000","last_unstale":"2023-07-14T19:29:02.299819+0000","last_undegraded":"2023-07-14T19:29:02.299819+0000","last_fullsized":"2023-07-14T19:29:02.299819+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:29:00.604983+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:29:00.604983+0000","last_clean_scrub_stamp":"2023-07-14T19:29:00.604983+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T19:35:58.889437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1b","version":"83'6","reported_seq":59,"reported_epoch":83,"state":"active+clean","last_fresh":"2023-07-14T19:29:40.358777+0000","last_change":"2023-07-14T19:29:00.622257+0000","last_active":"2023-07-14T19:29:40.358777+0000","last_peered":"2023-07-14T19:29:40.358777+0000","last_clean":"2023-07-14T19:29:40.358777+0000","last_became_active":"2023-07-14T19:29:00.621984+0000","last_became_peered":"2023-07-14T19:29:00.621984+0000","last_unstale":"2023-07-14T19:29:40.358777+0000","last_undegraded":"2023-07-14T19:29:40.358777+0000","last_fullsized":"2023-07-14T19:29:40.358777+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:28:59.600128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:28:59.600128+0000","last_clean_scrub_stamp":"2023-07-14T19:28:59.600128+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T01:01:52.888462+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":964,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":39,"num_read_kb":3,"num_write":3,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":12,"reported_epoch":83,"state":"active+clean","last_fresh":"2023-07-14T19:29:02.660873+0000","last_change":"2023-07-14T19:29:01.306274+0000","last_active":"2023-07-14T19:29:02.660873+0000","last_peered":"2023-07-14T19:29:02.660873+0000","last_clean":"2023-07-14T19:29:02.660873+0000","last_became_active":"2023-07-14T19:29:01.305863+0000","last_became_peered":"2023-07-14T19:29:01.305863+0000","last_unstale":"2023-07-14T19:29:02.660873+0000","last_undegraded":"2023-07-14T19:29:02.660873+0000","last_fullsized":"2023-07-14T19:29:02.660873+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:29:00.604983+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:29:00.604983+0000","last_clean_scrub_stamp":"2023-07-14T19:29:00.604983+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T00:42:02.531260+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.1a","version":"83'3","reported_seq":17,"reported_epoch":83,"state":"active+clean","last_fresh":"2023-07-14T19:29:05.678594+0000","last_change":"2023-07-14T19:29:00.619509+0000","last_active":"2023-07-14T19:29:05.678594+0000","last_peered":"2023-07-14T19:29:05.678594+0000","last_clean":"2023-07-14T19:29:05.678594+0000","last_became_active":"2023-07-14T19:29:00.619193+0000","last_became_peered":"2023-07-14T19:29:00.619193+0000","last_unstale":"2023-07-14T19:29:05.678594+0000","last_undegraded":"2023-07-14T19:29:05.678594+0000","last_fullsized":"2023-07-14T19:29:05.678594+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:28:59.600128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:28:59.600128+0000","last_clean_scrub_stamp":"2023-07-14T19:28:59.600128+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T01:07:45.846978+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":12,"reported_epoch":83,"state":"active+clean","last_fresh":"2023-07-14T19:29:02.299714+0000","last_change":"2023-07-14T19:29:01.307535+0000","last_active":"2023-07-14T19:29:02.299714+0000","last_peered":"2023-07-14T19:29:02.299714+0000","last_clean":"2023-07-14T19:29:02.299714+0000","last_became_active":"2023-07-14T19:29:01.307209+0000","last_became_peered":"2023-07-14T19:29:01.307209+0000","last_unstale":"2023-07-14T19:29:02.299714+0000","last_undegraded":"2023-07-14T19:29:02.299714+0000","last_fullsized":"2023-07-14T19:29:02.299714+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:29:00.604983+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:29:00.604983+0000","last_clean_scrub_stamp":"2023-07-14T19:29:00.604983+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T03:49:13.610038+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.19","version":"83'2","reported_seq":16,"reported_epoch":83,"state":"active+clean","last_fresh":"2023-07-14T19:29:05.678992+0000","last_change":"2023-07-14T19:29:00.620671+0000","last_active":"2023-07-14T19:29:05.678992+0000","last_peered":"2023-07-14T19:29:05.678992+0000","last_clean":"2023-07-14T19:29:05.678992+0000","last_became_active":"2023-07-14T19:29:00.619786+0000","last_became_peered":"2023-07-14T19:29:00.619786+0000","last_unstale":"2023-07-14T19:29:05.678992+0000","last_undegraded":"2023-07-14T19:29:05.678992+0000","last_fullsized":"2023-07-14T19:29:05.678992+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:28:59.600128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:28:59.600128+0000","last_clean_scrub_stamp":"2023-07-14T19:28:59.600128+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T06:56:01.025242+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":12,"reported_epoch":83,"state":"active+clean","last_fresh":"2023-07-14T19:29:02.298842+0000","last_change":"2023-07-14T19:29:01.316248+0000","last_active":"2023-07-14T19:29:02.298842+0000","last_peered":"2023-07-14T19:29:02.298842+0000","last_clean":"2023-07-14T19:29:02.298842+0000","last_became_active":"2023-07-14T19:29:01.315974+0000","last_became_peered":"2023-07-14T19:29:01.315974+0000","last_unstale":"2023-07-14T19:29:02.298842+0000","last_undegraded":"2023-07-14T19:29:02.298842+0000","last_fullsized":"2023-07-14T19:29:02.298842+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:29:00.604983+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:29:00.604983+0000","last_clean_scrub_stamp":"2023-07-14T19:29:00.604983+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T01:48:48.147093+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"83'1","reported_seq":15,"reported_epoch":83,"state":"active+clean","last_fresh":"2023-07-14T19:29:02.299408+0000","last_change":"2023-07-14T19:29:00.624210+0000","last_active":"2023-07-14T19:29:02.299408+0000","last_peered":"2023-07-14T19:29:02.299408+0000","last_clean":"2023-07-14T19:29:02.299408+0000","last_became_active":"2023-07-14T19:29:00.623918+0000","last_became_peered":"2023-07-14T19:29:00.623918+0000","last_unstale":"2023-07-14T19:29:02.299408+0000","last_undegraded":"2023-07-14T19:29:02.299408+0000","last_fullsized":"2023-07-14T19:29:02.299408+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:28:59.600128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:28:59.600128+0000","last_clean_scrub_stamp":"2023-07-14T19:28:59.600128+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T01:38:53.165557+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":12,"reported_epoch":83,"state":"active+clean","last_fresh":"2023-07-14T19:29:02.864241+0000","last_change":"2023-07-14T19:29:01.313787+0000","last_active":"2023-07-14T19:29:02.864241+0000","last_peered":"2023-07-14T19:29:02.864241+0000","last_clean":"2023-07-14T19:29:02.864241+0000","last_became_active":"2023-07-14T19:29:01.313479+0000","last_became_peered":"2023-07-14T19:29:01.313479+0000","last_unstale":"2023-07-14T19:29:02.864241+0000","last_undegraded":"2023-07-14T19:29:02.864241+0000","last_fullsized":"2023-07-14T19:29:02.864241+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:29:00.604983+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:29:00.604983+0000","last_clean_scrub_stamp":"2023-07-14T19:29:00.604983+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T22:33:21.348977+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.17","version":"83'3","reported_seq":50,"reported_epoch":83,"state":"active+clean","last_fresh":"2023-07-14T19:29:37.353340+0000","last_change":"2023-07-14T19:29:00.623053+0000","last_active":"2023-07-14T19:29:37.353340+0000","last_peered":"2023-07-14T19:29:37.353340+0000","last_clean":"2023-07-14T19:29:37.353340+0000","last_became_active":"2023-07-14T19:29:00.622760+0000","last_became_peered":"2023-07-14T19:29:00.622760+0000","last_unstale":"2023-07-14T19:29:37.353340+0000","last_undegraded":"2023-07-14T19:29:37.353340+0000","last_fullsized":"2023-07-14T19:29:37.353340+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:28:59.600128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:28:59.600128+0000","last_clean_scrub_stamp":"2023-07-14T19:28:59.600128+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T21:15:50.464654+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":33,"num_read_kb":33,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":12,"reported_epoch":83,"state":"active+clean","last_fresh":"2023-07-14T19:29:02.299982+0000","last_change":"2023-07-14T19:29:01.317509+0000","last_active":"2023-07-14T19:29:02.299982+0000","last_peered":"2023-07-14T19:29:02.299982+0000","last_clean":"2023-07-14T19:29:02.299982+0000","last_became_active":"2023-07-14T19:29:01.317238+0000","last_became_peered":"2023-07-14T19:29:01.317238+0000","last_unstale":"2023-07-14T19:29:02.299982+0000","last_undegraded":"2023-07-14T19:29:02.299982+0000","last_fullsized":"2023-07-14T19:29:02.299982+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:29:00.604983+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:29:00.604983+0000","last_clean_scrub_stamp":"2023-07-14T19:29:00.604983+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T04:59:01.463446+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.16","version":"83'2","reported_seq":16,"reported_epoch":83,"state":"active+clean","last_fresh":"2023-07-14T19:29:04.671803+0000","last_change":"2023-07-14T19:29:00.624024+0000","last_active":"2023-07-14T19:29:04.671803+0000","last_peered":"2023-07-14T19:29:04.671803+0000","last_clean":"2023-07-14T19:29:04.671803+0000","last_became_active":"2023-07-14T19:29:00.623638+0000","last_became_peered":"2023-07-14T19:29:00.623638+0000","last_unstale":"2023-07-14T19:29:04.671803+0000","last_undegraded":"2023-07-14T19:29:04.671803+0000","last_fullsized":"2023-07-14T19:29:04.671803+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:28:59.600128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:28:59.600128+0000","last_clean_scrub_stamp":"2023-07-14T19:28:59.600128+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T03:50:19.311702+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":12,"reported_epoch":83,"state":"active+clean","last_fresh":"2023-07-14T19:29:02.603804+0000","last_change":"2023-07-14T19:29:01.309164+0000","last_active":"2023-07-14T19:29:02.603804+0000","last_peered":"2023-07-14T19:29:02.603804+0000","last_clean":"2023-07-14T19:29:02.603804+0000","last_became_active":"2023-07-14T19:29:01.308797+0000","last_became_peered":"2023-07-14T19:29:01.308797+0000","last_unstale":"2023-07-14T19:29:02.603804+0000","last_undegraded":"2023-07-14T19:29:02.603804+0000","last_fullsized":"2023-07-14T19:29:02.603804+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:29:00.604983+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:29:00.604983+0000","last_clean_scrub_stamp":"2023-07-14T19:29:00.604983+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T23:09:54.074320+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"83'4","reported_seq":19,"reported_epoch":83,"state":"active+clean","last_fresh":"2023-07-14T19:29:04.671757+0000","last_change":"2023-07-14T19:29:00.625435+0000","last_active":"2023-07-14T19:29:04.671757+0000","last_peered":"2023-07-14T19:29:04.671757+0000","last_clean":"2023-07-14T19:29:04.671757+0000","last_became_active":"2023-07-14T19:29:00.625092+0000","last_became_peered":"2023-07-14T19:29:00.625092+0000","last_unstale":"2023-07-14T19:29:04.671757+0000","last_undegraded":"2023-07-14T19:29:04.671757+0000","last_fullsized":"2023-07-14T19:29:04.671757+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:28:59.600128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:28:59.600128+0000","last_clean_scrub_stamp":"2023-07-14T19:28:59.600128+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T02:44:25.187321+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":12,"reported_epoch":83,"state":"active+clean","last_fresh":"2023-07-14T19:29:02.300025+0000","last_change":"2023-07-14T19:29:01.315672+0000","last_active":"2023-07-14T19:29:02.300025+0000","last_peered":"2023-07-14T19:29:02.300025+0000","last_clean":"2023-07-14T19:29:02.300025+0000","last_became_active":"2023-07-14T19:29:01.315331+0000","last_became_peered":"2023-07-14T19:29:01.315331+0000","last_unstale":"2023-07-14T19:29:02.300025+0000","last_undegraded":"2023-07-14T19:29:02.300025+0000","last_fullsized":"2023-07-14T19:29:02.300025+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:29:00.604983+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:29:00.604983+0000","last_clean_scrub_stamp":"2023-07-14T19:29:00.604983+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T02:46:22.959698+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.14","version":"83'26","reported_seq":113,"reported_epoch":83,"state":"active+clean","last_fresh":"2023-07-14T19:29:38.354734+0000","last_change":"2023-07-14T19:29:00.622910+0000","last_active":"2023-07-14T19:29:38.354734+0000","last_peered":"2023-07-14T19:29:38.354734+0000","last_clean":"2023-07-14T19:29:38.354734+0000","last_became_active":"2023-07-14T19:29:00.622582+0000","last_became_peered":"2023-07-14T19:29:00.622582+0000","last_unstale":"2023-07-14T19:29:38.354734+0000","last_undegraded":"2023-07-14T19:29:38.354734+0000","last_fullsized":"2023-07-14T19:29:38.354734+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:28:59.600128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:28:59.600128+0000","last_clean_scrub_stamp":"2023-07-14T19:28:59.600128+0000","objects_scrubbed":0,"log_size":26,"log_dups_size":0,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T20:35:52.757984+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":54660,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":73,"num_read_kb":89,"num_write":27,"num_write_kb":62,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":12,"reported_epoch":83,"state":"active+clean","last_fresh":"2023-07-14T19:29:02.660955+0000","last_change":"2023-07-14T19:29:01.306169+0000","last_active":"2023-07-14T19:29:02.660955+0000","last_peered":"2023-07-14T19:29:02.660955+0000","last_clean":"2023-07-14T19:29:02.660955+0000","last_became_active":"2023-07-14T19:29:01.305830+0000","last_became_peered":"2023-07-14T19:29:01.305830+0000","last_unstale":"2023-07-14T19:29:02.660955+0000","last_undegraded":"2023-07-14T19:29:02.660955+0000","last_fullsized":"2023-07-14T19:29:02.660955+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:29:00.604983+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:29:00.604983+0000","last_clean_scrub_stamp":"2023-07-14T19:29:00.604983+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T00:12:18.670125+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.13","version":"83'6","reported_seq":21,"reported_epoch":83,"state":"active+clean","last_fresh":"2023-07-14T19:29:05.679614+0000","last_change":"2023-07-14T19:29:00.626500+0000","last_active":"2023-07-14T19:29:05.679614+0000","last_peered":"2023-07-14T19:29:05.679614+0000","last_clean":"2023-07-14T19:29:05.679614+0000","last_became_active":"2023-07-14T19:29:00.626130+0000","last_became_peered":"2023-07-14T19:29:00.626130+0000","last_unstale":"2023-07-14T19:29:05.679614+0000","last_undegraded":"2023-07-14T19:29:05.679614+0000","last_fullsized":"2023-07-14T19:29:05.679614+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:28:59.600128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:28:59.600128+0000","last_clean_scrub_stamp":"2023-07-14T19:28:59.600128+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T01:08:40.766374+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":12,"reported_epoch":83,"state":"active+clean","last_fresh":"2023-07-14T19:29:02.444400+0000","last_change":"2023-07-14T19:29:01.317902+0000","last_active":"2023-07-14T19:29:02.444400+0000","last_peered":"2023-07-14T19:29:02.444400+0000","last_clean":"2023-07-14T19:29:02.444400+0000","last_became_active":"2023-07-14T19:29:01.317587+0000","last_became_peered":"2023-07-14T19:29:01.317587+0000","last_unstale":"2023-07-14T19:29:02.444400+0000","last_undegraded":"2023-07-14T19:29:02.444400+0000","last_fullsized":"2023-07-14T19:29:02.444400+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:29:00.604983+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:29:00.604983+0000","last_clean_scrub_stamp":"2023-07-14T19:29:00.604983+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T03:39:39.441094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.12","version":"83'3","reported_seq":17,"reported_epoch":83,"state":"active+clean","last_fresh":"2023-07-14T19:29:04.671869+0000","last_change":"2023-07-14T19:29:00.626345+0000","last_active":"2023-07-14T19:29:04.671869+0000","last_peered":"2023-07-14T19:29:04.671869+0000","last_clean":"2023-07-14T19:29:04.671869+0000","last_became_active":"2023-07-14T19:29:00.626059+0000","last_became_peered":"2023-07-14T19:29:00.626059+0000","last_unstale":"2023-07-14T19:29:04.671869+0000","last_undegraded":"2023-07-14T19:29:04.671869+0000","last_fullsized":"2023-07-14T19:29:04.671869+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:28:59.600128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:28:59.600128+0000","last_clean_scrub_stamp":"2023-07-14T19:28:59.600128+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T21:18:49.525099+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":12,"reported_epoch":83,"state":"active+clean","last_fresh":"2023-07-14T19:29:03.339214+0000","last_change":"2023-07-14T19:29:01.310518+0000","last_active":"2023-07-14T19:29:03.339214+0000","last_peered":"2023-07-14T19:29:03.339214+0000","last_clean":"2023-07-14T19:29:03.339214+0000","last_became_active":"2023-07-14T19:29:01.310213+0000","last_became_peered":"2023-07-14T19:29:01.310213+0000","last_unstale":"2023-07-14T19:29:03.339214+0000","last_undegraded":"2023-07-14T19:29:03.339214+0000","last_fullsized":"2023-07-14T19:29:03.339214+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:29:00.604983+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:29:00.604983+0000","last_clean_scrub_stamp":"2023-07-14T19:29:00.604983+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T01:11:48.079392+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"83'4","reported_seq":56,"reported_epoch":83,"state":"active+clean","last_fresh":"2023-07-14T19:29:39.356261+0000","last_change":"2023-07-14T19:29:00.620450+0000","last_active":"2023-07-14T19:29:39.356261+0000","last_peered":"2023-07-14T19:29:39.356261+0000","last_clean":"2023-07-14T19:29:39.356261+0000","last_became_active":"2023-07-14T19:29:00.620100+0000","last_became_peered":"2023-07-14T19:29:00.620100+0000","last_unstale":"2023-07-14T19:29:39.356261+0000","last_undegraded":"2023-07-14T19:29:39.356261+0000","last_fullsized":"2023-07-14T19:29:39.356261+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:28:59.600128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:28:59.600128+0000","last_clean_scrub_stamp":"2023-07-14T19:28:59.600128+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T23:01:06.885919+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":930,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":39,"num_read_kb":4,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":12,"reported_epoch":83,"state":"active+clean","last_fresh":"2023-07-14T19:29:03.338482+0000","last_change":"2023-07-14T19:29:01.320612+0000","last_active":"2023-07-14T19:29:03.338482+0000","last_peered":"2023-07-14T19:29:03.338482+0000","last_clean":"2023-07-14T19:29:03.338482+0000","last_became_active":"2023-07-14T19:29:01.319569+0000","last_became_peered":"2023-07-14T19:29:01.319569+0000","last_unstale":"2023-07-14T19:29:03.338482+0000","last_undegraded":"2023-07-14T19:29:03.338482+0000","last_fullsized":"2023-07-14T19:29:03.338482+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:29:00.604983+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:29:00.604983+0000","last_clean_scrub_stamp":"2023-07-14T19:29:00.604983+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T05:46:45.094405+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.10","version":"83'15","reported_seq":30,"reported_epoch":83,"state":"active+clean","last_fresh":"2023-07-14T19:29:05.680022+0000","last_change":"2023-07-14T19:29:00.618619+0000","last_active":"2023-07-14T19:29:05.680022+0000","last_peered":"2023-07-14T19:29:05.680022+0000","last_clean":"2023-07-14T19:29:05.680022+0000","last_became_active":"2023-07-14T19:29:00.617735+0000","last_became_peered":"2023-07-14T19:29:00.617735+0000","last_unstale":"2023-07-14T19:29:05.680022+0000","last_undegraded":"2023-07-14T19:29:05.680022+0000","last_fullsized":"2023-07-14T19:29:05.680022+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:28:59.600128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:28:59.600128+0000","last_clean_scrub_stamp":"2023-07-14T19:28:59.600128+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T22:03:33.860671+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":12,"reported_epoch":83,"state":"active+clean","last_fresh":"2023-07-14T19:29:02.299414+0000","last_change":"2023-07-14T19:29:01.303053+0000","last_active":"2023-07-14T19:29:02.299414+0000","last_peered":"2023-07-14T19:29:02.299414+0000","last_clean":"2023-07-14T19:29:02.299414+0000","last_became_active":"2023-07-14T19:29:01.302679+0000","last_became_peered":"2023-07-14T19:29:01.302679+0000","last_unstale":"2023-07-14T19:29:02.299414+0000","last_undegraded":"2023-07-14T19:29:02.299414+0000","last_fullsized":"2023-07-14T19:29:02.299414+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:29:00.604983+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:29:00.604983+0000","last_clean_scrub_stamp":"2023-07-14T19:29:00.604983+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T03:30:02.623040+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"83'2","reported_seq":17,"reported_epoch":83,"state":"active+clean","last_fresh":"2023-07-14T19:29:06.299906+0000","last_change":"2023-07-14T19:29:00.622494+0000","last_active":"2023-07-14T19:29:06.299906+0000","last_peered":"2023-07-14T19:29:06.299906+0000","last_clean":"2023-07-14T19:29:06.299906+0000","last_became_active":"2023-07-14T19:29:00.622195+0000","last_became_peered":"2023-07-14T19:29:00.622195+0000","last_unstale":"2023-07-14T19:29:06.299906+0000","last_undegraded":"2023-07-14T19:29:06.299906+0000","last_fullsized":"2023-07-14T19:29:06.299906+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:28:59.600128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:28:59.600128+0000","last_clean_scrub_stamp":"2023-07-14T19:28:59.600128+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T01:34:47.084212+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":12,"reported_epoch":83,"state":"active+clean","last_fresh":"2023-07-14T19:29:02.573720+0000","last_change":"2023-07-14T19:29:01.319375+0000","last_active":"2023-07-14T19:29:02.573720+0000","last_peered":"2023-07-14T19:29:02.573720+0000","last_clean":"2023-07-14T19:29:02.573720+0000","last_became_active":"2023-07-14T19:29:01.319068+0000","last_became_peered":"2023-07-14T19:29:01.319068+0000","last_unstale":"2023-07-14T19:29:02.573720+0000","last_undegraded":"2023-07-14T19:29:02.573720+0000","last_fullsized":"2023-07-14T19:29:02.573720+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:29:00.604983+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:29:00.604983+0000","last_clean_scrub_stamp":"2023-07-14T19:29:00.604983+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T06:31:40.249974+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":14,"reported_epoch":83,"state":"active+clean","last_fresh":"2023-07-14T19:29:02.864257+0000","last_change":"2023-07-14T19:29:00.622364+0000","last_active":"2023-07-14T19:29:02.864257+0000","last_peered":"2023-07-14T19:29:02.864257+0000","last_clean":"2023-07-14T19:29:02.864257+0000","last_became_active":"2023-07-14T19:29:00.622073+0000","last_became_peered":"2023-07-14T19:29:00.622073+0000","last_unstale":"2023-07-14T19:29:02.864257+0000","last_undegraded":"2023-07-14T19:29:02.864257+0000","last_fullsized":"2023-07-14T19:29:02.864257+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:28:59.600128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:28:59.600128+0000","last_clean_scrub_stamp":"2023-07-14T19:28:59.600128+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T20:41:09.339550+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"83'1","reported_seq":13,"reported_epoch":83,"state":"active+clean","last_fresh":"2023-07-14T19:29:03.934768+0000","last_change":"2023-07-14T19:29:01.314076+0000","last_active":"2023-07-14T19:29:03.934768+0000","last_peered":"2023-07-14T19:29:03.934768+0000","last_clean":"2023-07-14T19:29:03.934768+0000","last_became_active":"2023-07-14T19:29:01.313769+0000","last_became_peered":"2023-07-14T19:29:01.313769+0000","last_unstale":"2023-07-14T19:29:03.934768+0000","last_undegraded":"2023-07-14T19:29:03.934768+0000","last_fullsized":"2023-07-14T19:29:03.934768+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:29:00.604983+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:29:00.604983+0000","last_clean_scrub_stamp":"2023-07-14T19:29:00.604983+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T02:26:15.354399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":123,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.2","version":"83'1","reported_seq":15,"reported_epoch":83,"state":"active+clean","last_fresh":"2023-07-14T19:29:03.338943+0000","last_change":"2023-07-14T19:29:00.622928+0000","last_active":"2023-07-14T19:29:03.338943+0000","last_peered":"2023-07-14T19:29:03.338943+0000","last_clean":"2023-07-14T19:29:03.338943+0000","last_became_active":"2023-07-14T19:29:00.622628+0000","last_became_peered":"2023-07-14T19:29:00.622628+0000","last_unstale":"2023-07-14T19:29:03.338943+0000","last_undegraded":"2023-07-14T19:29:03.338943+0000","last_fullsized":"2023-07-14T19:29:03.338943+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:28:59.600128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:28:59.600128+0000","last_clean_scrub_stamp":"2023-07-14T19:28:59.600128+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T21:41:30.895447+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":12,"reported_epoch":83,"state":"active+clean","last_fresh":"2023-07-14T19:29:03.339904+0000","last_change":"2023-07-14T19:29:01.322519+0000","last_active":"2023-07-14T19:29:03.339904+0000","last_peered":"2023-07-14T19:29:03.339904+0000","last_clean":"2023-07-14T19:29:03.339904+0000","last_became_active":"2023-07-14T19:29:01.322233+0000","last_became_peered":"2023-07-14T19:29:01.322233+0000","last_unstale":"2023-07-14T19:29:03.339904+0000","last_undegraded":"2023-07-14T19:29:03.339904+0000","last_fullsized":"2023-07-14T19:29:03.339904+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:29:00.604983+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:29:00.604983+0000","last_clean_scrub_stamp":"2023-07-14T19:29:00.604983+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T23:32:02.997469+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":12,"reported_epoch":83,"state":"active+clean","last_fresh":"2023-07-14T19:29:02.299512+0000","last_change":"2023-07-14T19:29:01.305439+0000","last_active":"2023-07-14T19:29:02.299512+0000","last_peered":"2023-07-14T19:29:02.299512+0000","last_clean":"2023-07-14T19:29:02.299512+0000","last_became_active":"2023-07-14T19:29:01.305060+0000","last_became_peered":"2023-07-14T19:29:01.305060+0000","last_unstale":"2023-07-14T19:29:02.299512+0000","last_undegraded":"2023-07-14T19:29:02.299512+0000","last_fullsized":"2023-07-14T19:29:02.299512+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:29:00.604983+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:29:00.604983+0000","last_clean_scrub_stamp":"2023-07-14T19:29:00.604983+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T06:07:47.229804+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"83'4","reported_seq":52,"reported_epoch":83,"state":"active+clean","last_fresh":"2023-07-14T19:29:38.354064+0000","last_change":"2023-07-14T19:29:00.620867+0000","last_active":"2023-07-14T19:29:38.354064+0000","last_peered":"2023-07-14T19:29:38.354064+0000","last_clean":"2023-07-14T19:29:38.354064+0000","last_became_active":"2023-07-14T19:29:00.620370+0000","last_became_peered":"2023-07-14T19:29:00.620370+0000","last_unstale":"2023-07-14T19:29:38.354064+0000","last_undegraded":"2023-07-14T19:29:38.354064+0000","last_fullsized":"2023-07-14T19:29:38.354064+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:28:59.600128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:28:59.600128+0000","last_clean_scrub_stamp":"2023-07-14T19:28:59.600128+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T05:53:10.376311+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":34,"num_read_kb":34,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":12,"reported_epoch":83,"state":"active+clean","last_fresh":"2023-07-14T19:29:02.299185+0000","last_change":"2023-07-14T19:29:01.306289+0000","last_active":"2023-07-14T19:29:02.299185+0000","last_peered":"2023-07-14T19:29:02.299185+0000","last_clean":"2023-07-14T19:29:02.299185+0000","last_became_active":"2023-07-14T19:29:01.305987+0000","last_became_peered":"2023-07-14T19:29:01.305987+0000","last_unstale":"2023-07-14T19:29:02.299185+0000","last_undegraded":"2023-07-14T19:29:02.299185+0000","last_fullsized":"2023-07-14T19:29:02.299185+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:29:00.604983+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:29:00.604983+0000","last_clean_scrub_stamp":"2023-07-14T19:29:00.604983+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T07:21:20.067549+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"83'3","reported_seq":19,"reported_epoch":83,"state":"active+clean","last_fresh":"2023-07-14T19:29:06.299569+0000","last_change":"2023-07-14T19:29:00.620937+0000","last_active":"2023-07-14T19:29:06.299569+0000","last_peered":"2023-07-14T19:29:06.299569+0000","last_clean":"2023-07-14T19:29:06.299569+0000","last_became_active":"2023-07-14T19:29:00.620616+0000","last_became_peered":"2023-07-14T19:29:00.620616+0000","last_unstale":"2023-07-14T19:29:06.299569+0000","last_undegraded":"2023-07-14T19:29:06.299569+0000","last_fullsized":"2023-07-14T19:29:06.299569+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:28:59.600128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:28:59.600128+0000","last_clean_scrub_stamp":"2023-07-14T19:28:59.600128+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T20:32:12.990940+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":4,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"22'137","reported_seq":341,"reported_epoch":83,"state":"active+clean","last_fresh":"2023-07-14T19:29:03.338586+0000","last_change":"2023-07-14T19:28:01.948805+0000","last_active":"2023-07-14T19:29:03.338586+0000","last_peered":"2023-07-14T19:29:03.338586+0000","last_clean":"2023-07-14T19:29:03.338586+0000","last_became_active":"2023-07-14T19:28:01.948295+0000","last_became_peered":"2023-07-14T19:28:01.948295+0000","last_unstale":"2023-07-14T19:29:03.338586+0000","last_undegraded":"2023-07-14T19:29:03.338586+0000","last_fullsized":"2023-07-14T19:29:03.338586+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:25:10.240820+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:25:10.240820+0000","last_clean_scrub_stamp":"2023-07-14T19:25:10.240820+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T23:39:19.337174+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":12,"reported_epoch":83,"state":"active+clean","last_fresh":"2023-07-14T19:29:02.444463+0000","last_change":"2023-07-14T19:29:01.321025+0000","last_active":"2023-07-14T19:29:02.444463+0000","last_peered":"2023-07-14T19:29:02.444463+0000","last_clean":"2023-07-14T19:29:02.444463+0000","last_became_active":"2023-07-14T19:29:01.320787+0000","last_became_peered":"2023-07-14T19:29:01.320787+0000","last_unstale":"2023-07-14T19:29:02.444463+0000","last_undegraded":"2023-07-14T19:29:02.444463+0000","last_fullsized":"2023-07-14T19:29:02.444463+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:29:00.604983+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:29:00.604983+0000","last_clean_scrub_stamp":"2023-07-14T19:29:00.604983+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T02:26:20.883377+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"83'3","reported_seq":17,"reported_epoch":83,"state":"active+clean","last_fresh":"2023-07-14T19:29:05.679444+0000","last_change":"2023-07-14T19:29:00.623891+0000","last_active":"2023-07-14T19:29:05.679444+0000","last_peered":"2023-07-14T19:29:05.679444+0000","last_clean":"2023-07-14T19:29:05.679444+0000","last_became_active":"2023-07-14T19:29:00.623562+0000","last_became_peered":"2023-07-14T19:29:00.623562+0000","last_unstale":"2023-07-14T19:29:05.679444+0000","last_undegraded":"2023-07-14T19:29:05.679444+0000","last_fullsized":"2023-07-14T19:29:05.679444+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:28:59.600128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:28:59.600128+0000","last_clean_scrub_stamp":"2023-07-14T19:28:59.600128+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T19:35:24.864363+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.5","version":"83'1","reported_seq":16,"reported_epoch":83,"state":"active+clean","last_fresh":"2023-07-14T19:29:04.670790+0000","last_change":"2023-07-14T19:29:00.620282+0000","last_active":"2023-07-14T19:29:04.670790+0000","last_peered":"2023-07-14T19:29:04.670790+0000","last_clean":"2023-07-14T19:29:04.670790+0000","last_became_active":"2023-07-14T19:29:00.620005+0000","last_became_peered":"2023-07-14T19:29:00.620005+0000","last_unstale":"2023-07-14T19:29:04.670790+0000","last_undegraded":"2023-07-14T19:29:04.670790+0000","last_fullsized":"2023-07-14T19:29:04.670790+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:28:59.600128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:28:59.600128+0000","last_clean_scrub_stamp":"2023-07-14T19:28:59.600128+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T03:20:25.333743+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":12,"reported_epoch":83,"state":"active+clean","last_fresh":"2023-07-14T19:29:02.444350+0000","last_change":"2023-07-14T19:29:01.311708+0000","last_active":"2023-07-14T19:29:02.444350+0000","last_peered":"2023-07-14T19:29:02.444350+0000","last_clean":"2023-07-14T19:29:02.444350+0000","last_became_active":"2023-07-14T19:29:01.311400+0000","last_became_peered":"2023-07-14T19:29:01.311400+0000","last_unstale":"2023-07-14T19:29:02.444350+0000","last_undegraded":"2023-07-14T19:29:02.444350+0000","last_fullsized":"2023-07-14T19:29:02.444350+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:29:00.604983+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:29:00.604983+0000","last_clean_scrub_stamp":"2023-07-14T19:29:00.604983+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T23:01:26.499242+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.6","version":"83'5","reported_seq":19,"reported_epoch":83,"state":"active+clean","last_fresh":"2023-07-14T19:29:10.296968+0000","last_change":"2023-07-14T19:29:00.617411+0000","last_active":"2023-07-14T19:29:10.296968+0000","last_peered":"2023-07-14T19:29:10.296968+0000","last_clean":"2023-07-14T19:29:10.296968+0000","last_became_active":"2023-07-14T19:29:00.617078+0000","last_became_peered":"2023-07-14T19:29:00.617078+0000","last_unstale":"2023-07-14T19:29:10.296968+0000","last_undegraded":"2023-07-14T19:29:10.296968+0000","last_fullsized":"2023-07-14T19:29:10.296968+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:28:59.600128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:28:59.600128+0000","last_clean_scrub_stamp":"2023-07-14T19:28:59.600128+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T21:38:43.985609+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":12,"reported_epoch":83,"state":"active+clean","last_fresh":"2023-07-14T19:29:02.299351+0000","last_change":"2023-07-14T19:29:01.307282+0000","last_active":"2023-07-14T19:29:02.299351+0000","last_peered":"2023-07-14T19:29:02.299351+0000","last_clean":"2023-07-14T19:29:02.299351+0000","last_became_active":"2023-07-14T19:29:01.306966+0000","last_became_peered":"2023-07-14T19:29:01.306966+0000","last_unstale":"2023-07-14T19:29:02.299351+0000","last_undegraded":"2023-07-14T19:29:02.299351+0000","last_fullsized":"2023-07-14T19:29:02.299351+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:29:00.604983+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:29:00.604983+0000","last_clean_scrub_stamp":"2023-07-14T19:29:00.604983+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T23:06:00.918796+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"83'4","reported_seq":18,"reported_epoch":83,"state":"active+clean","last_fresh":"2023-07-14T19:29:04.671123+0000","last_change":"2023-07-14T19:29:00.624599+0000","last_active":"2023-07-14T19:29:04.671123+0000","last_peered":"2023-07-14T19:29:04.671123+0000","last_clean":"2023-07-14T19:29:04.671123+0000","last_became_active":"2023-07-14T19:29:00.624272+0000","last_became_peered":"2023-07-14T19:29:00.624272+0000","last_unstale":"2023-07-14T19:29:04.671123+0000","last_undegraded":"2023-07-14T19:29:04.671123+0000","last_fullsized":"2023-07-14T19:29:04.671123+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:28:59.600128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:28:59.600128+0000","last_clean_scrub_stamp":"2023-07-14T19:28:59.600128+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T20:00:31.406685+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":12,"reported_epoch":83,"state":"active+clean","last_fresh":"2023-07-14T19:29:02.603663+0000","last_change":"2023-07-14T19:29:01.309411+0000","last_active":"2023-07-14T19:29:02.603663+0000","last_peered":"2023-07-14T19:29:02.603663+0000","last_clean":"2023-07-14T19:29:02.603663+0000","last_became_active":"2023-07-14T19:29:01.309113+0000","last_became_peered":"2023-07-14T19:29:01.309113+0000","last_unstale":"2023-07-14T19:29:02.603663+0000","last_undegraded":"2023-07-14T19:29:02.603663+0000","last_fullsized":"2023-07-14T19:29:02.603663+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:29:00.604983+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:29:00.604983+0000","last_clean_scrub_stamp":"2023-07-14T19:29:00.604983+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T07:09:16.642605+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"83'2","reported_seq":16,"reported_epoch":83,"state":"active+clean","last_fresh":"2023-07-14T19:29:04.671782+0000","last_change":"2023-07-14T19:29:00.622758+0000","last_active":"2023-07-14T19:29:04.671782+0000","last_peered":"2023-07-14T19:29:04.671782+0000","last_clean":"2023-07-14T19:29:04.671782+0000","last_became_active":"2023-07-14T19:29:00.622460+0000","last_became_peered":"2023-07-14T19:29:00.622460+0000","last_unstale":"2023-07-14T19:29:04.671782+0000","last_undegraded":"2023-07-14T19:29:04.671782+0000","last_fullsized":"2023-07-14T19:29:04.671782+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:28:59.600128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:28:59.600128+0000","last_clean_scrub_stamp":"2023-07-14T19:28:59.600128+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T23:30:15.407153+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":12,"reported_epoch":83,"state":"active+clean","last_fresh":"2023-07-14T19:29:02.660717+0000","last_change":"2023-07-14T19:29:01.317046+0000","last_active":"2023-07-14T19:29:02.660717+0000","last_peered":"2023-07-14T19:29:02.660717+0000","last_clean":"2023-07-14T19:29:02.660717+0000","last_became_active":"2023-07-14T19:29:01.316653+0000","last_became_peered":"2023-07-14T19:29:01.316653+0000","last_unstale":"2023-07-14T19:29:02.660717+0000","last_undegraded":"2023-07-14T19:29:02.660717+0000","last_fullsized":"2023-07-14T19:29:02.660717+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:29:00.604983+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:29:00.604983+0000","last_clean_scrub_stamp":"2023-07-14T19:29:00.604983+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T04:50:19.726545+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.9","version":"83'3","reported_seq":17,"reported_epoch":83,"state":"active+clean","last_fresh":"2023-07-14T19:29:04.670955+0000","last_change":"2023-07-14T19:29:00.619474+0000","last_active":"2023-07-14T19:29:04.670955+0000","last_peered":"2023-07-14T19:29:04.670955+0000","last_clean":"2023-07-14T19:29:04.670955+0000","last_became_active":"2023-07-14T19:29:00.619240+0000","last_became_peered":"2023-07-14T19:29:00.619240+0000","last_unstale":"2023-07-14T19:29:04.670955+0000","last_undegraded":"2023-07-14T19:29:04.670955+0000","last_fullsized":"2023-07-14T19:29:04.670955+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:28:59.600128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:28:59.600128+0000","last_clean_scrub_stamp":"2023-07-14T19:28:59.600128+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T02:51:09.423733+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":12,"reported_epoch":83,"state":"active+clean","last_fresh":"2023-07-14T19:29:02.573459+0000","last_change":"2023-07-14T19:29:01.308964+0000","last_active":"2023-07-14T19:29:02.573459+0000","last_peered":"2023-07-14T19:29:02.573459+0000","last_clean":"2023-07-14T19:29:02.573459+0000","last_became_active":"2023-07-14T19:29:01.308638+0000","last_became_peered":"2023-07-14T19:29:01.308638+0000","last_unstale":"2023-07-14T19:29:02.573459+0000","last_undegraded":"2023-07-14T19:29:02.573459+0000","last_fullsized":"2023-07-14T19:29:02.573459+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:29:00.604983+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:29:00.604983+0000","last_clean_scrub_stamp":"2023-07-14T19:29:00.604983+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T03:51:09.875511+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.a","version":"83'1","reported_seq":15,"reported_epoch":83,"state":"active+clean","last_fresh":"2023-07-14T19:29:03.960747+0000","last_change":"2023-07-14T19:29:00.617894+0000","last_active":"2023-07-14T19:29:03.960747+0000","last_peered":"2023-07-14T19:29:03.960747+0000","last_clean":"2023-07-14T19:29:03.960747+0000","last_became_active":"2023-07-14T19:29:00.617584+0000","last_became_peered":"2023-07-14T19:29:00.617584+0000","last_unstale":"2023-07-14T19:29:03.960747+0000","last_undegraded":"2023-07-14T19:29:03.960747+0000","last_fullsized":"2023-07-14T19:29:03.960747+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:28:59.600128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:28:59.600128+0000","last_clean_scrub_stamp":"2023-07-14T19:28:59.600128+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T23:24:30.467604+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":12,"reported_epoch":83,"state":"active+clean","last_fresh":"2023-07-14T19:29:02.299239+0000","last_change":"2023-07-14T19:29:01.317153+0000","last_active":"2023-07-14T19:29:02.299239+0000","last_peered":"2023-07-14T19:29:02.299239+0000","last_clean":"2023-07-14T19:29:02.299239+0000","last_became_active":"2023-07-14T19:29:01.316800+0000","last_became_peered":"2023-07-14T19:29:01.316800+0000","last_unstale":"2023-07-14T19:29:02.299239+0000","last_undegraded":"2023-07-14T19:29:02.299239+0000","last_fullsized":"2023-07-14T19:29:02.299239+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:29:00.604983+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:29:00.604983+0000","last_clean_scrub_stamp":"2023-07-14T19:29:00.604983+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T04:58:24.726933+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"83'4","reported_seq":19,"reported_epoch":83,"state":"active+clean","last_fresh":"2023-07-14T19:29:04.672156+0000","last_change":"2023-07-14T19:29:00.618469+0000","last_active":"2023-07-14T19:29:04.672156+0000","last_peered":"2023-07-14T19:29:04.672156+0000","last_clean":"2023-07-14T19:29:04.672156+0000","last_became_active":"2023-07-14T19:29:00.618218+0000","last_became_peered":"2023-07-14T19:29:00.618218+0000","last_unstale":"2023-07-14T19:29:04.672156+0000","last_undegraded":"2023-07-14T19:29:04.672156+0000","last_fullsized":"2023-07-14T19:29:04.672156+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:28:59.600128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:28:59.600128+0000","last_clean_scrub_stamp":"2023-07-14T19:28:59.600128+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T22:41:36.925396+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":12,"reported_epoch":83,"state":"active+clean","last_fresh":"2023-07-14T19:29:03.339251+0000","last_change":"2023-07-14T19:29:01.319023+0000","last_active":"2023-07-14T19:29:03.339251+0000","last_peered":"2023-07-14T19:29:03.339251+0000","last_clean":"2023-07-14T19:29:03.339251+0000","last_became_active":"2023-07-14T19:29:01.311530+0000","last_became_peered":"2023-07-14T19:29:01.311530+0000","last_unstale":"2023-07-14T19:29:03.339251+0000","last_undegraded":"2023-07-14T19:29:03.339251+0000","last_fullsized":"2023-07-14T19:29:03.339251+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:29:00.604983+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:29:00.604983+0000","last_clean_scrub_stamp":"2023-07-14T19:29:00.604983+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T01:46:30.175019+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"83'3","reported_seq":18,"reported_epoch":83,"state":"active+clean","last_fresh":"2023-07-14T19:29:05.678777+0000","last_change":"2023-07-14T19:29:00.624227+0000","last_active":"2023-07-14T19:29:05.678777+0000","last_peered":"2023-07-14T19:29:05.678777+0000","last_clean":"2023-07-14T19:29:05.678777+0000","last_became_active":"2023-07-14T19:29:00.623953+0000","last_became_peered":"2023-07-14T19:29:00.623953+0000","last_unstale":"2023-07-14T19:29:05.678777+0000","last_undegraded":"2023-07-14T19:29:05.678777+0000","last_fullsized":"2023-07-14T19:29:05.678777+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:28:59.600128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:28:59.600128+0000","last_clean_scrub_stamp":"2023-07-14T19:28:59.600128+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T04:31:39.691751+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":12,"reported_epoch":83,"state":"active+clean","last_fresh":"2023-07-14T19:29:03.340024+0000","last_change":"2023-07-14T19:29:01.319936+0000","last_active":"2023-07-14T19:29:03.340024+0000","last_peered":"2023-07-14T19:29:03.340024+0000","last_clean":"2023-07-14T19:29:03.340024+0000","last_became_active":"2023-07-14T19:29:01.319437+0000","last_became_peered":"2023-07-14T19:29:01.319437+0000","last_unstale":"2023-07-14T19:29:03.340024+0000","last_undegraded":"2023-07-14T19:29:03.340024+0000","last_fullsized":"2023-07-14T19:29:03.340024+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:29:00.604983+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:29:00.604983+0000","last_clean_scrub_stamp":"2023-07-14T19:29:00.604983+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T07:14:58.343588+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"82'1","reported_seq":16,"reported_epoch":83,"state":"active+clean","last_fresh":"2023-07-14T19:29:02.299764+0000","last_change":"2023-07-14T19:29:00.628103+0000","last_active":"2023-07-14T19:29:02.299764+0000","last_peered":"2023-07-14T19:29:02.299764+0000","last_clean":"2023-07-14T19:29:02.299764+0000","last_became_active":"2023-07-14T19:29:00.627689+0000","last_became_peered":"2023-07-14T19:29:00.627689+0000","last_unstale":"2023-07-14T19:29:02.299764+0000","last_undegraded":"2023-07-14T19:29:02.299764+0000","last_fullsized":"2023-07-14T19:29:02.299764+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:28:59.600128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:28:59.600128+0000","last_clean_scrub_stamp":"2023-07-14T19:28:59.600128+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T23:24:22.985508+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"83'1","reported_seq":13,"reported_epoch":83,"state":"active+clean","last_fresh":"2023-07-14T19:29:05.363002+0000","last_change":"2023-07-14T19:29:01.319519+0000","last_active":"2023-07-14T19:29:05.363002+0000","last_peered":"2023-07-14T19:29:05.363002+0000","last_clean":"2023-07-14T19:29:05.363002+0000","last_became_active":"2023-07-14T19:29:01.318900+0000","last_became_peered":"2023-07-14T19:29:01.318900+0000","last_unstale":"2023-07-14T19:29:05.363002+0000","last_undegraded":"2023-07-14T19:29:05.363002+0000","last_fullsized":"2023-07-14T19:29:05.363002+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:29:00.604983+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:29:00.604983+0000","last_clean_scrub_stamp":"2023-07-14T19:29:00.604983+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T20:31:55.947593+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":123,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"82'2","reported_seq":16,"reported_epoch":83,"state":"active+clean","last_fresh":"2023-07-14T19:29:02.444432+0000","last_change":"2023-07-14T19:29:00.619280+0000","last_active":"2023-07-14T19:29:02.444432+0000","last_peered":"2023-07-14T19:29:02.444432+0000","last_clean":"2023-07-14T19:29:02.444432+0000","last_became_active":"2023-07-14T19:29:00.618980+0000","last_became_peered":"2023-07-14T19:29:00.618980+0000","last_unstale":"2023-07-14T19:29:02.444432+0000","last_undegraded":"2023-07-14T19:29:02.444432+0000","last_fullsized":"2023-07-14T19:29:02.444432+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:28:59.600128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:28:59.600128+0000","last_clean_scrub_stamp":"2023-07-14T19:28:59.600128+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T22:09:48.530415+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":12,"reported_epoch":83,"state":"active+clean","last_fresh":"2023-07-14T19:29:03.339356+0000","last_change":"2023-07-14T19:29:01.320160+0000","last_active":"2023-07-14T19:29:03.339356+0000","last_peered":"2023-07-14T19:29:03.339356+0000","last_clean":"2023-07-14T19:29:03.339356+0000","last_became_active":"2023-07-14T19:29:01.319850+0000","last_became_peered":"2023-07-14T19:29:01.319850+0000","last_unstale":"2023-07-14T19:29:03.339356+0000","last_undegraded":"2023-07-14T19:29:03.339356+0000","last_fullsized":"2023-07-14T19:29:03.339356+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:29:00.604983+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:29:00.604983+0000","last_clean_scrub_stamp":"2023-07-14T19:29:00.604983+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T04:25:40.648850+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":12,"reported_epoch":83,"state":"active+clean","last_fresh":"2023-07-14T19:29:02.573560+0000","last_change":"2023-07-14T19:29:01.309770+0000","last_active":"2023-07-14T19:29:02.573560+0000","last_peered":"2023-07-14T19:29:02.573560+0000","last_clean":"2023-07-14T19:29:02.573560+0000","last_became_active":"2023-07-14T19:29:01.308677+0000","last_became_peered":"2023-07-14T19:29:01.308677+0000","last_unstale":"2023-07-14T19:29:02.573560+0000","last_undegraded":"2023-07-14T19:29:02.573560+0000","last_fullsized":"2023-07-14T19:29:02.573560+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:29:00.604983+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:29:00.604983+0000","last_clean_scrub_stamp":"2023-07-14T19:29:00.604983+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T21:27:03.743070+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":12,"reported_epoch":83,"state":"active+clean","last_fresh":"2023-07-14T19:29:03.221821+0000","last_change":"2023-07-14T19:29:01.319447+0000","last_active":"2023-07-14T19:29:03.221821+0000","last_peered":"2023-07-14T19:29:03.221821+0000","last_clean":"2023-07-14T19:29:03.221821+0000","last_became_active":"2023-07-14T19:29:01.319120+0000","last_became_peered":"2023-07-14T19:29:01.319120+0000","last_unstale":"2023-07-14T19:29:03.221821+0000","last_undegraded":"2023-07-14T19:29:03.221821+0000","last_fullsized":"2023-07-14T19:29:03.221821+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:29:00.604983+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:29:00.604983+0000","last_clean_scrub_stamp":"2023-07-14T19:29:00.604983+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T07:03:28.055942+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":12,"reported_epoch":83,"state":"active+clean","last_fresh":"2023-07-14T19:29:03.221929+0000","last_change":"2023-07-14T19:29:01.310554+0000","last_active":"2023-07-14T19:29:03.221929+0000","last_peered":"2023-07-14T19:29:03.221929+0000","last_clean":"2023-07-14T19:29:03.221929+0000","last_became_active":"2023-07-14T19:29:01.310302+0000","last_became_peered":"2023-07-14T19:29:01.310302+0000","last_unstale":"2023-07-14T19:29:03.221929+0000","last_undegraded":"2023-07-14T19:29:03.221929+0000","last_fullsized":"2023-07-14T19:29:03.221929+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:29:00.604983+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:29:00.604983+0000","last_clean_scrub_stamp":"2023-07-14T19:29:00.604983+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T04:50:18.928722+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":12,"reported_epoch":83,"state":"active+clean","last_fresh":"2023-07-14T19:29:02.603764+0000","last_change":"2023-07-14T19:29:01.318753+0000","last_active":"2023-07-14T19:29:02.603764+0000","last_peered":"2023-07-14T19:29:02.603764+0000","last_clean":"2023-07-14T19:29:02.603764+0000","last_became_active":"2023-07-14T19:29:01.318476+0000","last_became_peered":"2023-07-14T19:29:01.318476+0000","last_unstale":"2023-07-14T19:29:02.603764+0000","last_undegraded":"2023-07-14T19:29:02.603764+0000","last_fullsized":"2023-07-14T19:29:02.603764+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:29:00.604983+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:29:00.604983+0000","last_clean_scrub_stamp":"2023-07-14T19:29:00.604983+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T07:27:40.189788+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":12,"reported_epoch":83,"state":"active+clean","last_fresh":"2023-07-14T19:29:02.444422+0000","last_change":"2023-07-14T19:29:01.317843+0000","last_active":"2023-07-14T19:29:02.444422+0000","last_peered":"2023-07-14T19:29:02.444422+0000","last_clean":"2023-07-14T19:29:02.444422+0000","last_became_active":"2023-07-14T19:29:01.317550+0000","last_became_peered":"2023-07-14T19:29:01.317550+0000","last_unstale":"2023-07-14T19:29:02.444422+0000","last_undegraded":"2023-07-14T19:29:02.444422+0000","last_fullsized":"2023-07-14T19:29:02.444422+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:29:00.604983+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:29:00.604983+0000","last_clean_scrub_stamp":"2023-07-14T19:29:00.604983+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T22:59:15.139142+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":12,"reported_epoch":83,"state":"active+clean","last_fresh":"2023-07-14T19:29:02.573731+0000","last_change":"2023-07-14T19:29:01.309103+0000","last_active":"2023-07-14T19:29:02.573731+0000","last_peered":"2023-07-14T19:29:02.573731+0000","last_clean":"2023-07-14T19:29:02.573731+0000","last_became_active":"2023-07-14T19:29:01.308786+0000","last_became_peered":"2023-07-14T19:29:01.308786+0000","last_unstale":"2023-07-14T19:29:02.573731+0000","last_undegraded":"2023-07-14T19:29:02.573731+0000","last_fullsized":"2023-07-14T19:29:02.573731+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:29:00.604983+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:29:00.604983+0000","last_clean_scrub_stamp":"2023-07-14T19:29:00.604983+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T02:55:34.529448+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":12,"reported_epoch":83,"state":"active+clean","last_fresh":"2023-07-14T19:29:03.222213+0000","last_change":"2023-07-14T19:29:01.309940+0000","last_active":"2023-07-14T19:29:03.222213+0000","last_peered":"2023-07-14T19:29:03.222213+0000","last_clean":"2023-07-14T19:29:03.222213+0000","last_became_active":"2023-07-14T19:29:01.309638+0000","last_became_peered":"2023-07-14T19:29:01.309638+0000","last_unstale":"2023-07-14T19:29:03.222213+0000","last_undegraded":"2023-07-14T19:29:03.222213+0000","last_fullsized":"2023-07-14T19:29:03.222213+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:29:00.604983+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:29:00.604983+0000","last_clean_scrub_stamp":"2023-07-14T19:29:00.604983+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T03:28:26.563593+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":12,"reported_epoch":83,"state":"active+clean","last_fresh":"2023-07-14T19:29:02.299283+0000","last_change":"2023-07-14T19:29:01.305957+0000","last_active":"2023-07-14T19:29:02.299283+0000","last_peered":"2023-07-14T19:29:02.299283+0000","last_clean":"2023-07-14T19:29:02.299283+0000","last_became_active":"2023-07-14T19:29:01.305564+0000","last_became_peered":"2023-07-14T19:29:01.305564+0000","last_unstale":"2023-07-14T19:29:02.299283+0000","last_undegraded":"2023-07-14T19:29:02.299283+0000","last_fullsized":"2023-07-14T19:29:02.299283+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:29:00.604983+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:29:00.604983+0000","last_clean_scrub_stamp":"2023-07-14T19:29:00.604983+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T21:47:06.297622+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":12,"reported_epoch":83,"state":"active+clean","last_fresh":"2023-07-14T19:29:02.299375+0000","last_change":"2023-07-14T19:29:01.310521+0000","last_active":"2023-07-14T19:29:02.299375+0000","last_peered":"2023-07-14T19:29:02.299375+0000","last_clean":"2023-07-14T19:29:02.299375+0000","last_became_active":"2023-07-14T19:29:01.310211+0000","last_became_peered":"2023-07-14T19:29:01.310211+0000","last_unstale":"2023-07-14T19:29:02.299375+0000","last_undegraded":"2023-07-14T19:29:02.299375+0000","last_fullsized":"2023-07-14T19:29:02.299375+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:29:00.604983+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:29:00.604983+0000","last_clean_scrub_stamp":"2023-07-14T19:29:00.604983+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T19:41:09.317667+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":12,"reported_epoch":83,"state":"active+clean","last_fresh":"2023-07-14T19:29:02.298983+0000","last_change":"2023-07-14T19:29:01.308033+0000","last_active":"2023-07-14T19:29:02.298983+0000","last_peered":"2023-07-14T19:29:02.298983+0000","last_clean":"2023-07-14T19:29:02.298983+0000","last_became_active":"2023-07-14T19:29:01.307166+0000","last_became_peered":"2023-07-14T19:29:01.307166+0000","last_unstale":"2023-07-14T19:29:02.298983+0000","last_undegraded":"2023-07-14T19:29:02.298983+0000","last_fullsized":"2023-07-14T19:29:02.298983+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:29:00.604983+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:29:00.604983+0000","last_clean_scrub_stamp":"2023-07-14T19:29:00.604983+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T06:29:57.949181+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":12,"reported_epoch":83,"state":"active+clean","last_fresh":"2023-07-14T19:29:02.299844+0000","last_change":"2023-07-14T19:29:01.308122+0000","last_active":"2023-07-14T19:29:02.299844+0000","last_peered":"2023-07-14T19:29:02.299844+0000","last_clean":"2023-07-14T19:29:02.299844+0000","last_became_active":"2023-07-14T19:29:01.307568+0000","last_became_peered":"2023-07-14T19:29:01.307568+0000","last_unstale":"2023-07-14T19:29:02.299844+0000","last_undegraded":"2023-07-14T19:29:02.299844+0000","last_fullsized":"2023-07-14T19:29:02.299844+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:29:00.604983+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:29:00.604983+0000","last_clean_scrub_stamp":"2023-07-14T19:29:00.604983+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T04:08:56.378075+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":12,"reported_epoch":83,"state":"active+clean","last_fresh":"2023-07-14T19:29:02.444527+0000","last_change":"2023-07-14T19:29:01.314261+0000","last_active":"2023-07-14T19:29:02.444527+0000","last_peered":"2023-07-14T19:29:02.444527+0000","last_clean":"2023-07-14T19:29:02.444527+0000","last_became_active":"2023-07-14T19:29:01.313967+0000","last_became_peered":"2023-07-14T19:29:01.313967+0000","last_unstale":"2023-07-14T19:29:02.444527+0000","last_undegraded":"2023-07-14T19:29:02.444527+0000","last_fullsized":"2023-07-14T19:29:02.444527+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:29:00.604983+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:29:00.604983+0000","last_clean_scrub_stamp":"2023-07-14T19:29:00.604983+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T04:19:07.923742+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":12,"reported_epoch":83,"state":"active+clean","last_fresh":"2023-07-14T19:29:02.573578+0000","last_change":"2023-07-14T19:29:01.319605+0000","last_active":"2023-07-14T19:29:02.573578+0000","last_peered":"2023-07-14T19:29:02.573578+0000","last_clean":"2023-07-14T19:29:02.573578+0000","last_became_active":"2023-07-14T19:29:01.319310+0000","last_became_peered":"2023-07-14T19:29:01.319310+0000","last_unstale":"2023-07-14T19:29:02.573578+0000","last_undegraded":"2023-07-14T19:29:02.573578+0000","last_fullsized":"2023-07-14T19:29:02.573578+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:29:00.604983+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:29:00.604983+0000","last_clean_scrub_stamp":"2023-07-14T19:29:00.604983+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T20:14:29.714925+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":246,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":738,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":4751360,"data_stored":4722944,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":58954,"num_objects":60,"num_object_clones":0,"num_object_copies":180,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":60,"num_whiteouts":0,"num_read":231,"num_read_kb":176,"num_write":149,"num_write_kb":100,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":405504,"data_stored":176862,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":131,"ondisk_log_size":131,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":78,"seq":335007449107,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27892,"kb_used_data":876,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717260,"statfs":{"total":95995035648,"available":95966474240,"internally_reserved":0,"allocated":897024,"data_stored":429039,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":27654411},"hb_peers":[0,1,2,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jul 14 19:29:19 2023","interfaces":[{"interface":"back","average":{"1min":0.52800000000000002,"5min":0.52800000000000002,"15min":0.52800000000000002},"min":{"1min":0.33400000000000002,"5min":0.33400000000000002,"15min":0.33400000000000002},"max":{"1min":0.82999999999999996,"5min":0.82999999999999996,"15min":0.82999999999999996},"last":0.76500000000000001},{"interface":"front","average":{"1min":0.56100000000000005,"5min":0.56100000000000005,"15min":0.56100000000000005},"min":{"1min":0.38800000000000001,"5min":0.38800000000000001,"15min":0.38800000000000001},"max":{"1min":0.95599999999999996,"5min":0.95599999999999996,"15min":0.95599999999999996},"last":0.623}]},{"osd":1,"last update":"Fri Jul 14 19:29:19 2023","interfaces":[{"interface":"back","average":{"1min":0.65800000000000003,"5min":0.65800000000000003,"15min":0.65800000000000003},"min":{"1min":0.41499999999999998,"5min":0.41499999999999998,"15min":0.41499999999999998},"max":{"1min":0.99399999999999999,"5min":0.99399999999999999,"15min":0.99399999999999999},"last":0.60199999999999998},{"interface":"front","average":{"1min":0.60699999999999998,"5min":0.60699999999999998,"15min":0.60699999999999998},"min":{"1min":0.42099999999999999,"5min":0.42099999999999999,"15min":0.42099999999999999},"max":{"1min":1.5640000000000001,"5min":1.5640000000000001,"15min":1.5640000000000001},"last":0.51200000000000001}]},{"osd":2,"last update":"Fri Jul 14 19:29:19 2023","interfaces":[{"interface":"back","average":{"1min":0.52800000000000002,"5min":0.52800000000000002,"15min":0.52800000000000002},"min":{"1min":0.30299999999999999,"5min":0.30299999999999999,"15min":0.30299999999999999},"max":{"1min":1.097,"5min":1.097,"15min":1.097},"last":0.41999999999999998},{"interface":"front","average":{"1min":0.51800000000000002,"5min":0.51800000000000002,"15min":0.51800000000000002},"min":{"1min":0.30099999999999999,"5min":0.30099999999999999,"15min":0.30099999999999999},"max":{"1min":0.86399999999999999,"5min":0.86399999999999999,"15min":0.86399999999999999},"last":0.48299999999999998}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.37}]},{"osd":4,"last update":"Fri Jul 14 19:29:19 2023","interfaces":[{"interface":"back","average":{"1min":0.68600000000000005,"5min":0.68600000000000005,"15min":0.68600000000000005},"min":{"1min":0.39300000000000002,"5min":0.39300000000000002,"15min":0.39300000000000002},"max":{"1min":1.0389999999999999,"5min":1.0389999999999999,"15min":1.0389999999999999},"last":0.66800000000000004},{"interface":"front","average":{"1min":0.63700000000000001,"5min":0.63700000000000001,"15min":0.63700000000000001},"min":{"1min":0.41699999999999998,"5min":0.41699999999999998,"15min":0.41699999999999998},"max":{"1min":0.91800000000000004,"5min":0.91800000000000004,"15min":0.91800000000000004},"last":0.45800000000000002}]},{"osd":5,"last update":"Fri Jul 14 19:29:19 2023","interfaces":[{"interface":"back","average":{"1min":0.61299999999999999,"5min":0.61299999999999999,"15min":0.61299999999999999},"min":{"1min":0.32500000000000001,"5min":0.32500000000000001,"15min":0.32500000000000001},"max":{"1min":1.1930000000000001,"5min":1.1930000000000001,"15min":1.1930000000000001},"last":0.54000000000000004},{"interface":"front","average":{"1min":0.61599999999999999,"5min":0.61599999999999999,"15min":0.61599999999999999},"min":{"1min":0.36699999999999999,"5min":0.36699999999999999,"15min":0.36699999999999999},"max":{"1min":1.018,"5min":1.018,"15min":1.018},"last":0.73999999999999999}]},{"osd":6,"last update":"Fri Jul 14 19:29:19 2023","interfaces":[{"interface":"back","average":{"1min":0.62,"5min":0.62,"15min":0.62},"min":{"1min":0.40100000000000002,"5min":0.40100000000000002,"15min":0.40100000000000002},"max":{"1min":0.97599999999999998,"5min":0.97599999999999998,"15min":0.97599999999999998},"last":0.64400000000000002},{"interface":"front","average":{"1min":0.70899999999999996,"5min":0.70899999999999996,"15min":0.70899999999999996},"min":{"1min":0.436,"5min":0.436,"15min":0.436},"max":{"1min":1,"5min":1,"15min":1},"last":0.68999999999999995}]},{"osd":7,"last update":"Fri Jul 14 19:29:19 2023","interfaces":[{"interface":"back","average":{"1min":0.76900000000000002,"5min":0.76900000000000002,"15min":0.76900000000000002},"min":{"1min":0.52400000000000002,"5min":0.52400000000000002,"15min":0.52400000000000002},"max":{"1min":1.032,"5min":1.032,"15min":1.032},"last":0.81200000000000006},{"interface":"front","average":{"1min":0.71299999999999997,"5min":0.71299999999999997,"15min":0.71299999999999997},"min":{"1min":0.54100000000000004,"5min":0.54100000000000004,"15min":0.54100000000000004},"max":{"1min":1.0269999999999999,"5min":1.0269999999999999,"15min":1.0269999999999999},"last":0.71099999999999997}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.754}]},{"osd":9,"last update":"Fri Jul 14 19:29:19 2023","interfaces":[{"interface":"back","average":{"1min":0.77600000000000002,"5min":0.77600000000000002,"15min":0.77600000000000002},"min":{"1min":0.52600000000000002,"5min":0.52600000000000002,"15min":0.52600000000000002},"max":{"1min":1.2230000000000001,"5min":1.2230000000000001,"15min":1.2230000000000001},"last":0.72699999999999998},{"interface":"front","average":{"1min":0.78000000000000003,"5min":0.78000000000000003,"15min":0.78000000000000003},"min":{"1min":0.51700000000000002,"5min":0.51700000000000002,"15min":0.51700000000000002},"max":{"1min":1.0700000000000001,"5min":1.0700000000000001,"15min":1.0700000000000001},"last":0.84099999999999997}]},{"osd":10,"last update":"Fri Jul 14 19:29:19 2023","interfaces":[{"interface":"back","average":{"1min":0.74099999999999999,"5min":0.74099999999999999,"15min":0.74099999999999999},"min":{"1min":0.56299999999999994,"5min":0.56299999999999994,"15min":0.56299999999999994},"max":{"1min":0.93100000000000005,"5min":0.93100000000000005,"15min":0.93100000000000005},"last":0.77800000000000002},{"interface":"front","average":{"1min":0.81200000000000006,"5min":0.81200000000000006,"15min":0.81200000000000006},"min":{"1min":0.56799999999999995,"5min":0.56799999999999995,"15min":0.56799999999999995},"max":{"1min":1.2609999999999999,"5min":1.2609999999999999,"15min":1.2609999999999999},"last":0.79700000000000004}]}]},{"osd":10,"up_from":70,"seq":300647710744,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28564,"kb_used_data":1548,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716588,"statfs":{"total":95995035648,"available":95965786112,"internally_reserved":0,"allocated":1585152,"data_stored":1075197,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":27654411},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.40400000000000003}]},{"osd":1,"last update":"Fri Jul 14 19:29:00 2023","interfaces":[{"interface":"back","average":{"1min":0.44800000000000001,"5min":0.44800000000000001,"15min":0.44800000000000001},"min":{"1min":0.27000000000000002,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":0.73299999999999998,"5min":0.73299999999999998,"15min":0.73299999999999998},"last":0.47899999999999998},{"interface":"front","average":{"1min":0.46800000000000003,"5min":0.46800000000000003,"15min":0.46800000000000003},"min":{"1min":0.28599999999999998,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":0.80800000000000005,"5min":0.80800000000000005,"15min":0.80800000000000005},"last":0.54500000000000004}]},{"osd":2,"last update":"Fri Jul 14 19:29:00 2023","interfaces":[{"interface":"back","average":{"1min":0.59899999999999998,"5min":0.59899999999999998,"15min":0.59899999999999998},"min":{"1min":0.40200000000000002,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":0.83699999999999997,"5min":0.83699999999999997,"15min":0.83699999999999997},"last":0.61399999999999999},{"interface":"front","average":{"1min":0.64300000000000002,"5min":0.64300000000000002,"15min":0.64300000000000002},"min":{"1min":0.46800000000000003,"5min":0.46800000000000003,"15min":0.46800000000000003},"max":{"1min":0.89100000000000001,"5min":0.89100000000000001,"15min":0.89100000000000001},"last":0.78100000000000003}]},{"osd":3,"last update":"Fri Jul 14 19:29:00 2023","interfaces":[{"interface":"back","average":{"1min":0.58199999999999996,"5min":0.58199999999999996,"15min":0.58199999999999996},"min":{"1min":0.35999999999999999,"5min":0.35999999999999999,"15min":0.35999999999999999},"max":{"1min":0.77800000000000002,"5min":0.77800000000000002,"15min":0.77800000000000002},"last":0.64100000000000001},{"interface":"front","average":{"1min":0.63200000000000001,"5min":0.63200000000000001,"15min":0.63200000000000001},"min":{"1min":0.45200000000000001,"5min":0.45200000000000001,"15min":0.45200000000000001},"max":{"1min":0.89700000000000002,"5min":0.89700000000000002,"15min":0.89700000000000002},"last":0.50600000000000001}]},{"osd":4,"last update":"Fri Jul 14 19:29:00 2023","interfaces":[{"interface":"back","average":{"1min":0.55600000000000005,"5min":0.55600000000000005,"15min":0.55600000000000005},"min":{"1min":0.316,"5min":0.316,"15min":0.316},"max":{"1min":0.99399999999999999,"5min":0.99399999999999999,"15min":0.99399999999999999},"last":0.434},{"interface":"front","average":{"1min":0.54800000000000004,"5min":0.54800000000000004,"15min":0.54800000000000004},"min":{"1min":0.29299999999999998,"5min":0.29299999999999998,"15min":0.29299999999999998},"max":{"1min":0.94999999999999996,"5min":0.94999999999999996,"15min":0.94999999999999996},"last":0.57399999999999995}]},{"osd":5,"last update":"Fri Jul 14 19:29:00 2023","interfaces":[{"interface":"back","average":{"1min":0.68600000000000005,"5min":0.68600000000000005,"15min":0.68600000000000005},"min":{"1min":0.45300000000000001,"5min":0.45300000000000001,"15min":0.45300000000000001},"max":{"1min":1.0129999999999999,"5min":1.0129999999999999,"15min":1.0129999999999999},"last":0.68300000000000005},{"interface":"front","average":{"1min":0.67100000000000004,"5min":0.67100000000000004,"15min":0.67100000000000004},"min":{"1min":0.374,"5min":0.374,"15min":0.374},"max":{"1min":0.96499999999999997,"5min":0.96499999999999997,"15min":0.96499999999999997},"last":0.748}]},{"osd":6,"last update":"Fri Jul 14 19:29:00 2023","interfaces":[{"interface":"back","average":{"1min":0.65700000000000003,"5min":0.65700000000000003,"15min":0.65700000000000003},"min":{"1min":0.40400000000000003,"5min":0.40400000000000003,"15min":0.40400000000000003},"max":{"1min":1.032,"5min":1.032,"15min":1.032},"last":0.69799999999999995},{"interface":"front","average":{"1min":0.68799999999999994,"5min":0.68799999999999994,"15min":0.68799999999999994},"min":{"1min":0.45700000000000002,"5min":0.45700000000000002,"15min":0.45700000000000002},"max":{"1min":1.0649999999999999,"5min":1.0649999999999999,"15min":1.0649999999999999},"last":0.59299999999999997}]},{"osd":7,"last update":"Fri Jul 14 19:29:00 2023","interfaces":[{"interface":"back","average":{"1min":0.66000000000000003,"5min":0.66000000000000003,"15min":0.66000000000000003},"min":{"1min":0.34200000000000003,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":1.0960000000000001,"5min":1.0960000000000001,"15min":1.0960000000000001},"last":0.46000000000000002},{"interface":"front","average":{"1min":0.69399999999999995,"5min":0.69399999999999995,"15min":0.69399999999999995},"min":{"1min":0.44600000000000001,"5min":0.44600000000000001,"15min":0.44600000000000001},"max":{"1min":0.97899999999999998,"5min":0.97899999999999998,"15min":0.97899999999999998},"last":0.44600000000000001}]},{"osd":8,"last update":"Fri Jul 14 19:29:00 2023","interfaces":[{"interface":"back","average":{"1min":0.75900000000000001,"5min":0.75900000000000001,"15min":0.75900000000000001},"min":{"1min":0.52800000000000002,"5min":0.52800000000000002,"15min":0.52800000000000002},"max":{"1min":1.143,"5min":1.143,"15min":1.143},"last":0.71599999999999997},{"interface":"front","average":{"1min":0.77600000000000002,"5min":0.77600000000000002,"15min":0.77600000000000002},"min":{"1min":0.47599999999999998,"5min":0.47599999999999998,"15min":0.47599999999999998},"max":{"1min":1.1060000000000001,"5min":1.1060000000000001,"15min":1.1060000000000001},"last":0.79500000000000004}]},{"osd":9,"last update":"Fri Jul 14 19:29:00 2023","interfaces":[{"interface":"back","average":{"1min":0.754,"5min":0.754,"15min":0.754},"min":{"1min":0.54500000000000004,"5min":0.54500000000000004,"15min":0.54500000000000004},"max":{"1min":1.115,"5min":1.115,"15min":1.115},"last":0.66700000000000004},{"interface":"front","average":{"1min":0.77000000000000002,"5min":0.77000000000000002,"15min":0.77000000000000002},"min":{"1min":0.49299999999999999,"5min":0.49299999999999999,"15min":0.49299999999999999},"max":{"1min":1.127,"5min":1.127,"15min":1.127},"last":0.73299999999999998}]},{"osd":11,"last update":"Fri Jul 14 19:29:25 2023","interfaces":[{"interface":"back","average":{"1min":0.75,"5min":0.75,"15min":0.75},"min":{"1min":0.54000000000000004,"5min":0.54000000000000004,"15min":0.54000000000000004},"max":{"1min":1.0489999999999999,"5min":1.0489999999999999,"15min":1.0489999999999999},"last":0.82099999999999995},{"interface":"front","average":{"1min":0.75800000000000001,"5min":0.75800000000000001,"15min":0.75800000000000001},"min":{"1min":0.54900000000000004,"5min":0.54900000000000004,"15min":0.54900000000000004},"max":{"1min":0.99299999999999999,"5min":0.99299999999999999,"15min":0.99299999999999999},"last":0.76400000000000001}]}]},{"osd":9,"up_from":65,"seq":279172874269,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27908,"kb_used_data":892,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717244,"statfs":{"total":95995035648,"available":95966457856,"internally_reserved":0,"allocated":913408,"data_stored":429275,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":27654411},"hb_peers":[0,1,2,3,4,5,6,7,8,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jul 14 19:29:38 2023","interfaces":[{"interface":"back","average":{"1min":0.34699999999999998,"5min":0.41599999999999998,"15min":0.42799999999999999},"min":{"1min":0.218,"5min":0.218,"15min":0.218},"max":{"1min":0.55000000000000004,"5min":0.55000000000000004,"15min":0.55000000000000004},"last":0.28699999999999998},{"interface":"front","average":{"1min":0.53200000000000003,"5min":0.48399999999999999,"15min":0.47599999999999998},"min":{"1min":0.28699999999999998,"5min":0.26100000000000001,"15min":0.26100000000000001},"max":{"1min":2.0619999999999998,"5min":2.0619999999999998,"15min":2.0619999999999998},"last":0.26100000000000001}]},{"osd":1,"last update":"Fri Jul 14 19:29:38 2023","interfaces":[{"interface":"back","average":{"1min":0.50900000000000001,"5min":0.60699999999999998,"15min":0.623},"min":{"1min":0.35099999999999998,"5min":0.35099999999999998,"15min":0.35099999999999998},"max":{"1min":0.73599999999999999,"5min":0.82199999999999995,"15min":0.82199999999999995},"last":0.41499999999999998},{"interface":"front","average":{"1min":0.53600000000000003,"5min":0.54500000000000004,"15min":0.54700000000000004},"min":{"1min":0.34100000000000003,"5min":0.34100000000000003,"15min":0.34100000000000003},"max":{"1min":0.86699999999999999,"5min":0.86699999999999999,"15min":0.86699999999999999},"last":0.27500000000000002}]},{"osd":2,"last update":"Fri Jul 14 19:29:38 2023","interfaces":[{"interface":"back","average":{"1min":0.53600000000000003,"5min":0.61099999999999999,"15min":0.623},"min":{"1min":0.28899999999999998,"5min":0.28899999999999998,"15min":0.28899999999999998},"max":{"1min":0.66100000000000003,"5min":0.75800000000000001,"15min":0.75800000000000001},"last":0.44700000000000001},{"interface":"front","average":{"1min":0.57099999999999995,"5min":0.58499999999999996,"15min":0.58699999999999997},"min":{"1min":0.36499999999999999,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":0.72899999999999998,"5min":0.72899999999999998,"15min":0.72899999999999998},"last":0.499}]},{"osd":3,"last update":"Fri Jul 14 19:29:38 2023","interfaces":[{"interface":"back","average":{"1min":0.57399999999999995,"5min":0.53000000000000003,"15min":0.52200000000000002},"min":{"1min":0.249,"5min":0.249,"15min":0.249},"max":{"1min":2.4260000000000002,"5min":2.4260000000000002,"15min":2.4260000000000002},"last":0.33300000000000002},{"interface":"front","average":{"1min":0.42399999999999999,"5min":0.55200000000000005,"15min":0.57299999999999995},"min":{"1min":0.29799999999999999,"5min":0.29799999999999999,"15min":0.29799999999999999},"max":{"1min":0.84299999999999997,"5min":0.85699999999999998,"15min":0.85699999999999998},"last":0.376}]},{"osd":4,"last update":"Fri Jul 14 19:29:38 2023","interfaces":[{"interface":"back","average":{"1min":0.54900000000000004,"5min":0.58599999999999997,"15min":0.59199999999999997},"min":{"1min":0.34100000000000003,"5min":0.34100000000000003,"15min":0.34100000000000003},"max":{"1min":0.76700000000000002,"5min":0.77200000000000002,"15min":0.77200000000000002},"last":0.34300000000000003},{"interface":"front","average":{"1min":0.72599999999999998,"5min":0.67300000000000004,"15min":0.66400000000000003},"min":{"1min":0.42599999999999999,"5min":0.42599999999999999,"15min":0.42599999999999999},"max":{"1min":2.3450000000000002,"5min":2.3450000000000002,"15min":2.3450000000000002},"last":0.35899999999999999}]},{"osd":5,"last update":"Fri Jul 14 19:29:38 2023","interfaces":[{"interface":"back","average":{"1min":0.76100000000000001,"5min":0.68899999999999995,"15min":0.67700000000000005},"min":{"1min":0.39100000000000001,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":2.274,"5min":2.274,"15min":2.274},"last":0.42799999999999999},{"interface":"front","average":{"1min":0.61199999999999999,"5min":0.71299999999999997,"15min":0.72999999999999998},"min":{"1min":0.35499999999999998,"5min":0.35499999999999998,"15min":0.35499999999999998},"max":{"1min":0.88900000000000001,"5min":0.90600000000000003,"15min":0.90600000000000003},"last":0.53500000000000003}]},{"osd":6,"last update":"Fri Jul 14 19:29:38 2023","interfaces":[{"interface":"back","average":{"1min":0.49099999999999999,"5min":0.65300000000000002,"15min":0.68000000000000005},"min":{"1min":0.26800000000000002,"5min":0.26800000000000002,"15min":0.26800000000000002},"max":{"1min":0.98899999999999999,"5min":0.98899999999999999,"15min":0.98899999999999999},"last":0.48399999999999999},{"interface":"front","average":{"1min":0.90800000000000003,"5min":0.66400000000000003,"15min":0.624},"min":{"1min":0.29799999999999999,"5min":0.29799999999999999,"15min":0.29799999999999999},"max":{"1min":8.0980000000000008,"5min":8.0980000000000008,"15min":8.0980000000000008},"last":0.316}]},{"osd":7,"last update":"Fri Jul 14 19:29:38 2023","interfaces":[{"interface":"back","average":{"1min":0.79400000000000004,"5min":0.76500000000000001,"15min":0.76000000000000001},"min":{"1min":0.45500000000000002,"5min":0.45500000000000002,"15min":0.45500000000000002},"max":{"1min":2.3980000000000001,"5min":2.3980000000000001,"15min":2.3980000000000001},"last":0.47599999999999998},{"interface":"front","average":{"1min":0.61099999999999999,"5min":0.74199999999999999,"15min":0.76400000000000001},"min":{"1min":0.46999999999999997,"5min":0.46999999999999997,"15min":0.46999999999999997},"max":{"1min":0.93799999999999994,"5min":0.95299999999999996,"15min":0.95299999999999996},"last":0.51600000000000001}]},{"osd":8,"last update":"Fri Jul 14 19:29:38 2023","interfaces":[{"interface":"back","average":{"1min":1.0549999999999999,"5min":0.78400000000000003,"15min":0.73899999999999999},"min":{"1min":0.439,"5min":0.439,"15min":0.439},"max":{"1min":8.0129999999999999,"5min":8.0129999999999999,"15min":8.0129999999999999},"last":0.52600000000000002},{"interface":"front","average":{"1min":0.82799999999999996,"5min":0.75900000000000001,"15min":0.747},"min":{"1min":0.41199999999999998,"5min":0.41199999999999998,"15min":0.41199999999999998},"max":{"1min":2.3130000000000002,"5min":2.3130000000000002,"15min":2.3130000000000002},"last":0.50900000000000001}]},{"osd":10,"last update":"Fri Jul 14 19:29:00 2023","interfaces":[{"interface":"back","average":{"1min":0.70299999999999996,"5min":0.70299999999999996,"15min":0.70299999999999996},"min":{"1min":0.439,"5min":0.439,"15min":0.439},"max":{"1min":0.95199999999999996,"5min":0.95199999999999996,"15min":0.95199999999999996},"last":0.46800000000000003},{"interface":"front","average":{"1min":0.73799999999999999,"5min":0.73799999999999999,"15min":0.73799999999999999},"min":{"1min":0.47799999999999998,"5min":0.47799999999999998,"15min":0.47799999999999998},"max":{"1min":1.016,"5min":1.016,"15min":1.016},"last":0.45800000000000002}]}]},{"osd":8,"up_from":57,"seq":244813135904,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27904,"kb_used_data":888,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717248,"statfs":{"total":95995035648,"available":95966461952,"internally_reserved":0,"allocated":909312,"data_stored":428914,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":27654411},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jul 14 19:29:19 2023","interfaces":[{"interface":"back","average":{"1min":0.51100000000000001,"5min":0.54600000000000004,"15min":0.55200000000000005},"min":{"1min":0.36899999999999999,"5min":0.30099999999999999,"15min":0.30099999999999999},"max":{"1min":0.82099999999999995,"5min":0.82099999999999995,"15min":0.82099999999999995},"last":0.38400000000000001},{"interface":"front","average":{"1min":0.61199999999999999,"5min":0.55800000000000005,"15min":0.54900000000000004},"min":{"1min":0.33400000000000002,"5min":0.32100000000000001,"15min":0.32100000000000001},"max":{"1min":2.0270000000000001,"5min":2.0270000000000001,"15min":2.0270000000000001},"last":0.496}]},{"osd":1,"last update":"Fri Jul 14 19:29:19 2023","interfaces":[{"interface":"back","average":{"1min":0.623,"5min":0.58999999999999997,"15min":0.58399999999999996},"min":{"1min":0.372,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":0.83599999999999997,"5min":0.85499999999999998,"15min":0.85499999999999998},"last":0.53200000000000003},{"interface":"front","average":{"1min":0.61799999999999999,"5min":0.60599999999999998,"15min":0.60399999999999998},"min":{"1min":0.42099999999999999,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":0.81999999999999995,"5min":0.93400000000000005,"15min":0.93400000000000005},"last":0.54700000000000004}]},{"osd":2,"last update":"Fri Jul 14 19:29:19 2023","interfaces":[{"interface":"back","average":{"1min":0.49399999999999999,"5min":0.47599999999999998,"15min":0.47299999999999998},"min":{"1min":0.311,"5min":0.27700000000000002,"15min":0.27700000000000002},"max":{"1min":0.63900000000000001,"5min":0.67200000000000004,"15min":0.67200000000000004},"last":0.443},{"interface":"front","average":{"1min":0.48999999999999999,"5min":0.46200000000000002,"15min":0.45700000000000002},"min":{"1min":0.29899999999999999,"5min":0.25600000000000001,"15min":0.25600000000000001},"max":{"1min":0.80700000000000005,"5min":0.80700000000000005,"15min":0.80700000000000005},"last":0.51400000000000001}]},{"osd":3,"last update":"Fri Jul 14 19:29:19 2023","interfaces":[{"interface":"back","average":{"1min":0.60199999999999998,"5min":0.57799999999999996,"15min":0.57399999999999995},"min":{"1min":0.38700000000000001,"5min":0.36199999999999999,"15min":0.36199999999999999},"max":{"1min":1.1120000000000001,"5min":1.1120000000000001,"15min":1.1120000000000001},"last":0.42299999999999999},{"interface":"front","average":{"1min":0.65500000000000003,"5min":0.59799999999999998,"15min":0.58799999999999997},"min":{"1min":0.32500000000000001,"5min":0.32500000000000001,"15min":0.32500000000000001},"max":{"1min":1.1890000000000001,"5min":1.1890000000000001,"15min":1.1890000000000001},"last":0.66200000000000003}]},{"osd":4,"last update":"Fri Jul 14 19:29:19 2023","interfaces":[{"interface":"back","average":{"1min":0.64400000000000002,"5min":0.61199999999999999,"15min":0.60599999999999998},"min":{"1min":0.34300000000000003,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":0.84799999999999998,"5min":0.86399999999999999,"15min":0.86399999999999999},"last":0.59599999999999997},{"interface":"front","average":{"1min":0.67400000000000004,"5min":0.64400000000000002,"15min":0.63900000000000001},"min":{"1min":0.41899999999999998,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":0.88100000000000001,"5min":0.88100000000000001,"15min":0.88100000000000001},"last":0.60499999999999998}]},{"osd":5,"last update":"Fri Jul 14 19:29:19 2023","interfaces":[{"interface":"back","average":{"1min":0.58999999999999997,"5min":0.55800000000000005,"15min":0.55300000000000005},"min":{"1min":0.39700000000000002,"5min":0.311,"15min":0.311},"max":{"1min":0.88100000000000001,"5min":0.88100000000000001,"15min":0.88100000000000001},"last":0.47599999999999998},{"interface":"front","average":{"1min":0.61499999999999999,"5min":0.57499999999999996,"15min":0.56899999999999995},"min":{"1min":0.433,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":0.84399999999999997,"5min":0.90900000000000003,"15min":0.90900000000000003},"last":0.64000000000000001}]},{"osd":6,"last update":"Fri Jul 14 19:29:19 2023","interfaces":[{"interface":"back","average":{"1min":0.77500000000000002,"5min":0.67800000000000005,"15min":0.66200000000000003},"min":{"1min":0.501,"5min":0.436,"15min":0.436},"max":{"1min":2.012,"5min":2.012,"15min":2.012},"last":0.68799999999999994},{"interface":"front","average":{"1min":0.67500000000000004,"5min":0.67900000000000005,"15min":0.67900000000000005},"min":{"1min":0.48699999999999999,"5min":0.45600000000000002,"15min":0.45600000000000002},"max":{"1min":0.92200000000000004,"5min":1.1479999999999999,"15min":1.1479999999999999},"last":0.57599999999999996}]},{"osd":7,"last update":"Fri Jul 14 19:29:19 2023","interfaces":[{"interface":"back","average":{"1min":0.71899999999999997,"5min":0.72399999999999998,"15min":0.72499999999999998},"min":{"1min":0.54600000000000004,"5min":0.53400000000000003,"15min":0.53400000000000003},"max":{"1min":0.96299999999999997,"5min":1.2070000000000001,"15min":1.2070000000000001},"last":0.622},{"interface":"front","average":{"1min":0.75700000000000001,"5min":0.72099999999999997,"15min":0.71499999999999997},"min":{"1min":0.56299999999999994,"5min":0.47499999999999998,"15min":0.47499999999999998},"max":{"1min":0.90900000000000003,"5min":1.127,"15min":1.127},"last":0.79400000000000004}]},{"osd":9,"last update":"Fri Jul 14 19:28:39 2023","interfaces":[{"interface":"back","average":{"1min":0.68000000000000005,"5min":0.68000000000000005,"15min":0.68000000000000005},"min":{"1min":0.497,"5min":0.497,"15min":0.497},"max":{"1min":0.997,"5min":0.997,"15min":0.997},"last":0.745},{"interface":"front","average":{"1min":0.68400000000000005,"5min":0.68400000000000005,"15min":0.68400000000000005},"min":{"1min":0.48299999999999998,"5min":0.48299999999999998,"15min":0.48299999999999998},"max":{"1min":0.86299999999999999,"5min":0.86299999999999999,"15min":0.86299999999999999},"last":0.75800000000000001}]},{"osd":10,"last update":"Fri Jul 14 19:28:58 2023","interfaces":[{"interface":"back","average":{"1min":0.69599999999999995,"5min":0.69599999999999995,"15min":0.69599999999999995},"min":{"1min":0.55800000000000005,"5min":0.55800000000000005,"15min":0.55800000000000005},"max":{"1min":0.94799999999999995,"5min":0.94799999999999995,"15min":0.94799999999999995},"last":0.76700000000000002},{"interface":"front","average":{"1min":0.72299999999999998,"5min":0.72299999999999998,"15min":0.72299999999999998},"min":{"1min":0.55200000000000005,"5min":0.55200000000000005,"15min":0.55200000000000005},"max":{"1min":0.91100000000000003,"5min":0.91100000000000003,"15min":0.91100000000000003},"last":0.77900000000000003}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.70099999999999996}]}]},{"osd":7,"up_from":50,"seq":214748364836,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28488,"kb_used_data":1472,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716664,"statfs":{"total":95995035648,"available":95965863936,"internally_reserved":0,"allocated":1507328,"data_stored":1021424,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":27654411},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jul 14 19:28:59 2023","interfaces":[{"interface":"back","average":{"1min":0.60799999999999998,"5min":0.57299999999999995,"15min":0.56699999999999995},"min":{"1min":0.41599999999999998,"5min":0.40400000000000003,"15min":0.40400000000000003},"max":{"1min":0.95899999999999996,"5min":1.0349999999999999,"15min":1.0349999999999999},"last":0.45100000000000001},{"interface":"front","average":{"1min":0.56499999999999995,"5min":0.54200000000000004,"15min":0.53800000000000003},"min":{"1min":0.34899999999999998,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":1.0089999999999999,"5min":1.0089999999999999,"15min":1.0089999999999999},"last":0.40999999999999998}]},{"osd":1,"last update":"Fri Jul 14 19:28:59 2023","interfaces":[{"interface":"back","average":{"1min":0.47299999999999998,"5min":0.45600000000000002,"15min":0.45300000000000001},"min":{"1min":0.27600000000000002,"5min":0.27600000000000002,"15min":0.27600000000000002},"max":{"1min":0.76000000000000001,"5min":0.76000000000000001,"15min":0.76000000000000001},"last":0.437},{"interface":"front","average":{"1min":0.48899999999999999,"5min":0.47699999999999998,"15min":0.47499999999999998},"min":{"1min":0.251,"5min":0.251,"15min":0.251},"max":{"1min":0.81999999999999995,"5min":0.81999999999999995,"15min":0.81999999999999995},"last":0.47899999999999998}]},{"osd":2,"last update":"Fri Jul 14 19:28:59 2023","interfaces":[{"interface":"back","average":{"1min":0.64700000000000002,"5min":0.621,"15min":0.61699999999999999},"min":{"1min":0.41399999999999998,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":1.091,"5min":1.091,"15min":1.091},"last":0.76000000000000001},{"interface":"front","average":{"1min":0.63800000000000001,"5min":0.61799999999999999,"15min":0.61499999999999999},"min":{"1min":0.35199999999999998,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":1.1040000000000001,"5min":1.1040000000000001,"15min":1.1040000000000001},"last":0.625}]},{"osd":3,"last update":"Fri Jul 14 19:28:59 2023","interfaces":[{"interface":"back","average":{"1min":0.624,"5min":0.57999999999999996,"15min":0.57299999999999995},"min":{"1min":0.45100000000000001,"5min":0.38800000000000001,"15min":0.38800000000000001},"max":{"1min":0.90200000000000002,"5min":1.0109999999999999,"15min":1.0109999999999999},"last":0.56299999999999994},{"interface":"front","average":{"1min":0.64900000000000002,"5min":0.621,"15min":0.61599999999999999},"min":{"1min":0.495,"5min":0.47299999999999998,"15min":0.47299999999999998},"max":{"1min":0.88500000000000001,"5min":0.88600000000000001,"15min":0.88600000000000001},"last":0.64500000000000002}]},{"osd":4,"last update":"Fri Jul 14 19:28:59 2023","interfaces":[{"interface":"back","average":{"1min":0.60799999999999998,"5min":0.58899999999999997,"15min":0.58599999999999997},"min":{"1min":0.36099999999999999,"5min":0.36099999999999999,"15min":0.36099999999999999},"max":{"1min":1.125,"5min":1.125,"15min":1.125},"last":0.57899999999999996},{"interface":"front","average":{"1min":0.58799999999999997,"5min":0.54200000000000004,"15min":0.53400000000000003},"min":{"1min":0.33600000000000002,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":0.879,"5min":0.879,"15min":0.879},"last":0.46000000000000002}]},{"osd":5,"last update":"Fri Jul 14 19:28:59 2023","interfaces":[{"interface":"back","average":{"1min":0.69899999999999995,"5min":0.68300000000000005,"15min":0.68000000000000005},"min":{"1min":0.38300000000000001,"5min":0.38300000000000001,"15min":0.38300000000000001},"max":{"1min":1.173,"5min":1.173,"15min":1.173},"last":0.67600000000000005},{"interface":"front","average":{"1min":0.69299999999999995,"5min":0.69299999999999995,"15min":0.69299999999999995},"min":{"1min":0.40200000000000002,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":1.0329999999999999,"5min":1.0329999999999999,"15min":1.0329999999999999},"last":0.83199999999999996}]},{"osd":6,"last update":"Fri Jul 14 19:28:59 2023","interfaces":[{"interface":"back","average":{"1min":0.72699999999999998,"5min":0.68799999999999994,"15min":0.68200000000000005},"min":{"1min":0.48499999999999999,"5min":0.42799999999999999,"15min":0.42799999999999999},"max":{"1min":1.0309999999999999,"5min":1.0309999999999999,"15min":1.0309999999999999},"last":0.73699999999999999},{"interface":"front","average":{"1min":0.71899999999999997,"5min":0.69299999999999995,"15min":0.68899999999999995},"min":{"1min":0.50800000000000001,"5min":0.443,"15min":0.443},"max":{"1min":0.93999999999999995,"5min":1.0209999999999999,"15min":1.0209999999999999},"last":0.51000000000000001}]},{"osd":8,"last update":"Fri Jul 14 19:29:22 2023","interfaces":[{"interface":"back","average":{"1min":0.72399999999999998,"5min":0.748,"15min":0.752},"min":{"1min":0.44600000000000001,"5min":0.44600000000000001,"15min":0.44600000000000001},"max":{"1min":0.92700000000000005,"5min":1.0720000000000001,"15min":1.0720000000000001},"last":0.88600000000000001},{"interface":"front","average":{"1min":0.71399999999999997,"5min":0.745,"15min":0.75},"min":{"1min":0.42899999999999999,"5min":0.42899999999999999,"15min":0.42899999999999999},"max":{"1min":0.96699999999999997,"5min":1.141,"15min":1.141},"last":0.84899999999999998}]},{"osd":9,"last update":"Fri Jul 14 19:28:39 2023","interfaces":[{"interface":"back","average":{"1min":0.74099999999999999,"5min":0.74099999999999999,"15min":0.74099999999999999},"min":{"1min":0.53400000000000003,"5min":0.53400000000000003,"15min":0.53400000000000003},"max":{"1min":1.0489999999999999,"5min":1.0489999999999999,"15min":1.0489999999999999},"last":0.55400000000000005},{"interface":"front","average":{"1min":0.75900000000000001,"5min":0.75900000000000001,"15min":0.75900000000000001},"min":{"1min":0.53300000000000003,"5min":0.53300000000000003,"15min":0.53300000000000003},"max":{"1min":1.1519999999999999,"5min":1.1519999999999999,"15min":1.1519999999999999},"last":0.69499999999999995}]},{"osd":10,"last update":"Fri Jul 14 19:28:59 2023","interfaces":[{"interface":"back","average":{"1min":0.72999999999999998,"5min":0.72999999999999998,"15min":0.72999999999999998},"min":{"1min":0.38700000000000001,"5min":0.38700000000000001,"15min":0.38700000000000001},"max":{"1min":1.077,"5min":1.077,"15min":1.077},"last":0.79600000000000004},{"interface":"front","average":{"1min":0.74199999999999999,"5min":0.74199999999999999,"15min":0.74199999999999999},"min":{"1min":0.38500000000000001,"5min":0.38500000000000001,"15min":0.38500000000000001},"max":{"1min":1.117,"5min":1.117,"15min":1.117},"last":0.78200000000000003}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.86199999999999999}]}]},{"osd":6,"up_from":45,"seq":193273528361,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27896,"kb_used_data":880,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717256,"statfs":{"total":95995035648,"available":95966470144,"internally_reserved":0,"allocated":901120,"data_stored":428670,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":27654411},"hb_peers":[0,1,2,3,4,5,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jul 14 19:28:36 2023","interfaces":[{"interface":"back","average":{"1min":0.42099999999999999,"5min":0.38200000000000001,"15min":0.376},"min":{"1min":0.29099999999999998,"5min":0.23400000000000001,"15min":0.23400000000000001},"max":{"1min":0.63,"5min":0.63,"15min":0.63},"last":0.50800000000000001},{"interface":"front","average":{"1min":0.42699999999999999,"5min":0.38900000000000001,"15min":0.38300000000000001},"min":{"1min":0.32000000000000001,"5min":0.249,"15min":0.249},"max":{"1min":0.63900000000000001,"5min":0.63900000000000001,"15min":0.63900000000000001},"last":0.34000000000000002}]},{"osd":1,"last update":"Fri Jul 14 19:28:36 2023","interfaces":[{"interface":"back","average":{"1min":0.66400000000000003,"5min":0.53600000000000003,"15min":0.51400000000000001},"min":{"1min":0.41699999999999998,"5min":0.29899999999999999,"15min":0.29899999999999999},"max":{"1min":0.83999999999999997,"5min":0.83999999999999997,"15min":0.83999999999999997},"last":0.71599999999999997},{"interface":"front","average":{"1min":0.58199999999999996,"5min":0.505,"15min":0.49199999999999999},"min":{"1min":0.36699999999999999,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":0.872,"5min":0.872,"15min":0.872},"last":0.47399999999999998}]},{"osd":2,"last update":"Fri Jul 14 19:28:36 2023","interfaces":[{"interface":"back","average":{"1min":0.58399999999999996,"5min":0.58799999999999997,"15min":0.58899999999999997},"min":{"1min":0.40600000000000003,"5min":0.27600000000000002,"15min":0.27600000000000002},"max":{"1min":0.68799999999999994,"5min":0.76800000000000002,"15min":0.76800000000000002},"last":0.627},{"interface":"front","average":{"1min":0.628,"5min":0.61199999999999999,"15min":0.60899999999999999},"min":{"1min":0.46500000000000002,"5min":0.378,"15min":0.378},"max":{"1min":0.83299999999999996,"5min":0.83299999999999996,"15min":0.83299999999999996},"last":0.68500000000000005}]},{"osd":3,"last update":"Fri Jul 14 19:28:36 2023","interfaces":[{"interface":"back","average":{"1min":0.47399999999999998,"5min":0.47599999999999998,"15min":0.47599999999999998},"min":{"1min":0.30099999999999999,"5min":0.30099999999999999,"15min":0.30099999999999999},"max":{"1min":0.64200000000000002,"5min":0.78500000000000003,"15min":0.78500000000000003},"last":0.57799999999999996},{"interface":"front","average":{"1min":0.45100000000000001,"5min":0.441,"15min":0.439},"min":{"1min":0.315,"5min":0.28999999999999998,"15min":0.28999999999999998},"max":{"1min":0.75700000000000001,"5min":0.75700000000000001,"15min":0.75700000000000001},"last":0.434}]},{"osd":4,"last update":"Fri Jul 14 19:28:36 2023","interfaces":[{"interface":"back","average":{"1min":0.61099999999999999,"5min":0.53500000000000003,"15min":0.52200000000000002},"min":{"1min":0.39500000000000002,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":0.82899999999999996,"5min":0.82899999999999996,"15min":0.82899999999999996},"last":0.59299999999999997},{"interface":"front","average":{"1min":0.67300000000000004,"5min":0.62,"15min":0.61099999999999999},"min":{"1min":0.38700000000000001,"5min":0.38700000000000001,"15min":0.38700000000000001},"max":{"1min":0.96899999999999997,"5min":0.96899999999999997,"15min":0.96899999999999997},"last":0.74099999999999999}]},{"osd":5,"last update":"Fri Jul 14 19:28:36 2023","interfaces":[{"interface":"back","average":{"1min":0.56799999999999995,"5min":0.57399999999999995,"15min":0.57499999999999996},"min":{"1min":0.38300000000000001,"5min":0.38300000000000001,"15min":0.38300000000000001},"max":{"1min":0.752,"5min":0.81799999999999995,"15min":0.81799999999999995},"last":0.64600000000000002},{"interface":"front","average":{"1min":0.64400000000000002,"5min":0.63500000000000001,"15min":0.63300000000000001},"min":{"1min":0.47399999999999998,"5min":0.46000000000000002,"15min":0.46000000000000002},"max":{"1min":0.81399999999999995,"5min":0.88500000000000001,"15min":0.88500000000000001},"last":0.67200000000000004}]},{"osd":7,"last update":"Fri Jul 14 19:28:59 2023","interfaces":[{"interface":"back","average":{"1min":0.68600000000000005,"5min":0.68100000000000005,"15min":0.68000000000000005},"min":{"1min":0.47599999999999998,"5min":0.44500000000000001,"15min":0.44500000000000001},"max":{"1min":1.0029999999999999,"5min":1.0029999999999999,"15min":1.0029999999999999},"last":0.79700000000000004},{"interface":"front","average":{"1min":0.69699999999999995,"5min":0.69699999999999995,"15min":0.69699999999999995},"min":{"1min":0.50900000000000001,"5min":0.46700000000000003,"15min":0.46700000000000003},"max":{"1min":0.86599999999999999,"5min":1.0189999999999999,"15min":1.0189999999999999},"last":0.83599999999999997}]},{"osd":8,"last update":"Fri Jul 14 19:29:17 2023","interfaces":[{"interface":"back","average":{"1min":0.66500000000000004,"5min":0.70099999999999996,"15min":0.70699999999999996},"min":{"1min":0.48899999999999999,"5min":0.48899999999999999,"15min":0.48899999999999999},"max":{"1min":0.90200000000000002,"5min":1.1000000000000001,"15min":1.1000000000000001},"last":0.81899999999999995},{"interface":"front","average":{"1min":0.60799999999999998,"5min":0.59999999999999998,"15min":0.59799999999999998},"min":{"1min":0.46999999999999997,"5min":0.434,"15min":0.434},"max":{"1min":0.83299999999999996,"5min":0.83299999999999996,"15min":0.83299999999999996},"last":0.60699999999999998}]},{"osd":9,"last update":"Fri Jul 14 19:28:41 2023","interfaces":[{"interface":"back","average":{"1min":0.65100000000000002,"5min":0.65100000000000002,"15min":0.65100000000000002},"min":{"1min":0.434,"5min":0.434,"15min":0.434},"max":{"1min":0.90600000000000003,"5min":0.90600000000000003,"15min":0.90600000000000003},"last":0.53800000000000003},{"interface":"front","average":{"1min":0.76900000000000002,"5min":0.76900000000000002,"15min":0.76900000000000002},"min":{"1min":0.51100000000000001,"5min":0.51100000000000001,"15min":0.51100000000000001},"max":{"1min":1.1379999999999999,"5min":1.1379999999999999,"15min":1.1379999999999999},"last":0.77000000000000002}]},{"osd":10,"last update":"Fri Jul 14 19:28:59 2023","interfaces":[{"interface":"back","average":{"1min":0.63900000000000001,"5min":0.63900000000000001,"15min":0.63900000000000001},"min":{"1min":0.505,"5min":0.505,"15min":0.505},"max":{"1min":0.82999999999999996,"5min":0.82999999999999996,"15min":0.82999999999999996},"last":0.56799999999999995},{"interface":"front","average":{"1min":0.76800000000000002,"5min":0.76800000000000002,"15min":0.76800000000000002},"min":{"1min":0.58999999999999997,"5min":0.58999999999999997,"15min":0.58999999999999997},"max":{"1min":1.0329999999999999,"5min":1.0329999999999999,"15min":1.0329999999999999},"last":0.85199999999999998}]}]},{"osd":1,"up_from":13,"seq":55834574911,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27952,"kb_used_data":936,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717200,"statfs":{"total":95995035648,"available":95966412800,"internally_reserved":0,"allocated":958464,"data_stored":483117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":27654411},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jul 14 19:28:58 2023","interfaces":[{"interface":"back","average":{"1min":0.53800000000000003,"5min":0.498,"15min":0.47099999999999997},"min":{"1min":0.43099999999999999,"5min":0.215,"15min":0.215},"max":{"1min":0.82399999999999995,"5min":1.6970000000000001,"15min":1.6970000000000001},"last":0.71499999999999997},{"interface":"front","average":{"1min":0.58599999999999997,"5min":0.48699999999999999,"15min":0.46100000000000002},"min":{"1min":0.39100000000000001,"5min":0.23699999999999999,"15min":0.23699999999999999},"max":{"1min":0.82899999999999996,"5min":0.82899999999999996,"15min":0.82899999999999996},"last":0.55300000000000005}]},{"osd":2,"last update":"Fri Jul 14 19:29:15 2023","interfaces":[{"interface":"back","average":{"1min":0.625,"5min":0.52400000000000002,"15min":0.496},"min":{"1min":0.40400000000000003,"5min":0.31,"15min":0.31},"max":{"1min":0.93400000000000005,"5min":0.93400000000000005,"15min":0.93400000000000005},"last":0.73399999999999999},{"interface":"front","average":{"1min":0.61199999999999999,"5min":0.54900000000000004,"15min":0.50600000000000001},"min":{"1min":0.434,"5min":0.317,"15min":0.317},"max":{"1min":0.84199999999999997,"5min":1.74,"15min":1.74},"last":0.69299999999999995}]},{"osd":3,"last update":"Fri Jul 14 19:29:38 2023","interfaces":[{"interface":"back","average":{"1min":0.61599999999999999,"5min":0.57199999999999995,"15min":0.54200000000000004},"min":{"1min":0.38600000000000001,"5min":0.32200000000000001,"15min":0.32200000000000001},"max":{"1min":0.78400000000000003,"5min":1.8460000000000001,"15min":1.8460000000000001},"last":0.58099999999999996},{"interface":"front","average":{"1min":0.59999999999999998,"5min":0.56999999999999995,"15min":0.55200000000000005},"min":{"1min":0.38400000000000001,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.752,"5min":0.81499999999999995,"15min":0.81499999999999995},"last":0.78200000000000003}]},{"osd":4,"last update":"Fri Jul 14 19:28:58 2023","interfaces":[{"interface":"back","average":{"1min":0.51400000000000001,"5min":0.49199999999999999,"15min":0.502},"min":{"1min":0.314,"5min":0.29899999999999999,"15min":0.29899999999999999},"max":{"1min":0.78100000000000003,"5min":2.2029999999999998,"15min":2.2029999999999998},"last":0.60199999999999998},{"interface":"front","average":{"1min":0.53700000000000003,"5min":0.495,"15min":0.49199999999999999},"min":{"1min":0.309,"5min":0.26400000000000001,"15min":0.26400000000000001},"max":{"1min":0.90400000000000003,"5min":1.901,"15min":1.901},"last":0.50700000000000001}]},{"osd":5,"last update":"Fri Jul 14 19:29:15 2023","interfaces":[{"interface":"back","average":{"1min":0.68400000000000005,"5min":0.59799999999999998,"15min":0.57299999999999995},"min":{"1min":0.373,"5min":0.308,"15min":0.308},"max":{"1min":0.88100000000000001,"5min":0.88100000000000001,"15min":0.88100000000000001},"last":0.65300000000000002},{"interface":"front","average":{"1min":0.77700000000000002,"5min":0.68100000000000005,"15min":0.66100000000000003},"min":{"1min":0.57999999999999996,"5min":0.42699999999999999,"15min":0.42699999999999999},"max":{"1min":0.94899999999999995,"5min":1.8,"15min":1.8},"last":0.76400000000000001}]},{"osd":6,"last update":"Fri Jul 14 19:29:42 2023","interfaces":[{"interface":"back","average":{"1min":0.69299999999999995,"5min":0.64700000000000002,"15min":0.63700000000000001},"min":{"1min":0.48599999999999999,"5min":0.40999999999999998,"15min":0.40999999999999998},"max":{"1min":0.94699999999999995,"5min":1.8300000000000001,"15min":1.8300000000000001},"last":0.94699999999999995},{"interface":"front","average":{"1min":0.69099999999999995,"5min":0.66800000000000004,"15min":0.65900000000000003},"min":{"1min":0.54800000000000004,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":0.83299999999999996,"5min":1.875,"15min":1.875},"last":0.75}]},{"osd":7,"last update":"Fri Jul 14 19:28:58 2023","interfaces":[{"interface":"back","average":{"1min":0.67400000000000004,"5min":0.56599999999999995,"15min":0.54800000000000004},"min":{"1min":0.36199999999999999,"5min":0.36199999999999999,"15min":0.36199999999999999},"max":{"1min":0.95399999999999996,"5min":0.95399999999999996,"15min":0.95399999999999996},"last":0.81000000000000005},{"interface":"front","average":{"1min":0.70199999999999996,"5min":0.60799999999999998,"15min":0.59199999999999997},"min":{"1min":0.39200000000000002,"5min":0.38400000000000001,"15min":0.38400000000000001},"max":{"1min":1.24,"5min":1.24,"15min":1.24},"last":0.67500000000000004}]},{"osd":8,"last update":"Fri Jul 14 19:29:21 2023","interfaces":[{"interface":"back","average":{"1min":0.753,"5min":0.70399999999999996,"15min":0.69599999999999995},"min":{"1min":0.45500000000000002,"5min":0.42199999999999999,"15min":0.42199999999999999},"max":{"1min":1.091,"5min":1.091,"15min":1.091},"last":0.84799999999999998},{"interface":"front","average":{"1min":0.83599999999999997,"5min":0.75900000000000001,"15min":0.746},"min":{"1min":0.52400000000000002,"5min":0.505,"15min":0.505},"max":{"1min":1.2130000000000001,"5min":1.2130000000000001,"15min":1.2130000000000001},"last":0.89800000000000002}]},{"osd":9,"last update":"Fri Jul 14 19:29:42 2023","interfaces":[{"interface":"back","average":{"1min":0.96799999999999997,"5min":0.81399999999999995,"15min":0.78800000000000003},"min":{"1min":0.65100000000000002,"5min":0.52100000000000002,"15min":0.52100000000000002},"max":{"1min":3.0489999999999999,"5min":3.0489999999999999,"15min":3.0489999999999999},"last":0.92300000000000004},{"interface":"front","average":{"1min":0.92700000000000005,"5min":0.79400000000000004,"15min":0.77200000000000002},"min":{"1min":0.64400000000000002,"5min":0.51200000000000001,"15min":0.51200000000000001},"max":{"1min":3.0630000000000002,"5min":3.0630000000000002,"15min":3.0630000000000002},"last":0.79800000000000004}]},{"osd":10,"last update":"Fri Jul 14 19:28:58 2023","interfaces":[{"interface":"back","average":{"1min":0.73699999999999999,"5min":0.73699999999999999,"15min":0.73699999999999999},"min":{"1min":0.44800000000000001,"5min":0.44800000000000001,"15min":0.44800000000000001},"max":{"1min":1.079,"5min":1.079,"15min":1.079},"last":0.82599999999999996},{"interface":"front","average":{"1min":0.81799999999999995,"5min":0.81799999999999995,"15min":0.81799999999999995},"min":{"1min":0.58799999999999997,"5min":0.58799999999999997,"15min":0.58799999999999997},"max":{"1min":1.127,"5min":1.127,"15min":1.127},"last":0.85999999999999999}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.88300000000000001}]}]},{"osd":0,"up_from":8,"seq":34359738434,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27916,"kb_used_data":900,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717236,"statfs":{"total":95995035648,"available":95966449664,"internally_reserved":0,"allocated":921600,"data_stored":429935,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":27654411},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Jul 14 19:28:52 2023","interfaces":[{"interface":"back","average":{"1min":0.629,"5min":0.52600000000000002,"15min":0.503},"min":{"1min":0.35999999999999999,"5min":0.28000000000000003,"15min":0.28000000000000003},"max":{"1min":1.79,"5min":1.79,"15min":1.79},"last":0.626},{"interface":"front","average":{"1min":0.57199999999999995,"5min":0.51700000000000002,"15min":0.505},"min":{"1min":0.32000000000000001,"5min":0.313,"15min":0.313},"max":{"1min":1.605,"5min":1.605,"15min":1.605},"last":0.51500000000000001}]},{"osd":2,"last update":"Fri Jul 14 19:29:18 2023","interfaces":[{"interface":"back","average":{"1min":0.52200000000000002,"5min":0.54900000000000004,"15min":0.51900000000000002},"min":{"1min":0.33900000000000002,"5min":0.33900000000000002,"15min":0.33900000000000002},"max":{"1min":0.67500000000000004,"5min":0.77700000000000002,"15min":0.77700000000000002},"last":0.54600000000000004},{"interface":"front","average":{"1min":0.55200000000000005,"5min":0.54900000000000004,"15min":0.52400000000000002},"min":{"1min":0.34499999999999997,"5min":0.29999999999999999,"15min":0.29999999999999999},"max":{"1min":0.79900000000000004,"5min":0.81399999999999995,"15min":0.81399999999999995},"last":0.68799999999999994}]},{"osd":3,"last update":"Fri Jul 14 19:27:35 2023","interfaces":[{"interface":"back","average":{"1min":0.439,"5min":0.41099999999999998,"15min":0.40600000000000003},"min":{"1min":0.29399999999999998,"5min":0.26400000000000001,"15min":0.26400000000000001},"max":{"1min":0.72299999999999998,"5min":0.72299999999999998,"15min":0.72299999999999998},"last":0.38300000000000001},{"interface":"front","average":{"1min":0.44900000000000001,"5min":0.437,"15min":0.435},"min":{"1min":0.245,"5min":0.24099999999999999,"15min":0.24099999999999999},"max":{"1min":0.61499999999999999,"5min":0.61499999999999999,"15min":0.61499999999999999},"last":0.55700000000000005}]},{"osd":4,"last update":"Fri Jul 14 19:29:02 2023","interfaces":[{"interface":"back","average":{"1min":0.57099999999999995,"5min":0.53900000000000003,"15min":0.52500000000000002},"min":{"1min":0.38800000000000001,"5min":0.377,"15min":0.377},"max":{"1min":1.139,"5min":1.139,"15min":1.139},"last":0.45900000000000002},{"interface":"front","average":{"1min":0.58499999999999996,"5min":0.55000000000000004,"15min":0.53700000000000003},"min":{"1min":0.34799999999999998,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":1.2190000000000001,"5min":1.2190000000000001,"15min":1.2190000000000001},"last":0.71699999999999997}]},{"osd":5,"last update":"Fri Jul 14 19:29:18 2023","interfaces":[{"interface":"back","average":{"1min":0.56899999999999995,"5min":0.59899999999999998,"15min":0.59899999999999998},"min":{"1min":0.41699999999999998,"5min":0.41699999999999998,"15min":0.41699999999999998},"max":{"1min":0.71399999999999997,"5min":0.81799999999999995,"15min":0.81799999999999995},"last":0.754},{"interface":"front","average":{"1min":0.57999999999999996,"5min":0.60999999999999999,"15min":0.60899999999999999},"min":{"1min":0.38,"5min":0.38,"15min":0.38},"max":{"1min":0.76200000000000001,"5min":0.85499999999999998,"15min":0.85499999999999998},"last":0.82399999999999995}]},{"osd":6,"last update":"Fri Jul 14 19:29:38 2023","interfaces":[{"interface":"back","average":{"1min":0.56000000000000005,"5min":0.50800000000000001,"15min":0.5},"min":{"1min":0.30099999999999999,"5min":0.25900000000000001,"15min":0.25900000000000001},"max":{"1min":3.2650000000000001,"5min":3.2650000000000001,"15min":3.2650000000000001},"last":0.66400000000000003},{"interface":"front","average":{"1min":0.56200000000000006,"5min":0.52600000000000002,"15min":0.51200000000000001},"min":{"1min":0.32400000000000001,"5min":0.32400000000000001,"15min":0.32400000000000001},"max":{"1min":2.9129999999999998,"5min":2.9129999999999998,"15min":2.9129999999999998},"last":0.90000000000000002}]},{"osd":7,"last update":"Fri Jul 14 19:27:57 2023","interfaces":[{"interface":"back","average":{"1min":0.60299999999999998,"5min":0.60299999999999998,"15min":0.60299999999999998},"min":{"1min":0.41999999999999998,"5min":0.41999999999999998,"15min":0.41999999999999998},"max":{"1min":0.83599999999999997,"5min":0.83599999999999997,"15min":0.83599999999999997},"last":0.52100000000000002},{"interface":"front","average":{"1min":0.65200000000000002,"5min":0.65200000000000002,"15min":0.65200000000000002},"min":{"1min":0.47299999999999998,"5min":0.47299999999999998,"15min":0.47299999999999998},"max":{"1min":0.89500000000000002,"5min":0.89500000000000002,"15min":0.89500000000000002},"last":0.40799999999999997}]},{"osd":8,"last update":"Fri Jul 14 19:29:18 2023","interfaces":[{"interface":"back","average":{"1min":0.58699999999999997,"5min":0.67100000000000004,"15min":0.68500000000000005},"min":{"1min":0.36599999999999999,"5min":0.36599999999999999,"15min":0.36599999999999999},"max":{"1min":0.73899999999999999,"5min":0.80700000000000005,"15min":0.80700000000000005},"last":0.77200000000000002},{"interface":"front","average":{"1min":0.63200000000000001,"5min":0.69699999999999995,"15min":0.70799999999999996},"min":{"1min":0.40699999999999997,"5min":0.40699999999999997,"15min":0.40699999999999997},"max":{"1min":0.81899999999999995,"5min":0.86499999999999999,"15min":0.86499999999999999},"last":0.52800000000000002}]},{"osd":9,"last update":"Fri Jul 14 19:29:40 2023","interfaces":[{"interface":"back","average":{"1min":0.56999999999999995,"5min":0.622,"15min":0.63},"min":{"1min":0.34100000000000003,"5min":0.34100000000000003,"15min":0.34100000000000003},"max":{"1min":1.254,"5min":1.254,"15min":1.254},"last":0.79300000000000004},{"interface":"front","average":{"1min":0.54800000000000004,"5min":0.623,"15min":0.63500000000000001},"min":{"1min":0.35799999999999998,"5min":0.32300000000000001,"15min":0.32300000000000001},"max":{"1min":0.78300000000000003,"5min":0.96099999999999997,"15min":0.96099999999999997},"last":0.78300000000000003}]},{"osd":10,"last update":"Fri Jul 14 19:29:02 2023","interfaces":[{"interface":"back","average":{"1min":0.754,"5min":0.754,"15min":0.754},"min":{"1min":0.45900000000000002,"5min":0.45900000000000002,"15min":0.45900000000000002},"max":{"1min":1.776,"5min":1.776,"15min":1.776},"last":0.64500000000000002},{"interface":"front","average":{"1min":0.72499999999999998,"5min":0.72499999999999998,"15min":0.72499999999999998},"min":{"1min":0.502,"5min":0.502,"15min":0.502},"max":{"1min":1.522,"5min":1.522,"15min":1.522},"last":0.60899999999999999}]},{"osd":11,"last update":"Fri Jul 14 19:29:23 2023","interfaces":[{"interface":"back","average":{"1min":0.65900000000000003,"5min":0.65900000000000003,"15min":0.65900000000000003},"min":{"1min":0.48299999999999998,"5min":0.48299999999999998,"15min":0.48299999999999998},"max":{"1min":0.85999999999999999,"5min":0.85999999999999999,"15min":0.85999999999999999},"last":0.84899999999999998},{"interface":"front","average":{"1min":0.77300000000000002,"5min":0.77300000000000002,"15min":0.77300000000000002},"min":{"1min":0.53600000000000003,"5min":0.53600000000000003,"15min":0.53600000000000003},"max":{"1min":2.73,"5min":2.73,"15min":2.73},"last":0.872}]}]},{"osd":2,"up_from":18,"seq":77309411385,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27976,"kb_used_data":960,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717176,"statfs":{"total":95995035648,"available":95966388224,"internally_reserved":0,"allocated":983040,"data_stored":484809,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":27654411},"hb_peers":[0,1,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jul 14 19:29:16 2023","interfaces":[{"interface":"back","average":{"1min":0.58199999999999996,"5min":0.52200000000000002,"15min":0.498},"min":{"1min":0.34599999999999997,"5min":0.312,"15min":0.312},"max":{"1min":0.99199999999999999,"5min":0.99199999999999999,"15min":0.99199999999999999},"last":0.56599999999999995},{"interface":"front","average":{"1min":0.496,"5min":0.49299999999999999,"15min":0.48799999999999999},"min":{"1min":0.33600000000000002,"5min":0.31900000000000001,"15min":0.31900000000000001},"max":{"1min":0.66800000000000004,"5min":0.69699999999999995,"15min":0.69699999999999995},"last":0.53700000000000003}]},{"osd":1,"last update":"Fri Jul 14 19:29:16 2023","interfaces":[{"interface":"back","average":{"1min":0.626,"5min":0.55700000000000005,"15min":0.52300000000000002},"min":{"1min":0.42699999999999999,"5min":0.316,"15min":0.316},"max":{"1min":0.93100000000000005,"5min":0.93100000000000005,"15min":0.93100000000000005},"last":0.70399999999999996},{"interface":"front","average":{"1min":0.66600000000000004,"5min":0.57699999999999996,"15min":0.53300000000000003},"min":{"1min":0.38300000000000001,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":0.86499999999999999,"5min":0.91100000000000003,"15min":0.91100000000000003},"last":0.83299999999999996}]},{"osd":3,"last update":"Fri Jul 14 19:28:36 2023","interfaces":[{"interface":"back","average":{"1min":0.67900000000000005,"5min":0.57399999999999995,"15min":0.55400000000000005},"min":{"1min":0.48199999999999998,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":0.91700000000000004,"5min":0.91700000000000004,"15min":0.91700000000000004},"last":0.68300000000000005},{"interface":"front","average":{"1min":0.60299999999999998,"5min":0.54300000000000004,"15min":0.53300000000000003},"min":{"1min":0.44,"5min":0.34699999999999998,"15min":0.34699999999999998},"max":{"1min":0.76900000000000002,"5min":0.76900000000000002,"15min":0.76900000000000002},"last":0.62}]},{"osd":4,"last update":"Fri Jul 14 19:27:59 2023","interfaces":[{"interface":"back","average":{"1min":0.53800000000000003,"5min":0.52200000000000002,"15min":0.52000000000000002},"min":{"1min":0.38800000000000001,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":0.68400000000000005,"5min":0.70399999999999996,"15min":0.70399999999999996},"last":0.80300000000000005},{"interface":"front","average":{"1min":0.63600000000000001,"5min":0.58399999999999996,"15min":0.57599999999999996},"min":{"1min":0.40100000000000002,"5min":0.40100000000000002,"15min":0.40100000000000002},"max":{"1min":0.81699999999999995,"5min":0.81699999999999995,"15min":0.81699999999999995},"last":0.69399999999999995}]},{"osd":5,"last update":"Fri Jul 14 19:29:16 2023","interfaces":[{"interface":"back","average":{"1min":0.50600000000000001,"5min":0.44900000000000001,"15min":0.43099999999999999},"min":{"1min":0.33100000000000002,"5min":0.26500000000000001,"15min":0.26500000000000001},"max":{"1min":0.82199999999999995,"5min":0.82199999999999995,"15min":0.82199999999999995},"last":0.64200000000000002},{"interface":"front","average":{"1min":0.56799999999999995,"5min":0.499,"15min":0.47299999999999998},"min":{"1min":0.36199999999999999,"5min":0.313,"15min":0.313},"max":{"1min":0.84999999999999998,"5min":0.84999999999999998,"15min":0.84999999999999998},"last":0.60099999999999998}]},{"osd":6,"last update":"Fri Jul 14 19:28:43 2023","interfaces":[{"interface":"back","average":{"1min":0.59899999999999998,"5min":0.55500000000000005,"15min":0.54800000000000004},"min":{"1min":0.33500000000000002,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":0.93400000000000005,"5min":0.93400000000000005,"15min":0.93400000000000005},"last":0.498},{"interface":"front","average":{"1min":0.69199999999999995,"5min":0.60899999999999999,"15min":0.59499999999999997},"min":{"1min":0.46999999999999997,"5min":0.371,"15min":0.371},"max":{"1min":0.92000000000000004,"5min":0.92000000000000004,"15min":0.92000000000000004},"last":0.754}]},{"osd":7,"last update":"Fri Jul 14 19:27:59 2023","interfaces":[{"interface":"back","average":{"1min":0.622,"5min":0.622,"15min":0.622},"min":{"1min":0.44600000000000001,"5min":0.44600000000000001,"15min":0.44600000000000001},"max":{"1min":0.91900000000000004,"5min":0.91900000000000004,"15min":0.91900000000000004},"last":0.59899999999999998},{"interface":"front","average":{"1min":0.67500000000000004,"5min":0.67500000000000004,"15min":0.67500000000000004},"min":{"1min":0.433,"5min":0.433,"15min":0.433},"max":{"1min":0.96399999999999997,"5min":0.96399999999999997,"15min":0.96399999999999997},"last":0.66000000000000003}]},{"osd":8,"last update":"Fri Jul 14 19:29:16 2023","interfaces":[{"interface":"back","average":{"1min":0.63400000000000001,"5min":0.628,"15min":0.627},"min":{"1min":0.35499999999999998,"5min":0.35499999999999998,"15min":0.35499999999999998},"max":{"1min":0.91600000000000004,"5min":0.91600000000000004,"15min":0.91600000000000004},"last":0.71699999999999997},{"interface":"front","average":{"1min":0.60099999999999998,"5min":0.62,"15min":0.623},"min":{"1min":0.374,"5min":0.374,"15min":0.374},"max":{"1min":0.78600000000000003,"5min":0.94199999999999995,"15min":0.94199999999999995},"last":0.73899999999999999}]},{"osd":9,"last update":"Fri Jul 14 19:28:43 2023","interfaces":[{"interface":"back","average":{"1min":0.67800000000000005,"5min":0.67800000000000005,"15min":0.67800000000000005},"min":{"1min":0.45300000000000001,"5min":0.45300000000000001,"15min":0.45300000000000001},"max":{"1min":0.90700000000000003,"5min":0.90700000000000003,"15min":0.90700000000000003},"last":0.66600000000000004},{"interface":"front","average":{"1min":0.65300000000000002,"5min":0.65300000000000002,"15min":0.65300000000000002},"min":{"1min":0.55200000000000005,"5min":0.55200000000000005,"15min":0.55200000000000005},"max":{"1min":0.79700000000000004,"5min":0.79700000000000004,"15min":0.79700000000000004},"last":0.65900000000000003}]},{"osd":10,"last update":"Fri Jul 14 19:28:59 2023","interfaces":[{"interface":"back","average":{"1min":0.63200000000000001,"5min":0.63200000000000001,"15min":0.63200000000000001},"min":{"1min":0.46800000000000003,"5min":0.46800000000000003,"15min":0.46800000000000003},"max":{"1min":0.79700000000000004,"5min":0.79700000000000004,"15min":0.79700000000000004},"last":0.77100000000000002},{"interface":"front","average":{"1min":0.745,"5min":0.745,"15min":0.745},"min":{"1min":0.61899999999999999,"5min":0.61899999999999999,"15min":0.61899999999999999},"max":{"1min":0.88900000000000001,"5min":0.88900000000000001,"15min":0.88900000000000001},"last":0.88200000000000001}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.85899999999999999}]}]},{"osd":3,"up_from":25,"seq":107374182453,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27908,"kb_used_data":892,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717244,"statfs":{"total":95995035648,"available":95966457856,"internally_reserved":0,"allocated":913408,"data_stored":428656,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":27654411},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jul 14 19:29:36 2023","interfaces":[{"interface":"back","average":{"1min":0.35799999999999998,"5min":0.34899999999999998,"15min":0.32500000000000001},"min":{"1min":0.251,"5min":0.221,"15min":0.221},"max":{"1min":0.57599999999999996,"5min":0.61299999999999999,"15min":0.61299999999999999},"last":0.34599999999999997},{"interface":"front","average":{"1min":0.36499999999999999,"5min":0.378,"15min":0.35799999999999998},"min":{"1min":0.26600000000000001,"5min":0.23899999999999999,"15min":0.23899999999999999},"max":{"1min":0.55100000000000005,"5min":0.63600000000000001,"15min":0.63600000000000001},"last":0.29599999999999999}]},{"osd":1,"last update":"Fri Jul 14 19:29:36 2023","interfaces":[{"interface":"back","average":{"1min":0.56399999999999995,"5min":0.52400000000000002,"15min":0.499},"min":{"1min":0.34200000000000003,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":0.79600000000000004,"5min":0.84199999999999997,"15min":0.84199999999999997},"last":0.41099999999999998},{"interface":"front","average":{"1min":0.59299999999999997,"5min":0.54000000000000004,"15min":0.50900000000000001},"min":{"1min":0.40699999999999997,"5min":0.30399999999999999,"15min":0.30399999999999999},"max":{"1min":0.76200000000000001,"5min":0.76200000000000001,"15min":0.76200000000000001},"last":0.45300000000000001}]},{"osd":2,"last update":"Fri Jul 14 19:29:36 2023","interfaces":[{"interface":"back","average":{"1min":0.58299999999999996,"5min":0.57999999999999996,"15min":0.54500000000000004},"min":{"1min":0.40899999999999997,"5min":0.36199999999999999,"15min":0.36199999999999999},"max":{"1min":0.81299999999999994,"5min":0.93899999999999995,"15min":0.93899999999999995},"last":0.47099999999999997},{"interface":"front","average":{"1min":0.59599999999999997,"5min":0.56000000000000005,"15min":0.53200000000000003},"min":{"1min":0.33800000000000002,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":0.85399999999999998,"5min":0.85399999999999998,"15min":0.85399999999999998},"last":0.436}]},{"osd":4,"last update":"Fri Jul 14 19:28:51 2023","interfaces":[{"interface":"back","average":{"1min":0.66100000000000003,"5min":0.55600000000000005,"15min":0.52200000000000002},"min":{"1min":0.44800000000000001,"5min":0.32300000000000001,"15min":0.32300000000000001},"max":{"1min":0.91100000000000003,"5min":0.98499999999999999,"15min":0.98499999999999999},"last":0.59199999999999997},{"interface":"front","average":{"1min":0.60499999999999998,"5min":0.55000000000000004,"15min":0.52900000000000003},"min":{"1min":0.45500000000000002,"5min":0.39300000000000002,"15min":0.39300000000000002},"max":{"1min":0.90200000000000002,"5min":0.90200000000000002,"15min":0.90200000000000002},"last":0.54800000000000004}]},{"osd":5,"last update":"Fri Jul 14 19:29:13 2023","interfaces":[{"interface":"back","average":{"1min":0.56999999999999995,"5min":0.57999999999999996,"15min":0.57099999999999995},"min":{"1min":0.38200000000000001,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":0.79800000000000004,"5min":0.81299999999999994,"15min":0.81299999999999994},"last":0.48599999999999999},{"interface":"front","average":{"1min":0.67600000000000005,"5min":0.628,"15min":0.60199999999999998},"min":{"1min":0.44800000000000001,"5min":0.39300000000000002,"15min":0.39300000000000002},"max":{"1min":0.92200000000000004,"5min":0.98199999999999998,"15min":0.98199999999999998},"last":0.42399999999999999}]},{"osd":6,"last update":"Fri Jul 14 19:28:38 2023","interfaces":[{"interface":"back","average":{"1min":0.61299999999999999,"5min":0.501,"15min":0.48199999999999998},"min":{"1min":0.39100000000000001,"5min":0.34699999999999998,"15min":0.34699999999999998},"max":{"1min":0.82599999999999996,"5min":0.82599999999999996,"15min":0.82599999999999996},"last":0.39000000000000001},{"interface":"front","average":{"1min":0.51500000000000001,"5min":0.52000000000000002,"15min":0.52100000000000002},"min":{"1min":0.36499999999999999,"5min":0.36299999999999999,"15min":0.36299999999999999},"max":{"1min":0.67300000000000004,"5min":0.67300000000000004,"15min":0.67300000000000004},"last":0.497}]},{"osd":7,"last update":"Fri Jul 14 19:28:57 2023","interfaces":[{"interface":"back","average":{"1min":0.69699999999999995,"5min":0.68000000000000005,"15min":0.67700000000000005},"min":{"1min":0.48999999999999999,"5min":0.41699999999999998,"15min":0.41699999999999998},"max":{"1min":0.96399999999999997,"5min":1.196,"15min":1.196},"last":0.51100000000000001},{"interface":"front","average":{"1min":0.78700000000000003,"5min":0.69199999999999995,"15min":0.67600000000000005},"min":{"1min":0.58499999999999996,"5min":0.46100000000000002,"15min":0.46100000000000002},"max":{"1min":1.0289999999999999,"5min":1.0289999999999999,"15min":1.0289999999999999},"last":0.56999999999999995}]},{"osd":8,"last update":"Fri Jul 14 19:28:17 2023","interfaces":[{"interface":"back","average":{"1min":0.69899999999999995,"5min":0.69899999999999995,"15min":0.69899999999999995},"min":{"1min":0.48199999999999998,"5min":0.48199999999999998,"15min":0.48199999999999998},"max":{"1min":0.98799999999999999,"5min":0.98799999999999999,"15min":0.98799999999999999},"last":0.751},{"interface":"front","average":{"1min":0.749,"5min":0.749,"15min":0.749},"min":{"1min":0.51800000000000002,"5min":0.51800000000000002,"15min":0.51800000000000002},"max":{"1min":0.92500000000000004,"5min":0.92500000000000004,"15min":0.92500000000000004},"last":0.68100000000000005}]},{"osd":9,"last update":"Fri Jul 14 19:28:38 2023","interfaces":[{"interface":"back","average":{"1min":0.76600000000000001,"5min":0.76600000000000001,"15min":0.76600000000000001},"min":{"1min":0.42099999999999999,"5min":0.42099999999999999,"15min":0.42099999999999999},"max":{"1min":1.0549999999999999,"5min":1.0549999999999999,"15min":1.0549999999999999},"last":0.53400000000000003},{"interface":"front","average":{"1min":0.64900000000000002,"5min":0.64900000000000002,"15min":0.64900000000000002},"min":{"1min":0.40799999999999997,"5min":0.40799999999999997,"15min":0.40799999999999997},"max":{"1min":0.95099999999999996,"5min":0.95099999999999996,"15min":0.95099999999999996},"last":0.52200000000000002}]},{"osd":10,"last update":"Fri Jul 14 19:29:02 2023","interfaces":[{"interface":"back","average":{"1min":0.72699999999999998,"5min":0.72699999999999998,"15min":0.72699999999999998},"min":{"1min":0.56499999999999995,"5min":0.56499999999999995,"15min":0.56499999999999995},"max":{"1min":0.92700000000000005,"5min":0.92700000000000005,"15min":0.92700000000000005},"last":0.55900000000000005},{"interface":"front","average":{"1min":0.82099999999999995,"5min":0.82099999999999995,"15min":0.82099999999999995},"min":{"1min":0.59399999999999997,"5min":0.59399999999999997,"15min":0.59399999999999997},"max":{"1min":1.0880000000000001,"5min":1.0880000000000001,"15min":1.0880000000000001},"last":0.60999999999999999}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.58099999999999996}]}]},{"osd":4,"up_from":32,"seq":137438953521,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27900,"kb_used_data":884,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717252,"statfs":{"total":95995035648,"available":95966466048,"internally_reserved":0,"allocated":905216,"data_stored":429443,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":27654411},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jul 14 19:28:54 2023","interfaces":[{"interface":"back","average":{"1min":0.57599999999999996,"5min":0.52300000000000002,"15min":0.50900000000000001},"min":{"1min":0.39200000000000002,"5min":0.27400000000000002,"15min":0.27400000000000002},"max":{"1min":0.85299999999999998,"5min":0.85299999999999998,"15min":0.85299999999999998},"last":0.496},{"interface":"front","average":{"1min":0.56799999999999995,"5min":0.51700000000000002,"15min":0.497},"min":{"1min":0.35599999999999998,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":0.77600000000000002,"5min":0.90800000000000003,"15min":0.90800000000000003},"last":0.40699999999999997}]},{"osd":1,"last update":"Fri Jul 14 19:28:54 2023","interfaces":[{"interface":"back","average":{"1min":0.42799999999999999,"5min":0.378,"15min":0.35899999999999999},"min":{"1min":0.28699999999999998,"5min":0.224,"15min":0.224},"max":{"1min":0.77200000000000002,"5min":0.77200000000000002,"15min":0.77200000000000002},"last":0.39000000000000001},{"interface":"front","average":{"1min":0.48699999999999999,"5min":0.40500000000000003,"15min":0.38700000000000001},"min":{"1min":0.33800000000000002,"5min":0.247,"15min":0.247},"max":{"1min":0.72999999999999998,"5min":0.72999999999999998,"15min":0.72999999999999998},"last":0.42199999999999999}]},{"osd":2,"last update":"Fri Jul 14 19:28:54 2023","interfaces":[{"interface":"back","average":{"1min":0.61099999999999999,"5min":0.59799999999999998,"15min":0.59599999999999997},"min":{"1min":0.373,"5min":0.32800000000000001,"15min":0.32800000000000001},"max":{"1min":0.83799999999999997,"5min":0.86399999999999999,"15min":0.86399999999999999},"last":0.67000000000000004},{"interface":"front","average":{"1min":0.65200000000000002,"5min":0.59699999999999998,"15min":0.58299999999999996},"min":{"1min":0.43099999999999999,"5min":0.43099999999999999,"15min":0.43099999999999999},"max":{"1min":0.97099999999999997,"5min":0.97099999999999997,"15min":0.97099999999999997},"last":0.71599999999999997}]},{"osd":3,"last update":"Fri Jul 14 19:28:54 2023","interfaces":[{"interface":"back","average":{"1min":0.61299999999999999,"5min":0.55700000000000005,"15min":0.54400000000000004},"min":{"1min":0.442,"5min":0.36599999999999999,"15min":0.36599999999999999},"max":{"1min":0.98599999999999999,"5min":0.98599999999999999,"15min":0.98599999999999999},"last":0.46800000000000003},{"interface":"front","average":{"1min":0.59399999999999997,"5min":0.54600000000000004,"15min":0.53000000000000003},"min":{"1min":0.36399999999999999,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.876,"5min":0.876,"15min":0.876},"last":0.60699999999999998}]},{"osd":5,"last update":"Fri Jul 14 19:29:16 2023","interfaces":[{"interface":"back","average":{"1min":0.68700000000000006,"5min":0.61699999999999999,"15min":0.59899999999999998},"min":{"1min":0.55000000000000004,"5min":0.34699999999999998,"15min":0.34699999999999998},"max":{"1min":0.93500000000000005,"5min":0.93500000000000005,"15min":0.93500000000000005},"last":0.83299999999999996},{"interface":"front","average":{"1min":0.67600000000000005,"5min":0.60799999999999998,"15min":0.59099999999999997},"min":{"1min":0.436,"5min":0.314,"15min":0.314},"max":{"1min":0.92100000000000004,"5min":0.97099999999999997,"15min":0.97099999999999997},"last":0.64200000000000002}]},{"osd":6,"last update":"Fri Jul 14 19:29:38 2023","interfaces":[{"interface":"back","average":{"1min":0.61899999999999999,"5min":0.59799999999999998,"15min":0.58999999999999997},"min":{"1min":0.42199999999999999,"5min":0.378,"15min":0.378},"max":{"1min":0.89500000000000002,"5min":0.93000000000000005,"15min":0.93000000000000005},"last":0.53900000000000003},{"interface":"front","average":{"1min":0.60999999999999999,"5min":0.625,"15min":0.621},"min":{"1min":0.44900000000000001,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":0.76400000000000001,"5min":0.89500000000000002,"15min":0.89500000000000002},"last":0.58499999999999996}]},{"osd":7,"last update":"Fri Jul 14 19:28:59 2023","interfaces":[{"interface":"back","average":{"1min":0.61199999999999999,"5min":0.58299999999999996,"15min":0.57799999999999996},"min":{"1min":0.373,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":0.86499999999999999,"5min":0.89800000000000002,"15min":0.89800000000000002},"last":0.48199999999999998},{"interface":"front","average":{"1min":0.59199999999999997,"5min":0.55600000000000005,"15min":0.55000000000000004},"min":{"1min":0.38400000000000001,"5min":0.36199999999999999,"15min":0.36199999999999999},"max":{"1min":0.96899999999999997,"5min":0.96899999999999997,"15min":0.96899999999999997},"last":0.44800000000000001}]},{"osd":8,"last update":"Fri Jul 14 19:29:21 2023","interfaces":[{"interface":"back","average":{"1min":0.71999999999999997,"5min":0.67600000000000005,"15min":0.66800000000000004},"min":{"1min":0.51000000000000001,"5min":0.51000000000000001,"15min":0.51000000000000001},"max":{"1min":0.92500000000000004,"5min":0.92500000000000004,"15min":0.92500000000000004},"last":0.77100000000000002},{"interface":"front","average":{"1min":0.76100000000000001,"5min":0.71899999999999997,"15min":0.71199999999999997},"min":{"1min":0.622,"5min":0.49099999999999999,"15min":0.49099999999999999},"max":{"1min":0.86899999999999999,"5min":0.95899999999999996,"15min":0.95899999999999996},"last":0.78500000000000003}]},{"osd":9,"last update":"Fri Jul 14 19:29:38 2023","interfaces":[{"interface":"back","average":{"1min":0.68700000000000006,"5min":0.69499999999999995,"15min":0.69699999999999995},"min":{"1min":0.45200000000000001,"5min":0.45200000000000001,"15min":0.45200000000000001},"max":{"1min":1.036,"5min":1.036,"15min":1.036},"last":0.504},{"interface":"front","average":{"1min":0.751,"5min":0.74299999999999999,"15min":0.74199999999999999},"min":{"1min":0.53900000000000003,"5min":0.52000000000000002,"15min":0.52000000000000002},"max":{"1min":0.88,"5min":1.04,"15min":1.04},"last":0.52000000000000002}]},{"osd":10,"last update":"Fri Jul 14 19:28:59 2023","interfaces":[{"interface":"back","average":{"1min":0.68300000000000005,"5min":0.68300000000000005,"15min":0.68300000000000005},"min":{"1min":0.47299999999999998,"5min":0.47299999999999998,"15min":0.47299999999999998},"max":{"1min":0.95299999999999996,"5min":0.95299999999999996,"15min":0.95299999999999996},"last":0.755},{"interface":"front","average":{"1min":0.68000000000000005,"5min":0.68000000000000005,"15min":0.68000000000000005},"min":{"1min":0.443,"5min":0.443,"15min":0.443},"max":{"1min":0.995,"5min":0.995,"15min":0.995},"last":0.81499999999999995}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.80100000000000005}]}]},{"osd":5,"up_from":37,"seq":158913789997,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28500,"kb_used_data":1484,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716652,"statfs":{"total":95995035648,"available":95965851648,"internally_reserved":0,"allocated":1519616,"data_stored":1020766,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":27654411},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jul 14 19:29:18 2023","interfaces":[{"interface":"back","average":{"1min":0.51300000000000001,"5min":0.48599999999999999,"15min":0.46800000000000003},"min":{"1min":0.30499999999999999,"5min":0.28499999999999998,"15min":0.28499999999999998},"max":{"1min":0.75900000000000001,"5min":0.86399999999999999,"15min":0.86399999999999999},"last":0.40699999999999997},{"interface":"front","average":{"1min":0.48199999999999998,"5min":0.50700000000000001,"15min":0.502},"min":{"1min":0.32300000000000001,"5min":0.32300000000000001,"15min":0.32300000000000001},"max":{"1min":0.71899999999999997,"5min":0.84099999999999997,"15min":0.84099999999999997},"last":0.46600000000000003}]},{"osd":1,"last update":"Fri Jul 14 19:29:18 2023","interfaces":[{"interface":"back","average":{"1min":0.57699999999999996,"5min":0.55000000000000004,"15min":0.54000000000000004},"min":{"1min":0.36199999999999999,"5min":0.36199999999999999,"15min":0.36199999999999999},"max":{"1min":0.81599999999999995,"5min":1.0620000000000001,"15min":1.0620000000000001},"last":0.69199999999999995},{"interface":"front","average":{"1min":0.61499999999999999,"5min":0.56999999999999995,"15min":0.56000000000000005},"min":{"1min":0.48399999999999999,"5min":0.36099999999999999,"15min":0.36099999999999999},"max":{"1min":0.76900000000000002,"5min":0.93899999999999995,"15min":0.93899999999999995},"last":0.72099999999999997}]},{"osd":2,"last update":"Fri Jul 14 19:29:18 2023","interfaces":[{"interface":"back","average":{"1min":0.44800000000000001,"5min":0.443,"15min":0.44},"min":{"1min":0.28199999999999997,"5min":0.217,"15min":0.217},"max":{"1min":0.66500000000000004,"5min":0.67400000000000004,"15min":0.67400000000000004},"last":0.373},{"interface":"front","average":{"1min":0.441,"5min":0.45400000000000001,"15min":0.44900000000000001},"min":{"1min":0.30099999999999999,"5min":0.23699999999999999,"15min":0.23699999999999999},"max":{"1min":0.68700000000000006,"5min":0.82499999999999996,"15min":0.82499999999999996},"last":0.55000000000000004}]},{"osd":3,"last update":"Fri Jul 14 19:29:18 2023","interfaces":[{"interface":"back","average":{"1min":0.52900000000000003,"5min":0.56200000000000006,"15min":0.56699999999999995},"min":{"1min":0.39800000000000002,"5min":0.32200000000000001,"15min":0.32200000000000001},"max":{"1min":0.72199999999999998,"5min":0.90800000000000003,"15min":0.90800000000000003},"last":0.59199999999999997},{"interface":"front","average":{"1min":0.55000000000000004,"5min":0.57299999999999995,"15min":0.57099999999999995},"min":{"1min":0.39300000000000002,"5min":0.33900000000000002,"15min":0.33900000000000002},"max":{"1min":0.74399999999999999,"5min":0.94399999999999995,"15min":0.94399999999999995},"last":0.52600000000000002}]},{"osd":4,"last update":"Fri Jul 14 19:29:18 2023","interfaces":[{"interface":"back","average":{"1min":0.63800000000000001,"5min":0.61099999999999999,"15min":0.60099999999999998},"min":{"1min":0.43099999999999999,"5min":0.43099999999999999,"15min":0.43099999999999999},"max":{"1min":0.82899999999999996,"5min":0.86899999999999999,"15min":0.86899999999999999},"last":0.75800000000000001},{"interface":"front","average":{"1min":0.61199999999999999,"5min":0.59999999999999998,"15min":0.60299999999999998},"min":{"1min":0.38900000000000001,"5min":0.38,"15min":0.38},"max":{"1min":0.81399999999999995,"5min":0.81399999999999995,"15min":0.81399999999999995},"last":0.65600000000000003}]},{"osd":6,"last update":"Fri Jul 14 19:28:38 2023","interfaces":[{"interface":"back","average":{"1min":0.64900000000000002,"5min":0.60499999999999998,"15min":0.59699999999999998},"min":{"1min":0.443,"5min":0.38300000000000001,"15min":0.38300000000000001},"max":{"1min":0.85799999999999998,"5min":0.85799999999999998,"15min":0.85799999999999998},"last":0.57699999999999996},{"interface":"front","average":{"1min":0.70299999999999996,"5min":0.64500000000000002,"15min":0.63500000000000001},"min":{"1min":0.39900000000000002,"5min":0.36099999999999999,"15min":0.36099999999999999},"max":{"1min":1.0189999999999999,"5min":1.0189999999999999,"15min":1.0189999999999999},"last":0.63800000000000001}]},{"osd":7,"last update":"Fri Jul 14 19:28:59 2023","interfaces":[{"interface":"back","average":{"1min":0.65600000000000003,"5min":0.66400000000000003,"15min":0.66600000000000004},"min":{"1min":0.49099999999999999,"5min":0.49099999999999999,"15min":0.49099999999999999},"max":{"1min":0.88,"5min":0.95399999999999996,"15min":0.95399999999999996},"last":0.67800000000000005},{"interface":"front","average":{"1min":0.66800000000000004,"5min":0.68500000000000005,"15min":0.68799999999999994},"min":{"1min":0.46999999999999997,"5min":0.45800000000000002,"15min":0.45800000000000002},"max":{"1min":0.89500000000000002,"5min":0.96099999999999997,"15min":0.96099999999999997},"last":0.74199999999999999}]},{"osd":8,"last update":"Fri Jul 14 19:29:18 2023","interfaces":[{"interface":"back","average":{"1min":0.62,"5min":0.64500000000000002,"15min":0.64900000000000002},"min":{"1min":0.34200000000000003,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":0.93600000000000005,"5min":0.997,"15min":0.997},"last":0.62},{"interface":"front","average":{"1min":0.58699999999999997,"5min":0.60199999999999998,"15min":0.60399999999999998},"min":{"1min":0.36099999999999999,"5min":0.36099999999999999,"15min":0.36099999999999999},"max":{"1min":0.89600000000000002,"5min":0.89600000000000002,"15min":0.89600000000000002},"last":0.77100000000000002}]},{"osd":9,"last update":"Fri Jul 14 19:28:38 2023","interfaces":[{"interface":"back","average":{"1min":0.70299999999999996,"5min":0.70299999999999996,"15min":0.70299999999999996},"min":{"1min":0.50600000000000001,"5min":0.50600000000000001,"15min":0.50600000000000001},"max":{"1min":0.88700000000000001,"5min":0.88700000000000001,"15min":0.88700000000000001},"last":0.70799999999999996},{"interface":"front","average":{"1min":0.77800000000000002,"5min":0.77800000000000002,"15min":0.77800000000000002},"min":{"1min":0.504,"5min":0.504,"15min":0.504},"max":{"1min":1.111,"5min":1.111,"15min":1.111},"last":0.79500000000000004}]},{"osd":10,"last update":"Fri Jul 14 19:28:59 2023","interfaces":[{"interface":"back","average":{"1min":0.71699999999999997,"5min":0.71699999999999997,"15min":0.71699999999999997},"min":{"1min":0.46899999999999997,"5min":0.46899999999999997,"15min":0.46899999999999997},"max":{"1min":0.96599999999999997,"5min":0.96599999999999997,"15min":0.96599999999999997},"last":0.78200000000000003},{"interface":"front","average":{"1min":0.73199999999999998,"5min":0.73199999999999998,"15min":0.73199999999999998},"min":{"1min":0.54000000000000004,"5min":0.54000000000000004,"15min":0.54000000000000004},"max":{"1min":0.95099999999999996,"5min":0.95099999999999996,"15min":0.95099999999999996},"last":0.82499999999999996}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1478,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":54660,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":90112,"data_stored":56352,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":284,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":986,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1818,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":2476,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":738,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":818,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":98304,"data_stored":56580,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":246,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":123,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":123,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":123,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":123,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-07-14T19:29:46.002 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81c9387d2e7576da065747ec9c16d1c7d69016ec shell --fsid 9adea648-227b-11ee-9b33-001a4aab830c -- ceph pg dump --format=json 2023-07-14T19:29:46.555 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/9adea648-227b-11ee-9b33-001a4aab830c/mon.a/config 2023-07-14T19:29:46.917 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:29:46 smithi098 ceph-mon[127486]: from='client.14964 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-07-14T19:29:46.917 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:29:46 smithi098 ceph-mon[127486]: pgmap v275: 97 pgs: 97 active+clean; 634 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2023-07-14T19:29:46.941 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:29:46 smithi101 ceph-mon[128807]: from='client.14964 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-07-14T19:29:46.941 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:29:46 smithi101 ceph-mon[128807]: pgmap v275: 97 pgs: 97 active+clean; 634 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2023-07-14T19:29:46.970 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:29:46 smithi139 ceph-mon[129293]: from='client.14964 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-07-14T19:29:46.971 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:29:46 smithi139 ceph-mon[129293]: pgmap v275: 97 pgs: 97 active+clean; 634 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2023-07-14T19:29:47.608 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:47.607+0000 7fc8ff93f700 1 -- 172.21.15.98:0/3519360600 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc8f8103680 msgr2=0x7fc8f810b200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:29:47.608 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:47.607+0000 7fc8ff93f700 1 --2- 172.21.15.98:0/3519360600 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc8f8103680 0x7fc8f810b200 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7fc8f000b0a0 tx=0x7fc8f000b3b0 comp rx=0 tx=0).stop 2023-07-14T19:29:47.608 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:47.607+0000 7fc8ff93f700 1 -- 172.21.15.98:0/3519360600 shutdown_connections 2023-07-14T19:29:47.609 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:47.607+0000 7fc8ff93f700 1 --2- 172.21.15.98:0/3519360600 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc8f8103680 0x7fc8f810b200 unknown :-1 s=CLOSED pgs=291 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:29:47.609 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:47.607+0000 7fc8ff93f700 1 --2- 172.21.15.98:0/3519360600 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7fc8f80fe5e0 0x7fc8f80fe9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:29:47.609 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:47.607+0000 7fc8ff93f700 1 --2- 172.21.15.98:0/3519360600 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fc8f80fef80 0x7fc8f8102f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:29:47.609 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:47.607+0000 7fc8ff93f700 1 -- 172.21.15.98:0/3519360600 >> 172.21.15.98:0/3519360600 conn(0x7fc8f8075280 msgr2=0x7fc8f8075680 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:29:47.609 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:47.607+0000 7fc8ff93f700 1 -- 172.21.15.98:0/3519360600 shutdown_connections 2023-07-14T19:29:47.609 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:47.607+0000 7fc8ff93f700 1 -- 172.21.15.98:0/3519360600 wait complete. 2023-07-14T19:29:47.611 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:47.608+0000 7fc8ff93f700 1 Processor -- start 2023-07-14T19:29:47.611 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:47.608+0000 7fc8ff93f700 1 -- start start 2023-07-14T19:29:47.611 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:47.608+0000 7fc8ff93f700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc8f80fe5e0 0x7fc8f8195fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:29:47.612 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:47.608+0000 7fc8ff93f700 1 --2- >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7fc8f80fef80 0x7fc8f8196520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:29:47.612 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:47.608+0000 7fc8ff93f700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fc8f8103680 0x7fc8f819a990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:29:47.612 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:47.608+0000 7fc8ff93f700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc8f819afa0 con 0x7fc8f80fe5e0 2023-07-14T19:29:47.612 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:47.608+0000 7fc8ff93f700 1 -- --> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc8f819b110 con 0x7fc8f80fef80 2023-07-14T19:29:47.612 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:47.608+0000 7fc8ff93f700 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc8f819b2d0 con 0x7fc8f8103680 2023-07-14T19:29:47.612 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:47.608+0000 7fc8fd6db700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc8f80fe5e0 0x7fc8f8195fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:29:47.613 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:47.608+0000 7fc8fd6db700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc8f80fe5e0 0x7fc8f8195fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:46288/0 (socket says 172.21.15.98:46288) 2023-07-14T19:29:47.613 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:47.608+0000 7fc8fd6db700 1 -- 172.21.15.98:0/3805012271 learned_addr learned my addr 172.21.15.98:0/3805012271 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-07-14T19:29:47.613 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:47.608+0000 7fc8fdedc700 1 --2- 172.21.15.98:0/3805012271 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fc8f8103680 0x7fc8f819a990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:29:47.613 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:47.609+0000 7fc8fceda700 1 --2- 172.21.15.98:0/3805012271 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7fc8f80fef80 0x7fc8f8196520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:29:47.613 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:47.609+0000 7fc8fd6db700 1 -- 172.21.15.98:0/3805012271 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fc8f8103680 msgr2=0x7fc8f819a990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:29:47.613 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:47.609+0000 7fc8fd6db700 1 --2- 172.21.15.98:0/3805012271 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fc8f8103680 0x7fc8f819a990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:29:47.613 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:47.609+0000 7fc8fd6db700 1 -- 172.21.15.98:0/3805012271 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7fc8f80fef80 msgr2=0x7fc8f8196520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:29:47.613 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:47.609+0000 7fc8fd6db700 1 --2- 172.21.15.98:0/3805012271 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7fc8f80fef80 0x7fc8f8196520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:29:47.614 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:47.609+0000 7fc8fd6db700 1 -- 172.21.15.98:0/3805012271 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc8f0009d00 con 0x7fc8f80fe5e0 2023-07-14T19:29:47.614 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:47.609+0000 7fc8fd6db700 1 --2- 172.21.15.98:0/3805012271 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc8f80fe5e0 0x7fc8f8195fe0 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7fc8f400b530 tx=0x7fc8f400b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:29:47.614 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:47.609+0000 7fc8ee7fc700 1 -- 172.21.15.98:0/3805012271 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc8f4011830 con 0x7fc8f80fe5e0 2023-07-14T19:29:47.614 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:47.609+0000 7fc8ff93f700 1 -- 172.21.15.98:0/3805012271 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc8f819b5b0 con 0x7fc8f80fe5e0 2023-07-14T19:29:47.614 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:47.609+0000 7fc8ee7fc700 1 -- 172.21.15.98:0/3805012271 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fc8f4011e50 con 0x7fc8f80fe5e0 2023-07-14T19:29:47.614 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:47.609+0000 7fc8ee7fc700 1 -- 172.21.15.98:0/3805012271 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc8f400f2e0 con 0x7fc8f80fe5e0 2023-07-14T19:29:47.614 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:47.609+0000 7fc8ff93f700 1 -- 172.21.15.98:0/3805012271 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc8f81a2e10 con 0x7fc8f80fe5e0 2023-07-14T19:29:47.615 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:47.610+0000 7fc8ff93f700 1 -- 172.21.15.98:0/3805012271 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc8f804e340 con 0x7fc8f80fe5e0 2023-07-14T19:29:47.615 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:47.611+0000 7fc8ee7fc700 1 -- 172.21.15.98:0/3805012271 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 17) v1 ==== 135766+0+0 (secure 0 0 0) 0x7fc8f4010580 con 0x7fc8f80fe5e0 2023-07-14T19:29:47.616 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:47.611+0000 7fc8ee7fc700 1 --2- 172.21.15.98:0/3805012271 >> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] conn(0x7fc8e40a11e0 0x7fc8e40a3690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:29:47.616 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:47.612+0000 7fc8fceda700 1 --2- 172.21.15.98:0/3805012271 >> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] conn(0x7fc8e40a11e0 0x7fc8e40a3690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:29:47.616 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:47.612+0000 7fc8ee7fc700 1 -- 172.21.15.98:0/3805012271 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 7979+0+0 (secure 0 0 0) 0x7fc8f40cc310 con 0x7fc8f80fe5e0 2023-07-14T19:29:47.616 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:47.612+0000 7fc8fceda700 1 --2- 172.21.15.98:0/3805012271 >> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] conn(0x7fc8e40a11e0 0x7fc8e40a3690 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7fc8e8006fd0 tx=0x7fc8e8008040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:29:47.617 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:47.616+0000 7fc8ee7fc700 1 -- 172.21.15.98:0/3805012271 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7fc8f4095bc0 con 0x7fc8f80fe5e0 2023-07-14T19:29:47.798 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:47.798+0000 7fc8ff93f700 1 -- 172.21.15.98:0/3805012271 --> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fc8f8102640 con 0x7fc8e40a11e0 2023-07-14T19:29:47.808 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:47.808+0000 7fc8ee7fc700 1 -- 172.21.15.98:0/3805012271 <== mgr.14152 v2:172.21.15.98:6800/452042164 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+341250 (secure 0 0 0) 0x7fc8f8102640 con 0x7fc8e40a11e0 2023-07-14T19:29:47.811 INFO:teuthology.orchestra.run.smithi098.stdout: 2023-07-14T19:29:47.813 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:47.811+0000 7fc8ff93f700 1 -- 172.21.15.98:0/3805012271 >> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] conn(0x7fc8e40a11e0 msgr2=0x7fc8e40a3690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:29:47.813 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:47.811+0000 7fc8ff93f700 1 --2- 172.21.15.98:0/3805012271 >> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] conn(0x7fc8e40a11e0 0x7fc8e40a3690 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7fc8e8006fd0 tx=0x7fc8e8008040 comp rx=0 tx=0).stop 2023-07-14T19:29:47.813 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:47.811+0000 7fc8ff93f700 1 -- 172.21.15.98:0/3805012271 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc8f80fe5e0 msgr2=0x7fc8f8195fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:29:47.814 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:47.811+0000 7fc8ff93f700 1 --2- 172.21.15.98:0/3805012271 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc8f80fe5e0 0x7fc8f8195fe0 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7fc8f400b530 tx=0x7fc8f400b8f0 comp rx=0 tx=0).stop 2023-07-14T19:29:47.814 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:47.811+0000 7fc8ff93f700 1 -- 172.21.15.98:0/3805012271 shutdown_connections 2023-07-14T19:29:47.814 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:47.811+0000 7fc8ff93f700 1 --2- 172.21.15.98:0/3805012271 >> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] conn(0x7fc8e40a11e0 0x7fc8e40a3690 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:29:47.814 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:47.811+0000 7fc8ff93f700 1 --2- 172.21.15.98:0/3805012271 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fc8f8103680 0x7fc8f819a990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:29:47.814 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:47.811+0000 7fc8ff93f700 1 --2- 172.21.15.98:0/3805012271 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc8f80fe5e0 0x7fc8f8195fe0 unknown :-1 s=CLOSED pgs=292 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:29:47.814 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:47.811+0000 7fc8ff93f700 1 --2- 172.21.15.98:0/3805012271 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7fc8f80fef80 0x7fc8f8196520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:29:47.814 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:47.811+0000 7fc8ff93f700 1 -- 172.21.15.98:0/3805012271 >> 172.21.15.98:0/3805012271 conn(0x7fc8f8075280 msgr2=0x7fc8f8101f30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:29:47.814 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:47.811+0000 7fc8ff93f700 1 -- 172.21.15.98:0/3805012271 shutdown_connections 2023-07-14T19:29:47.815 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:47.811+0000 7fc8ff93f700 1 -- 172.21.15.98:0/3805012271 wait complete. 2023-07-14T19:29:47.815 INFO:teuthology.orchestra.run.smithi098.stderr:dumped all 2023-07-14T19:29:48.202 INFO:teuthology.orchestra.run.smithi098.stdout:{"pg_ready":true,"pg_map":{"version":276,"stamp":"2023-07-14T19:29:47.602895+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":649568,"num_objects":64,"num_object_clones":0,"num_object_copies":192,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":64,"num_whiteouts":0,"num_read":397,"num_read_kb":310,"num_write":332,"num_write_kb":3040,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":270,"ondisk_log_size":270,"up":291,"acting":291,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":291,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":336804,"kb_used_data":12612,"kb_used_omap":20,"kb_used_meta":324075,"kb_avail":1124605020,"statfs":{"total":1151940427776,"available":1151595540480,"internally_reserved":0,"allocated":12914688,"data_stored":7089245,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":21372,"internal_metadata":331852932},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":70,"num_read_kb":30,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002809"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":12,"reported_epoch":83,"state":"active+clean","last_fresh":"2023-07-14T19:29:02.573808+0000","last_change":"2023-07-14T19:29:01.313813+0000","last_active":"2023-07-14T19:29:02.573808+0000","last_peered":"2023-07-14T19:29:02.573808+0000","last_clean":"2023-07-14T19:29:02.573808+0000","last_became_active":"2023-07-14T19:29:01.313514+0000","last_became_peered":"2023-07-14T19:29:01.313514+0000","last_unstale":"2023-07-14T19:29:02.573808+0000","last_undegraded":"2023-07-14T19:29:02.573808+0000","last_fullsized":"2023-07-14T19:29:02.573808+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:29:00.604983+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:29:00.604983+0000","last_clean_scrub_stamp":"2023-07-14T19:29:00.604983+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T04:19:18.663334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":12,"reported_epoch":83,"state":"active+clean","last_fresh":"2023-07-14T19:29:02.603793+0000","last_change":"2023-07-14T19:29:01.316899+0000","last_active":"2023-07-14T19:29:02.603793+0000","last_peered":"2023-07-14T19:29:02.603793+0000","last_clean":"2023-07-14T19:29:02.603793+0000","last_became_active":"2023-07-14T19:29:01.316603+0000","last_became_peered":"2023-07-14T19:29:01.316603+0000","last_unstale":"2023-07-14T19:29:02.603793+0000","last_undegraded":"2023-07-14T19:29:02.603793+0000","last_fullsized":"2023-07-14T19:29:02.603793+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:29:00.604983+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:29:00.604983+0000","last_clean_scrub_stamp":"2023-07-14T19:29:00.604983+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T01:32:43.783686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":12,"reported_epoch":83,"state":"active+clean","last_fresh":"2023-07-14T19:29:02.603746+0000","last_change":"2023-07-14T19:29:01.316557+0000","last_active":"2023-07-14T19:29:02.603746+0000","last_peered":"2023-07-14T19:29:02.603746+0000","last_clean":"2023-07-14T19:29:02.603746+0000","last_became_active":"2023-07-14T19:29:01.316079+0000","last_became_peered":"2023-07-14T19:29:01.316079+0000","last_unstale":"2023-07-14T19:29:02.603746+0000","last_undegraded":"2023-07-14T19:29:02.603746+0000","last_fullsized":"2023-07-14T19:29:02.603746+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:29:00.604983+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:29:00.604983+0000","last_clean_scrub_stamp":"2023-07-14T19:29:00.604983+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T05:43:38.465282+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":12,"reported_epoch":83,"state":"active+clean","last_fresh":"2023-07-14T19:29:03.339617+0000","last_change":"2023-07-14T19:29:01.307992+0000","last_active":"2023-07-14T19:29:03.339617+0000","last_peered":"2023-07-14T19:29:03.339617+0000","last_clean":"2023-07-14T19:29:03.339617+0000","last_became_active":"2023-07-14T19:29:01.307687+0000","last_became_peered":"2023-07-14T19:29:01.307687+0000","last_unstale":"2023-07-14T19:29:03.339617+0000","last_undegraded":"2023-07-14T19:29:03.339617+0000","last_fullsized":"2023-07-14T19:29:03.339617+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:29:00.604983+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:29:00.604983+0000","last_clean_scrub_stamp":"2023-07-14T19:29:00.604983+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T03:04:56.480796+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":12,"reported_epoch":83,"state":"active+clean","last_fresh":"2023-07-14T19:29:02.660819+0000","last_change":"2023-07-14T19:29:01.311036+0000","last_active":"2023-07-14T19:29:02.660819+0000","last_peered":"2023-07-14T19:29:02.660819+0000","last_clean":"2023-07-14T19:29:02.660819+0000","last_became_active":"2023-07-14T19:29:01.310723+0000","last_became_peered":"2023-07-14T19:29:01.310723+0000","last_unstale":"2023-07-14T19:29:02.660819+0000","last_undegraded":"2023-07-14T19:29:02.660819+0000","last_fullsized":"2023-07-14T19:29:02.660819+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:29:00.604983+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:29:00.604983+0000","last_clean_scrub_stamp":"2023-07-14T19:29:00.604983+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T23:51:40.842901+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":12,"reported_epoch":83,"state":"active+clean","last_fresh":"2023-07-14T19:29:03.339712+0000","last_change":"2023-07-14T19:29:01.315246+0000","last_active":"2023-07-14T19:29:03.339712+0000","last_peered":"2023-07-14T19:29:03.339712+0000","last_clean":"2023-07-14T19:29:03.339712+0000","last_became_active":"2023-07-14T19:29:01.314885+0000","last_became_peered":"2023-07-14T19:29:01.314885+0000","last_unstale":"2023-07-14T19:29:03.339712+0000","last_undegraded":"2023-07-14T19:29:03.339712+0000","last_fullsized":"2023-07-14T19:29:03.339712+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:29:00.604983+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:29:00.604983+0000","last_clean_scrub_stamp":"2023-07-14T19:29:00.604983+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T04:22:25.924419+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":12,"reported_epoch":83,"state":"active+clean","last_fresh":"2023-07-14T19:29:03.338493+0000","last_change":"2023-07-14T19:29:01.306322+0000","last_active":"2023-07-14T19:29:03.338493+0000","last_peered":"2023-07-14T19:29:03.338493+0000","last_clean":"2023-07-14T19:29:03.338493+0000","last_became_active":"2023-07-14T19:29:01.306024+0000","last_became_peered":"2023-07-14T19:29:01.306024+0000","last_unstale":"2023-07-14T19:29:03.338493+0000","last_undegraded":"2023-07-14T19:29:03.338493+0000","last_fullsized":"2023-07-14T19:29:03.338493+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:29:00.604983+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:29:00.604983+0000","last_clean_scrub_stamp":"2023-07-14T19:29:00.604983+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T04:13:45.893986+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":12,"reported_epoch":83,"state":"active+clean","last_fresh":"2023-07-14T19:29:02.298956+0000","last_change":"2023-07-14T19:29:01.312514+0000","last_active":"2023-07-14T19:29:02.298956+0000","last_peered":"2023-07-14T19:29:02.298956+0000","last_clean":"2023-07-14T19:29:02.298956+0000","last_became_active":"2023-07-14T19:29:01.312260+0000","last_became_peered":"2023-07-14T19:29:01.312260+0000","last_unstale":"2023-07-14T19:29:02.298956+0000","last_undegraded":"2023-07-14T19:29:02.298956+0000","last_fullsized":"2023-07-14T19:29:02.298956+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:29:00.604983+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:29:00.604983+0000","last_clean_scrub_stamp":"2023-07-14T19:29:00.604983+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T21:31:03.757922+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":12,"reported_epoch":83,"state":"active+clean","last_fresh":"2023-07-14T19:29:02.300275+0000","last_change":"2023-07-14T19:29:01.312344+0000","last_active":"2023-07-14T19:29:02.300275+0000","last_peered":"2023-07-14T19:29:02.300275+0000","last_clean":"2023-07-14T19:29:02.300275+0000","last_became_active":"2023-07-14T19:29:01.311960+0000","last_became_peered":"2023-07-14T19:29:01.311960+0000","last_unstale":"2023-07-14T19:29:02.300275+0000","last_undegraded":"2023-07-14T19:29:02.300275+0000","last_fullsized":"2023-07-14T19:29:02.300275+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:29:00.604983+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:29:00.604983+0000","last_clean_scrub_stamp":"2023-07-14T19:29:00.604983+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T23:15:52.484949+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":12,"reported_epoch":83,"state":"active+clean","last_fresh":"2023-07-14T19:29:03.339513+0000","last_change":"2023-07-14T19:29:01.314868+0000","last_active":"2023-07-14T19:29:03.339513+0000","last_peered":"2023-07-14T19:29:03.339513+0000","last_clean":"2023-07-14T19:29:03.339513+0000","last_became_active":"2023-07-14T19:29:01.314617+0000","last_became_peered":"2023-07-14T19:29:01.314617+0000","last_unstale":"2023-07-14T19:29:03.339513+0000","last_undegraded":"2023-07-14T19:29:03.339513+0000","last_fullsized":"2023-07-14T19:29:03.339513+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:29:00.604983+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:29:00.604983+0000","last_clean_scrub_stamp":"2023-07-14T19:29:00.604983+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T20:33:35.726050+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":12,"reported_epoch":83,"state":"active+clean","last_fresh":"2023-07-14T19:29:03.339870+0000","last_change":"2023-07-14T19:29:01.319124+0000","last_active":"2023-07-14T19:29:03.339870+0000","last_peered":"2023-07-14T19:29:03.339870+0000","last_clean":"2023-07-14T19:29:03.339870+0000","last_became_active":"2023-07-14T19:29:01.318392+0000","last_became_peered":"2023-07-14T19:29:01.318392+0000","last_unstale":"2023-07-14T19:29:03.339870+0000","last_undegraded":"2023-07-14T19:29:03.339870+0000","last_fullsized":"2023-07-14T19:29:03.339870+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:29:00.604983+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:29:00.604983+0000","last_clean_scrub_stamp":"2023-07-14T19:29:00.604983+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T06:19:35.297680+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":12,"reported_epoch":83,"state":"active+clean","last_fresh":"2023-07-14T19:29:03.339129+0000","last_change":"2023-07-14T19:29:01.319003+0000","last_active":"2023-07-14T19:29:03.339129+0000","last_peered":"2023-07-14T19:29:03.339129+0000","last_clean":"2023-07-14T19:29:03.339129+0000","last_became_active":"2023-07-14T19:29:01.316550+0000","last_became_peered":"2023-07-14T19:29:01.316550+0000","last_unstale":"2023-07-14T19:29:03.339129+0000","last_undegraded":"2023-07-14T19:29:03.339129+0000","last_fullsized":"2023-07-14T19:29:03.339129+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:29:00.604983+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:29:00.604983+0000","last_clean_scrub_stamp":"2023-07-14T19:29:00.604983+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T22:06:18.517102+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":12,"reported_epoch":83,"state":"active+clean","last_fresh":"2023-07-14T19:29:02.444425+0000","last_change":"2023-07-14T19:29:01.318596+0000","last_active":"2023-07-14T19:29:02.444425+0000","last_peered":"2023-07-14T19:29:02.444425+0000","last_clean":"2023-07-14T19:29:02.444425+0000","last_became_active":"2023-07-14T19:29:01.318278+0000","last_became_peered":"2023-07-14T19:29:01.318278+0000","last_unstale":"2023-07-14T19:29:02.444425+0000","last_undegraded":"2023-07-14T19:29:02.444425+0000","last_fullsized":"2023-07-14T19:29:02.444425+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:29:00.604983+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:29:00.604983+0000","last_clean_scrub_stamp":"2023-07-14T19:29:00.604983+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T06:00:40.438416+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":12,"reported_epoch":83,"state":"active+clean","last_fresh":"2023-07-14T19:29:02.603553+0000","last_change":"2023-07-14T19:29:01.314344+0000","last_active":"2023-07-14T19:29:02.603553+0000","last_peered":"2023-07-14T19:29:02.603553+0000","last_clean":"2023-07-14T19:29:02.603553+0000","last_became_active":"2023-07-14T19:29:01.314054+0000","last_became_peered":"2023-07-14T19:29:01.314054+0000","last_unstale":"2023-07-14T19:29:02.603553+0000","last_undegraded":"2023-07-14T19:29:02.603553+0000","last_fullsized":"2023-07-14T19:29:02.603553+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:29:00.604983+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:29:00.604983+0000","last_clean_scrub_stamp":"2023-07-14T19:29:00.604983+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T23:03:20.122826+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":12,"reported_epoch":83,"state":"active+clean","last_fresh":"2023-07-14T19:29:02.603428+0000","last_change":"2023-07-14T19:29:01.314703+0000","last_active":"2023-07-14T19:29:02.603428+0000","last_peered":"2023-07-14T19:29:02.603428+0000","last_clean":"2023-07-14T19:29:02.603428+0000","last_became_active":"2023-07-14T19:29:01.314425+0000","last_became_peered":"2023-07-14T19:29:01.314425+0000","last_unstale":"2023-07-14T19:29:02.603428+0000","last_undegraded":"2023-07-14T19:29:02.603428+0000","last_fullsized":"2023-07-14T19:29:02.603428+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:29:00.604983+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:29:00.604983+0000","last_clean_scrub_stamp":"2023-07-14T19:29:00.604983+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T02:24:08.467559+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":12,"reported_epoch":83,"state":"active+clean","last_fresh":"2023-07-14T19:29:02.299763+0000","last_change":"2023-07-14T19:29:01.308769+0000","last_active":"2023-07-14T19:29:02.299763+0000","last_peered":"2023-07-14T19:29:02.299763+0000","last_clean":"2023-07-14T19:29:02.299763+0000","last_became_active":"2023-07-14T19:29:01.308450+0000","last_became_peered":"2023-07-14T19:29:01.308450+0000","last_unstale":"2023-07-14T19:29:02.299763+0000","last_undegraded":"2023-07-14T19:29:02.299763+0000","last_fullsized":"2023-07-14T19:29:02.299763+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:29:00.604983+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:29:00.604983+0000","last_clean_scrub_stamp":"2023-07-14T19:29:00.604983+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T06:36:10.772949+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":12,"reported_epoch":83,"state":"active+clean","last_fresh":"2023-07-14T19:29:02.573896+0000","last_change":"2023-07-14T19:29:01.313928+0000","last_active":"2023-07-14T19:29:02.573896+0000","last_peered":"2023-07-14T19:29:02.573896+0000","last_clean":"2023-07-14T19:29:02.573896+0000","last_became_active":"2023-07-14T19:29:01.313528+0000","last_became_peered":"2023-07-14T19:29:01.313528+0000","last_unstale":"2023-07-14T19:29:02.573896+0000","last_undegraded":"2023-07-14T19:29:02.573896+0000","last_fullsized":"2023-07-14T19:29:02.573896+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:29:00.604983+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:29:00.604983+0000","last_clean_scrub_stamp":"2023-07-14T19:29:00.604983+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T20:08:14.603380+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":12,"reported_epoch":83,"state":"active+clean","last_fresh":"2023-07-14T19:29:02.864370+0000","last_change":"2023-07-14T19:29:01.318903+0000","last_active":"2023-07-14T19:29:02.864370+0000","last_peered":"2023-07-14T19:29:02.864370+0000","last_clean":"2023-07-14T19:29:02.864370+0000","last_became_active":"2023-07-14T19:29:01.318610+0000","last_became_peered":"2023-07-14T19:29:01.318610+0000","last_unstale":"2023-07-14T19:29:02.864370+0000","last_undegraded":"2023-07-14T19:29:02.864370+0000","last_fullsized":"2023-07-14T19:29:02.864370+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:29:00.604983+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:29:00.604983+0000","last_clean_scrub_stamp":"2023-07-14T19:29:00.604983+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T20:01:45.469683+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":12,"reported_epoch":83,"state":"active+clean","last_fresh":"2023-07-14T19:29:03.222037+0000","last_change":"2023-07-14T19:29:01.315405+0000","last_active":"2023-07-14T19:29:03.222037+0000","last_peered":"2023-07-14T19:29:03.222037+0000","last_clean":"2023-07-14T19:29:03.222037+0000","last_became_active":"2023-07-14T19:29:01.315156+0000","last_became_peered":"2023-07-14T19:29:01.315156+0000","last_unstale":"2023-07-14T19:29:03.222037+0000","last_undegraded":"2023-07-14T19:29:03.222037+0000","last_fullsized":"2023-07-14T19:29:03.222037+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:29:00.604983+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:29:00.604983+0000","last_clean_scrub_stamp":"2023-07-14T19:29:00.604983+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T02:34:33.971469+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"83'9","reported_seq":24,"reported_epoch":83,"state":"active+clean","last_fresh":"2023-07-14T19:29:05.679207+0000","last_change":"2023-07-14T19:29:00.618385+0000","last_active":"2023-07-14T19:29:05.679207+0000","last_peered":"2023-07-14T19:29:05.679207+0000","last_clean":"2023-07-14T19:29:05.679207+0000","last_became_active":"2023-07-14T19:29:00.618081+0000","last_became_peered":"2023-07-14T19:29:00.618081+0000","last_unstale":"2023-07-14T19:29:05.679207+0000","last_undegraded":"2023-07-14T19:29:05.679207+0000","last_fullsized":"2023-07-14T19:29:05.679207+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:28:59.600128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:28:59.600128+0000","last_clean_scrub_stamp":"2023-07-14T19:28:59.600128+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T06:55:30.739187+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":12,"reported_epoch":83,"state":"active+clean","last_fresh":"2023-07-14T19:29:02.299334+0000","last_change":"2023-07-14T19:29:01.305197+0000","last_active":"2023-07-14T19:29:02.299334+0000","last_peered":"2023-07-14T19:29:02.299334+0000","last_clean":"2023-07-14T19:29:02.299334+0000","last_became_active":"2023-07-14T19:29:01.304902+0000","last_became_peered":"2023-07-14T19:29:01.304902+0000","last_unstale":"2023-07-14T19:29:02.299334+0000","last_undegraded":"2023-07-14T19:29:02.299334+0000","last_fullsized":"2023-07-14T19:29:02.299334+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:29:00.604983+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:29:00.604983+0000","last_clean_scrub_stamp":"2023-07-14T19:29:00.604983+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T20:54:29.407321+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1e","version":"83'2","reported_seq":16,"reported_epoch":83,"state":"active+clean","last_fresh":"2023-07-14T19:29:04.400749+0000","last_change":"2023-07-14T19:29:00.626316+0000","last_active":"2023-07-14T19:29:04.400749+0000","last_peered":"2023-07-14T19:29:04.400749+0000","last_clean":"2023-07-14T19:29:04.400749+0000","last_became_active":"2023-07-14T19:29:00.626035+0000","last_became_peered":"2023-07-14T19:29:00.626035+0000","last_unstale":"2023-07-14T19:29:04.400749+0000","last_undegraded":"2023-07-14T19:29:04.400749+0000","last_fullsized":"2023-07-14T19:29:04.400749+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:28:59.600128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:28:59.600128+0000","last_clean_scrub_stamp":"2023-07-14T19:28:59.600128+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T23:59:08.083433+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":12,"reported_epoch":83,"state":"active+clean","last_fresh":"2023-07-14T19:29:02.603883+0000","last_change":"2023-07-14T19:29:01.317118+0000","last_active":"2023-07-14T19:29:02.603883+0000","last_peered":"2023-07-14T19:29:02.603883+0000","last_clean":"2023-07-14T19:29:02.603883+0000","last_became_active":"2023-07-14T19:29:01.316805+0000","last_became_peered":"2023-07-14T19:29:01.316805+0000","last_unstale":"2023-07-14T19:29:02.603883+0000","last_undegraded":"2023-07-14T19:29:02.603883+0000","last_fullsized":"2023-07-14T19:29:02.603883+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:29:00.604983+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:29:00.604983+0000","last_clean_scrub_stamp":"2023-07-14T19:29:00.604983+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T04:47:32.634271+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1d","version":"83'4","reported_seq":18,"reported_epoch":83,"state":"active+clean","last_fresh":"2023-07-14T19:29:05.679368+0000","last_change":"2023-07-14T19:29:00.619435+0000","last_active":"2023-07-14T19:29:05.679368+0000","last_peered":"2023-07-14T19:29:05.679368+0000","last_clean":"2023-07-14T19:29:05.679368+0000","last_became_active":"2023-07-14T19:29:00.619131+0000","last_became_peered":"2023-07-14T19:29:00.619131+0000","last_unstale":"2023-07-14T19:29:05.679368+0000","last_undegraded":"2023-07-14T19:29:05.679368+0000","last_fullsized":"2023-07-14T19:29:05.679368+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:28:59.600128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:28:59.600128+0000","last_clean_scrub_stamp":"2023-07-14T19:28:59.600128+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T04:51:10.432113+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":12,"reported_epoch":83,"state":"active+clean","last_fresh":"2023-07-14T19:29:02.299884+0000","last_change":"2023-07-14T19:29:01.317947+0000","last_active":"2023-07-14T19:29:02.299884+0000","last_peered":"2023-07-14T19:29:02.299884+0000","last_clean":"2023-07-14T19:29:02.299884+0000","last_became_active":"2023-07-14T19:29:01.317633+0000","last_became_peered":"2023-07-14T19:29:01.317633+0000","last_unstale":"2023-07-14T19:29:02.299884+0000","last_undegraded":"2023-07-14T19:29:02.299884+0000","last_fullsized":"2023-07-14T19:29:02.299884+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:29:00.604983+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:29:00.604983+0000","last_clean_scrub_stamp":"2023-07-14T19:29:00.604983+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T00:25:43.200161+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1c","version":"82'2","reported_seq":16,"reported_epoch":83,"state":"active+clean","last_fresh":"2023-07-14T19:29:02.661063+0000","last_change":"2023-07-14T19:29:00.620401+0000","last_active":"2023-07-14T19:29:02.661063+0000","last_peered":"2023-07-14T19:29:02.661063+0000","last_clean":"2023-07-14T19:29:02.661063+0000","last_became_active":"2023-07-14T19:29:00.620134+0000","last_became_peered":"2023-07-14T19:29:00.620134+0000","last_unstale":"2023-07-14T19:29:02.661063+0000","last_undegraded":"2023-07-14T19:29:02.661063+0000","last_fullsized":"2023-07-14T19:29:02.661063+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:28:59.600128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:28:59.600128+0000","last_clean_scrub_stamp":"2023-07-14T19:28:59.600128+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T06:39:10.382383+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":12,"reported_epoch":83,"state":"active+clean","last_fresh":"2023-07-14T19:29:02.299819+0000","last_change":"2023-07-14T19:29:01.312174+0000","last_active":"2023-07-14T19:29:02.299819+0000","last_peered":"2023-07-14T19:29:02.299819+0000","last_clean":"2023-07-14T19:29:02.299819+0000","last_became_active":"2023-07-14T19:29:01.311891+0000","last_became_peered":"2023-07-14T19:29:01.311891+0000","last_unstale":"2023-07-14T19:29:02.299819+0000","last_undegraded":"2023-07-14T19:29:02.299819+0000","last_fullsized":"2023-07-14T19:29:02.299819+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:29:00.604983+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:29:00.604983+0000","last_clean_scrub_stamp":"2023-07-14T19:29:00.604983+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T19:35:58.889437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1b","version":"83'6","reported_seq":64,"reported_epoch":83,"state":"active+clean","last_fresh":"2023-07-14T19:29:45.366557+0000","last_change":"2023-07-14T19:29:00.622257+0000","last_active":"2023-07-14T19:29:45.366557+0000","last_peered":"2023-07-14T19:29:45.366557+0000","last_clean":"2023-07-14T19:29:45.366557+0000","last_became_active":"2023-07-14T19:29:00.621984+0000","last_became_peered":"2023-07-14T19:29:00.621984+0000","last_unstale":"2023-07-14T19:29:45.366557+0000","last_undegraded":"2023-07-14T19:29:45.366557+0000","last_fullsized":"2023-07-14T19:29:45.366557+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:28:59.600128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:28:59.600128+0000","last_clean_scrub_stamp":"2023-07-14T19:28:59.600128+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T01:01:52.888462+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":964,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":44,"num_read_kb":3,"num_write":3,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":12,"reported_epoch":83,"state":"active+clean","last_fresh":"2023-07-14T19:29:02.660873+0000","last_change":"2023-07-14T19:29:01.306274+0000","last_active":"2023-07-14T19:29:02.660873+0000","last_peered":"2023-07-14T19:29:02.660873+0000","last_clean":"2023-07-14T19:29:02.660873+0000","last_became_active":"2023-07-14T19:29:01.305863+0000","last_became_peered":"2023-07-14T19:29:01.305863+0000","last_unstale":"2023-07-14T19:29:02.660873+0000","last_undegraded":"2023-07-14T19:29:02.660873+0000","last_fullsized":"2023-07-14T19:29:02.660873+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:29:00.604983+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:29:00.604983+0000","last_clean_scrub_stamp":"2023-07-14T19:29:00.604983+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T00:42:02.531260+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.1a","version":"83'3","reported_seq":17,"reported_epoch":83,"state":"active+clean","last_fresh":"2023-07-14T19:29:05.678594+0000","last_change":"2023-07-14T19:29:00.619509+0000","last_active":"2023-07-14T19:29:05.678594+0000","last_peered":"2023-07-14T19:29:05.678594+0000","last_clean":"2023-07-14T19:29:05.678594+0000","last_became_active":"2023-07-14T19:29:00.619193+0000","last_became_peered":"2023-07-14T19:29:00.619193+0000","last_unstale":"2023-07-14T19:29:05.678594+0000","last_undegraded":"2023-07-14T19:29:05.678594+0000","last_fullsized":"2023-07-14T19:29:05.678594+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:28:59.600128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:28:59.600128+0000","last_clean_scrub_stamp":"2023-07-14T19:28:59.600128+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T01:07:45.846978+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":12,"reported_epoch":83,"state":"active+clean","last_fresh":"2023-07-14T19:29:02.299714+0000","last_change":"2023-07-14T19:29:01.307535+0000","last_active":"2023-07-14T19:29:02.299714+0000","last_peered":"2023-07-14T19:29:02.299714+0000","last_clean":"2023-07-14T19:29:02.299714+0000","last_became_active":"2023-07-14T19:29:01.307209+0000","last_became_peered":"2023-07-14T19:29:01.307209+0000","last_unstale":"2023-07-14T19:29:02.299714+0000","last_undegraded":"2023-07-14T19:29:02.299714+0000","last_fullsized":"2023-07-14T19:29:02.299714+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:29:00.604983+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:29:00.604983+0000","last_clean_scrub_stamp":"2023-07-14T19:29:00.604983+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T03:49:13.610038+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.19","version":"83'2","reported_seq":16,"reported_epoch":83,"state":"active+clean","last_fresh":"2023-07-14T19:29:05.678992+0000","last_change":"2023-07-14T19:29:00.620671+0000","last_active":"2023-07-14T19:29:05.678992+0000","last_peered":"2023-07-14T19:29:05.678992+0000","last_clean":"2023-07-14T19:29:05.678992+0000","last_became_active":"2023-07-14T19:29:00.619786+0000","last_became_peered":"2023-07-14T19:29:00.619786+0000","last_unstale":"2023-07-14T19:29:05.678992+0000","last_undegraded":"2023-07-14T19:29:05.678992+0000","last_fullsized":"2023-07-14T19:29:05.678992+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:28:59.600128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:28:59.600128+0000","last_clean_scrub_stamp":"2023-07-14T19:28:59.600128+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T06:56:01.025242+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":12,"reported_epoch":83,"state":"active+clean","last_fresh":"2023-07-14T19:29:02.298842+0000","last_change":"2023-07-14T19:29:01.316248+0000","last_active":"2023-07-14T19:29:02.298842+0000","last_peered":"2023-07-14T19:29:02.298842+0000","last_clean":"2023-07-14T19:29:02.298842+0000","last_became_active":"2023-07-14T19:29:01.315974+0000","last_became_peered":"2023-07-14T19:29:01.315974+0000","last_unstale":"2023-07-14T19:29:02.298842+0000","last_undegraded":"2023-07-14T19:29:02.298842+0000","last_fullsized":"2023-07-14T19:29:02.298842+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:29:00.604983+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:29:00.604983+0000","last_clean_scrub_stamp":"2023-07-14T19:29:00.604983+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T01:48:48.147093+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"83'1","reported_seq":15,"reported_epoch":83,"state":"active+clean","last_fresh":"2023-07-14T19:29:02.299408+0000","last_change":"2023-07-14T19:29:00.624210+0000","last_active":"2023-07-14T19:29:02.299408+0000","last_peered":"2023-07-14T19:29:02.299408+0000","last_clean":"2023-07-14T19:29:02.299408+0000","last_became_active":"2023-07-14T19:29:00.623918+0000","last_became_peered":"2023-07-14T19:29:00.623918+0000","last_unstale":"2023-07-14T19:29:02.299408+0000","last_undegraded":"2023-07-14T19:29:02.299408+0000","last_fullsized":"2023-07-14T19:29:02.299408+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:28:59.600128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:28:59.600128+0000","last_clean_scrub_stamp":"2023-07-14T19:28:59.600128+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T01:38:53.165557+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":12,"reported_epoch":83,"state":"active+clean","last_fresh":"2023-07-14T19:29:02.864241+0000","last_change":"2023-07-14T19:29:01.313787+0000","last_active":"2023-07-14T19:29:02.864241+0000","last_peered":"2023-07-14T19:29:02.864241+0000","last_clean":"2023-07-14T19:29:02.864241+0000","last_became_active":"2023-07-14T19:29:01.313479+0000","last_became_peered":"2023-07-14T19:29:01.313479+0000","last_unstale":"2023-07-14T19:29:02.864241+0000","last_undegraded":"2023-07-14T19:29:02.864241+0000","last_fullsized":"2023-07-14T19:29:02.864241+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:29:00.604983+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:29:00.604983+0000","last_clean_scrub_stamp":"2023-07-14T19:29:00.604983+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T22:33:21.348977+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.17","version":"83'3","reported_seq":55,"reported_epoch":83,"state":"active+clean","last_fresh":"2023-07-14T19:29:42.361276+0000","last_change":"2023-07-14T19:29:00.623053+0000","last_active":"2023-07-14T19:29:42.361276+0000","last_peered":"2023-07-14T19:29:42.361276+0000","last_clean":"2023-07-14T19:29:42.361276+0000","last_became_active":"2023-07-14T19:29:00.622760+0000","last_became_peered":"2023-07-14T19:29:00.622760+0000","last_unstale":"2023-07-14T19:29:42.361276+0000","last_undegraded":"2023-07-14T19:29:42.361276+0000","last_fullsized":"2023-07-14T19:29:42.361276+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:28:59.600128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:28:59.600128+0000","last_clean_scrub_stamp":"2023-07-14T19:28:59.600128+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T21:15:50.464654+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":38,"num_read_kb":38,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":12,"reported_epoch":83,"state":"active+clean","last_fresh":"2023-07-14T19:29:02.299982+0000","last_change":"2023-07-14T19:29:01.317509+0000","last_active":"2023-07-14T19:29:02.299982+0000","last_peered":"2023-07-14T19:29:02.299982+0000","last_clean":"2023-07-14T19:29:02.299982+0000","last_became_active":"2023-07-14T19:29:01.317238+0000","last_became_peered":"2023-07-14T19:29:01.317238+0000","last_unstale":"2023-07-14T19:29:02.299982+0000","last_undegraded":"2023-07-14T19:29:02.299982+0000","last_fullsized":"2023-07-14T19:29:02.299982+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:29:00.604983+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:29:00.604983+0000","last_clean_scrub_stamp":"2023-07-14T19:29:00.604983+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T04:59:01.463446+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.16","version":"83'2","reported_seq":16,"reported_epoch":83,"state":"active+clean","last_fresh":"2023-07-14T19:29:04.671803+0000","last_change":"2023-07-14T19:29:00.624024+0000","last_active":"2023-07-14T19:29:04.671803+0000","last_peered":"2023-07-14T19:29:04.671803+0000","last_clean":"2023-07-14T19:29:04.671803+0000","last_became_active":"2023-07-14T19:29:00.623638+0000","last_became_peered":"2023-07-14T19:29:00.623638+0000","last_unstale":"2023-07-14T19:29:04.671803+0000","last_undegraded":"2023-07-14T19:29:04.671803+0000","last_fullsized":"2023-07-14T19:29:04.671803+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:28:59.600128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:28:59.600128+0000","last_clean_scrub_stamp":"2023-07-14T19:28:59.600128+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T03:50:19.311702+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":12,"reported_epoch":83,"state":"active+clean","last_fresh":"2023-07-14T19:29:02.603804+0000","last_change":"2023-07-14T19:29:01.309164+0000","last_active":"2023-07-14T19:29:02.603804+0000","last_peered":"2023-07-14T19:29:02.603804+0000","last_clean":"2023-07-14T19:29:02.603804+0000","last_became_active":"2023-07-14T19:29:01.308797+0000","last_became_peered":"2023-07-14T19:29:01.308797+0000","last_unstale":"2023-07-14T19:29:02.603804+0000","last_undegraded":"2023-07-14T19:29:02.603804+0000","last_fullsized":"2023-07-14T19:29:02.603804+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:29:00.604983+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:29:00.604983+0000","last_clean_scrub_stamp":"2023-07-14T19:29:00.604983+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T23:09:54.074320+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"83'4","reported_seq":19,"reported_epoch":83,"state":"active+clean","last_fresh":"2023-07-14T19:29:04.671757+0000","last_change":"2023-07-14T19:29:00.625435+0000","last_active":"2023-07-14T19:29:04.671757+0000","last_peered":"2023-07-14T19:29:04.671757+0000","last_clean":"2023-07-14T19:29:04.671757+0000","last_became_active":"2023-07-14T19:29:00.625092+0000","last_became_peered":"2023-07-14T19:29:00.625092+0000","last_unstale":"2023-07-14T19:29:04.671757+0000","last_undegraded":"2023-07-14T19:29:04.671757+0000","last_fullsized":"2023-07-14T19:29:04.671757+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:28:59.600128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:28:59.600128+0000","last_clean_scrub_stamp":"2023-07-14T19:28:59.600128+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T02:44:25.187321+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":12,"reported_epoch":83,"state":"active+clean","last_fresh":"2023-07-14T19:29:02.300025+0000","last_change":"2023-07-14T19:29:01.315672+0000","last_active":"2023-07-14T19:29:02.300025+0000","last_peered":"2023-07-14T19:29:02.300025+0000","last_clean":"2023-07-14T19:29:02.300025+0000","last_became_active":"2023-07-14T19:29:01.315331+0000","last_became_peered":"2023-07-14T19:29:01.315331+0000","last_unstale":"2023-07-14T19:29:02.300025+0000","last_undegraded":"2023-07-14T19:29:02.300025+0000","last_fullsized":"2023-07-14T19:29:02.300025+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:29:00.604983+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:29:00.604983+0000","last_clean_scrub_stamp":"2023-07-14T19:29:00.604983+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T02:46:22.959698+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.14","version":"83'26","reported_seq":123,"reported_epoch":83,"state":"active+clean","last_fresh":"2023-07-14T19:29:43.362807+0000","last_change":"2023-07-14T19:29:00.622910+0000","last_active":"2023-07-14T19:29:43.362807+0000","last_peered":"2023-07-14T19:29:43.362807+0000","last_clean":"2023-07-14T19:29:43.362807+0000","last_became_active":"2023-07-14T19:29:00.622582+0000","last_became_peered":"2023-07-14T19:29:00.622582+0000","last_unstale":"2023-07-14T19:29:43.362807+0000","last_undegraded":"2023-07-14T19:29:43.362807+0000","last_fullsized":"2023-07-14T19:29:43.362807+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:28:59.600128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:28:59.600128+0000","last_clean_scrub_stamp":"2023-07-14T19:28:59.600128+0000","objects_scrubbed":0,"log_size":26,"log_dups_size":0,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T20:35:52.757984+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":54660,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":83,"num_read_kb":94,"num_write":27,"num_write_kb":62,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":12,"reported_epoch":83,"state":"active+clean","last_fresh":"2023-07-14T19:29:02.660955+0000","last_change":"2023-07-14T19:29:01.306169+0000","last_active":"2023-07-14T19:29:02.660955+0000","last_peered":"2023-07-14T19:29:02.660955+0000","last_clean":"2023-07-14T19:29:02.660955+0000","last_became_active":"2023-07-14T19:29:01.305830+0000","last_became_peered":"2023-07-14T19:29:01.305830+0000","last_unstale":"2023-07-14T19:29:02.660955+0000","last_undegraded":"2023-07-14T19:29:02.660955+0000","last_fullsized":"2023-07-14T19:29:02.660955+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:29:00.604983+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:29:00.604983+0000","last_clean_scrub_stamp":"2023-07-14T19:29:00.604983+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T00:12:18.670125+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.13","version":"83'6","reported_seq":21,"reported_epoch":83,"state":"active+clean","last_fresh":"2023-07-14T19:29:05.679614+0000","last_change":"2023-07-14T19:29:00.626500+0000","last_active":"2023-07-14T19:29:05.679614+0000","last_peered":"2023-07-14T19:29:05.679614+0000","last_clean":"2023-07-14T19:29:05.679614+0000","last_became_active":"2023-07-14T19:29:00.626130+0000","last_became_peered":"2023-07-14T19:29:00.626130+0000","last_unstale":"2023-07-14T19:29:05.679614+0000","last_undegraded":"2023-07-14T19:29:05.679614+0000","last_fullsized":"2023-07-14T19:29:05.679614+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:28:59.600128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:28:59.600128+0000","last_clean_scrub_stamp":"2023-07-14T19:28:59.600128+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T01:08:40.766374+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":12,"reported_epoch":83,"state":"active+clean","last_fresh":"2023-07-14T19:29:02.444400+0000","last_change":"2023-07-14T19:29:01.317902+0000","last_active":"2023-07-14T19:29:02.444400+0000","last_peered":"2023-07-14T19:29:02.444400+0000","last_clean":"2023-07-14T19:29:02.444400+0000","last_became_active":"2023-07-14T19:29:01.317587+0000","last_became_peered":"2023-07-14T19:29:01.317587+0000","last_unstale":"2023-07-14T19:29:02.444400+0000","last_undegraded":"2023-07-14T19:29:02.444400+0000","last_fullsized":"2023-07-14T19:29:02.444400+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:29:00.604983+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:29:00.604983+0000","last_clean_scrub_stamp":"2023-07-14T19:29:00.604983+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T03:39:39.441094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.12","version":"83'3","reported_seq":17,"reported_epoch":83,"state":"active+clean","last_fresh":"2023-07-14T19:29:04.671869+0000","last_change":"2023-07-14T19:29:00.626345+0000","last_active":"2023-07-14T19:29:04.671869+0000","last_peered":"2023-07-14T19:29:04.671869+0000","last_clean":"2023-07-14T19:29:04.671869+0000","last_became_active":"2023-07-14T19:29:00.626059+0000","last_became_peered":"2023-07-14T19:29:00.626059+0000","last_unstale":"2023-07-14T19:29:04.671869+0000","last_undegraded":"2023-07-14T19:29:04.671869+0000","last_fullsized":"2023-07-14T19:29:04.671869+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:28:59.600128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:28:59.600128+0000","last_clean_scrub_stamp":"2023-07-14T19:28:59.600128+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T21:18:49.525099+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":12,"reported_epoch":83,"state":"active+clean","last_fresh":"2023-07-14T19:29:03.339214+0000","last_change":"2023-07-14T19:29:01.310518+0000","last_active":"2023-07-14T19:29:03.339214+0000","last_peered":"2023-07-14T19:29:03.339214+0000","last_clean":"2023-07-14T19:29:03.339214+0000","last_became_active":"2023-07-14T19:29:01.310213+0000","last_became_peered":"2023-07-14T19:29:01.310213+0000","last_unstale":"2023-07-14T19:29:03.339214+0000","last_undegraded":"2023-07-14T19:29:03.339214+0000","last_fullsized":"2023-07-14T19:29:03.339214+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:29:00.604983+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:29:00.604983+0000","last_clean_scrub_stamp":"2023-07-14T19:29:00.604983+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T01:11:48.079392+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"83'4","reported_seq":61,"reported_epoch":83,"state":"active+clean","last_fresh":"2023-07-14T19:29:44.364268+0000","last_change":"2023-07-14T19:29:00.620450+0000","last_active":"2023-07-14T19:29:44.364268+0000","last_peered":"2023-07-14T19:29:44.364268+0000","last_clean":"2023-07-14T19:29:44.364268+0000","last_became_active":"2023-07-14T19:29:00.620100+0000","last_became_peered":"2023-07-14T19:29:00.620100+0000","last_unstale":"2023-07-14T19:29:44.364268+0000","last_undegraded":"2023-07-14T19:29:44.364268+0000","last_fullsized":"2023-07-14T19:29:44.364268+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:28:59.600128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:28:59.600128+0000","last_clean_scrub_stamp":"2023-07-14T19:28:59.600128+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T23:01:06.885919+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":930,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":44,"num_read_kb":4,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":12,"reported_epoch":83,"state":"active+clean","last_fresh":"2023-07-14T19:29:03.338482+0000","last_change":"2023-07-14T19:29:01.320612+0000","last_active":"2023-07-14T19:29:03.338482+0000","last_peered":"2023-07-14T19:29:03.338482+0000","last_clean":"2023-07-14T19:29:03.338482+0000","last_became_active":"2023-07-14T19:29:01.319569+0000","last_became_peered":"2023-07-14T19:29:01.319569+0000","last_unstale":"2023-07-14T19:29:03.338482+0000","last_undegraded":"2023-07-14T19:29:03.338482+0000","last_fullsized":"2023-07-14T19:29:03.338482+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:29:00.604983+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:29:00.604983+0000","last_clean_scrub_stamp":"2023-07-14T19:29:00.604983+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T05:46:45.094405+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.10","version":"83'15","reported_seq":30,"reported_epoch":83,"state":"active+clean","last_fresh":"2023-07-14T19:29:05.680022+0000","last_change":"2023-07-14T19:29:00.618619+0000","last_active":"2023-07-14T19:29:05.680022+0000","last_peered":"2023-07-14T19:29:05.680022+0000","last_clean":"2023-07-14T19:29:05.680022+0000","last_became_active":"2023-07-14T19:29:00.617735+0000","last_became_peered":"2023-07-14T19:29:00.617735+0000","last_unstale":"2023-07-14T19:29:05.680022+0000","last_undegraded":"2023-07-14T19:29:05.680022+0000","last_fullsized":"2023-07-14T19:29:05.680022+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:28:59.600128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:28:59.600128+0000","last_clean_scrub_stamp":"2023-07-14T19:28:59.600128+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T22:03:33.860671+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":12,"reported_epoch":83,"state":"active+clean","last_fresh":"2023-07-14T19:29:02.299414+0000","last_change":"2023-07-14T19:29:01.303053+0000","last_active":"2023-07-14T19:29:02.299414+0000","last_peered":"2023-07-14T19:29:02.299414+0000","last_clean":"2023-07-14T19:29:02.299414+0000","last_became_active":"2023-07-14T19:29:01.302679+0000","last_became_peered":"2023-07-14T19:29:01.302679+0000","last_unstale":"2023-07-14T19:29:02.299414+0000","last_undegraded":"2023-07-14T19:29:02.299414+0000","last_fullsized":"2023-07-14T19:29:02.299414+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:29:00.604983+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:29:00.604983+0000","last_clean_scrub_stamp":"2023-07-14T19:29:00.604983+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T03:30:02.623040+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"83'2","reported_seq":17,"reported_epoch":83,"state":"active+clean","last_fresh":"2023-07-14T19:29:06.299906+0000","last_change":"2023-07-14T19:29:00.622494+0000","last_active":"2023-07-14T19:29:06.299906+0000","last_peered":"2023-07-14T19:29:06.299906+0000","last_clean":"2023-07-14T19:29:06.299906+0000","last_became_active":"2023-07-14T19:29:00.622195+0000","last_became_peered":"2023-07-14T19:29:00.622195+0000","last_unstale":"2023-07-14T19:29:06.299906+0000","last_undegraded":"2023-07-14T19:29:06.299906+0000","last_fullsized":"2023-07-14T19:29:06.299906+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:28:59.600128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:28:59.600128+0000","last_clean_scrub_stamp":"2023-07-14T19:28:59.600128+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T01:34:47.084212+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":12,"reported_epoch":83,"state":"active+clean","last_fresh":"2023-07-14T19:29:02.573720+0000","last_change":"2023-07-14T19:29:01.319375+0000","last_active":"2023-07-14T19:29:02.573720+0000","last_peered":"2023-07-14T19:29:02.573720+0000","last_clean":"2023-07-14T19:29:02.573720+0000","last_became_active":"2023-07-14T19:29:01.319068+0000","last_became_peered":"2023-07-14T19:29:01.319068+0000","last_unstale":"2023-07-14T19:29:02.573720+0000","last_undegraded":"2023-07-14T19:29:02.573720+0000","last_fullsized":"2023-07-14T19:29:02.573720+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:29:00.604983+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:29:00.604983+0000","last_clean_scrub_stamp":"2023-07-14T19:29:00.604983+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T06:31:40.249974+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":14,"reported_epoch":83,"state":"active+clean","last_fresh":"2023-07-14T19:29:02.864257+0000","last_change":"2023-07-14T19:29:00.622364+0000","last_active":"2023-07-14T19:29:02.864257+0000","last_peered":"2023-07-14T19:29:02.864257+0000","last_clean":"2023-07-14T19:29:02.864257+0000","last_became_active":"2023-07-14T19:29:00.622073+0000","last_became_peered":"2023-07-14T19:29:00.622073+0000","last_unstale":"2023-07-14T19:29:02.864257+0000","last_undegraded":"2023-07-14T19:29:02.864257+0000","last_fullsized":"2023-07-14T19:29:02.864257+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:28:59.600128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:28:59.600128+0000","last_clean_scrub_stamp":"2023-07-14T19:28:59.600128+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T20:41:09.339550+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"83'1","reported_seq":13,"reported_epoch":83,"state":"active+clean","last_fresh":"2023-07-14T19:29:03.934768+0000","last_change":"2023-07-14T19:29:01.314076+0000","last_active":"2023-07-14T19:29:03.934768+0000","last_peered":"2023-07-14T19:29:03.934768+0000","last_clean":"2023-07-14T19:29:03.934768+0000","last_became_active":"2023-07-14T19:29:01.313769+0000","last_became_peered":"2023-07-14T19:29:01.313769+0000","last_unstale":"2023-07-14T19:29:03.934768+0000","last_undegraded":"2023-07-14T19:29:03.934768+0000","last_fullsized":"2023-07-14T19:29:03.934768+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:29:00.604983+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:29:00.604983+0000","last_clean_scrub_stamp":"2023-07-14T19:29:00.604983+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T02:26:15.354399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":123,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.2","version":"83'1","reported_seq":15,"reported_epoch":83,"state":"active+clean","last_fresh":"2023-07-14T19:29:03.338943+0000","last_change":"2023-07-14T19:29:00.622928+0000","last_active":"2023-07-14T19:29:03.338943+0000","last_peered":"2023-07-14T19:29:03.338943+0000","last_clean":"2023-07-14T19:29:03.338943+0000","last_became_active":"2023-07-14T19:29:00.622628+0000","last_became_peered":"2023-07-14T19:29:00.622628+0000","last_unstale":"2023-07-14T19:29:03.338943+0000","last_undegraded":"2023-07-14T19:29:03.338943+0000","last_fullsized":"2023-07-14T19:29:03.338943+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:28:59.600128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:28:59.600128+0000","last_clean_scrub_stamp":"2023-07-14T19:28:59.600128+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T21:41:30.895447+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":12,"reported_epoch":83,"state":"active+clean","last_fresh":"2023-07-14T19:29:03.339904+0000","last_change":"2023-07-14T19:29:01.322519+0000","last_active":"2023-07-14T19:29:03.339904+0000","last_peered":"2023-07-14T19:29:03.339904+0000","last_clean":"2023-07-14T19:29:03.339904+0000","last_became_active":"2023-07-14T19:29:01.322233+0000","last_became_peered":"2023-07-14T19:29:01.322233+0000","last_unstale":"2023-07-14T19:29:03.339904+0000","last_undegraded":"2023-07-14T19:29:03.339904+0000","last_fullsized":"2023-07-14T19:29:03.339904+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:29:00.604983+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:29:00.604983+0000","last_clean_scrub_stamp":"2023-07-14T19:29:00.604983+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T23:32:02.997469+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":12,"reported_epoch":83,"state":"active+clean","last_fresh":"2023-07-14T19:29:02.299512+0000","last_change":"2023-07-14T19:29:01.305439+0000","last_active":"2023-07-14T19:29:02.299512+0000","last_peered":"2023-07-14T19:29:02.299512+0000","last_clean":"2023-07-14T19:29:02.299512+0000","last_became_active":"2023-07-14T19:29:01.305060+0000","last_became_peered":"2023-07-14T19:29:01.305060+0000","last_unstale":"2023-07-14T19:29:02.299512+0000","last_undegraded":"2023-07-14T19:29:02.299512+0000","last_fullsized":"2023-07-14T19:29:02.299512+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:29:00.604983+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:29:00.604983+0000","last_clean_scrub_stamp":"2023-07-14T19:29:00.604983+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T06:07:47.229804+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"83'4","reported_seq":57,"reported_epoch":83,"state":"active+clean","last_fresh":"2023-07-14T19:29:43.361998+0000","last_change":"2023-07-14T19:29:00.620867+0000","last_active":"2023-07-14T19:29:43.361998+0000","last_peered":"2023-07-14T19:29:43.361998+0000","last_clean":"2023-07-14T19:29:43.361998+0000","last_became_active":"2023-07-14T19:29:00.620370+0000","last_became_peered":"2023-07-14T19:29:00.620370+0000","last_unstale":"2023-07-14T19:29:43.361998+0000","last_undegraded":"2023-07-14T19:29:43.361998+0000","last_fullsized":"2023-07-14T19:29:43.361998+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:28:59.600128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:28:59.600128+0000","last_clean_scrub_stamp":"2023-07-14T19:28:59.600128+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T05:53:10.376311+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":39,"num_read_kb":39,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":12,"reported_epoch":83,"state":"active+clean","last_fresh":"2023-07-14T19:29:02.299185+0000","last_change":"2023-07-14T19:29:01.306289+0000","last_active":"2023-07-14T19:29:02.299185+0000","last_peered":"2023-07-14T19:29:02.299185+0000","last_clean":"2023-07-14T19:29:02.299185+0000","last_became_active":"2023-07-14T19:29:01.305987+0000","last_became_peered":"2023-07-14T19:29:01.305987+0000","last_unstale":"2023-07-14T19:29:02.299185+0000","last_undegraded":"2023-07-14T19:29:02.299185+0000","last_fullsized":"2023-07-14T19:29:02.299185+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:29:00.604983+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:29:00.604983+0000","last_clean_scrub_stamp":"2023-07-14T19:29:00.604983+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T07:21:20.067549+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"83'3","reported_seq":19,"reported_epoch":83,"state":"active+clean","last_fresh":"2023-07-14T19:29:06.299569+0000","last_change":"2023-07-14T19:29:00.620937+0000","last_active":"2023-07-14T19:29:06.299569+0000","last_peered":"2023-07-14T19:29:06.299569+0000","last_clean":"2023-07-14T19:29:06.299569+0000","last_became_active":"2023-07-14T19:29:00.620616+0000","last_became_peered":"2023-07-14T19:29:00.620616+0000","last_unstale":"2023-07-14T19:29:06.299569+0000","last_undegraded":"2023-07-14T19:29:06.299569+0000","last_fullsized":"2023-07-14T19:29:06.299569+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:28:59.600128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:28:59.600128+0000","last_clean_scrub_stamp":"2023-07-14T19:28:59.600128+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T20:32:12.990940+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":4,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"22'137","reported_seq":341,"reported_epoch":83,"state":"active+clean","last_fresh":"2023-07-14T19:29:03.338586+0000","last_change":"2023-07-14T19:28:01.948805+0000","last_active":"2023-07-14T19:29:03.338586+0000","last_peered":"2023-07-14T19:29:03.338586+0000","last_clean":"2023-07-14T19:29:03.338586+0000","last_became_active":"2023-07-14T19:28:01.948295+0000","last_became_peered":"2023-07-14T19:28:01.948295+0000","last_unstale":"2023-07-14T19:29:03.338586+0000","last_undegraded":"2023-07-14T19:29:03.338586+0000","last_fullsized":"2023-07-14T19:29:03.338586+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:25:10.240820+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:25:10.240820+0000","last_clean_scrub_stamp":"2023-07-14T19:25:10.240820+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T23:39:19.337174+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":12,"reported_epoch":83,"state":"active+clean","last_fresh":"2023-07-14T19:29:02.444463+0000","last_change":"2023-07-14T19:29:01.321025+0000","last_active":"2023-07-14T19:29:02.444463+0000","last_peered":"2023-07-14T19:29:02.444463+0000","last_clean":"2023-07-14T19:29:02.444463+0000","last_became_active":"2023-07-14T19:29:01.320787+0000","last_became_peered":"2023-07-14T19:29:01.320787+0000","last_unstale":"2023-07-14T19:29:02.444463+0000","last_undegraded":"2023-07-14T19:29:02.444463+0000","last_fullsized":"2023-07-14T19:29:02.444463+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:29:00.604983+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:29:00.604983+0000","last_clean_scrub_stamp":"2023-07-14T19:29:00.604983+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T02:26:20.883377+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"83'3","reported_seq":17,"reported_epoch":83,"state":"active+clean","last_fresh":"2023-07-14T19:29:05.679444+0000","last_change":"2023-07-14T19:29:00.623891+0000","last_active":"2023-07-14T19:29:05.679444+0000","last_peered":"2023-07-14T19:29:05.679444+0000","last_clean":"2023-07-14T19:29:05.679444+0000","last_became_active":"2023-07-14T19:29:00.623562+0000","last_became_peered":"2023-07-14T19:29:00.623562+0000","last_unstale":"2023-07-14T19:29:05.679444+0000","last_undegraded":"2023-07-14T19:29:05.679444+0000","last_fullsized":"2023-07-14T19:29:05.679444+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:28:59.600128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:28:59.600128+0000","last_clean_scrub_stamp":"2023-07-14T19:28:59.600128+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T19:35:24.864363+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.5","version":"83'1","reported_seq":16,"reported_epoch":83,"state":"active+clean","last_fresh":"2023-07-14T19:29:04.670790+0000","last_change":"2023-07-14T19:29:00.620282+0000","last_active":"2023-07-14T19:29:04.670790+0000","last_peered":"2023-07-14T19:29:04.670790+0000","last_clean":"2023-07-14T19:29:04.670790+0000","last_became_active":"2023-07-14T19:29:00.620005+0000","last_became_peered":"2023-07-14T19:29:00.620005+0000","last_unstale":"2023-07-14T19:29:04.670790+0000","last_undegraded":"2023-07-14T19:29:04.670790+0000","last_fullsized":"2023-07-14T19:29:04.670790+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:28:59.600128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:28:59.600128+0000","last_clean_scrub_stamp":"2023-07-14T19:28:59.600128+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T03:20:25.333743+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":12,"reported_epoch":83,"state":"active+clean","last_fresh":"2023-07-14T19:29:02.444350+0000","last_change":"2023-07-14T19:29:01.311708+0000","last_active":"2023-07-14T19:29:02.444350+0000","last_peered":"2023-07-14T19:29:02.444350+0000","last_clean":"2023-07-14T19:29:02.444350+0000","last_became_active":"2023-07-14T19:29:01.311400+0000","last_became_peered":"2023-07-14T19:29:01.311400+0000","last_unstale":"2023-07-14T19:29:02.444350+0000","last_undegraded":"2023-07-14T19:29:02.444350+0000","last_fullsized":"2023-07-14T19:29:02.444350+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:29:00.604983+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:29:00.604983+0000","last_clean_scrub_stamp":"2023-07-14T19:29:00.604983+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T23:01:26.499242+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.6","version":"83'5","reported_seq":19,"reported_epoch":83,"state":"active+clean","last_fresh":"2023-07-14T19:29:10.296968+0000","last_change":"2023-07-14T19:29:00.617411+0000","last_active":"2023-07-14T19:29:10.296968+0000","last_peered":"2023-07-14T19:29:10.296968+0000","last_clean":"2023-07-14T19:29:10.296968+0000","last_became_active":"2023-07-14T19:29:00.617078+0000","last_became_peered":"2023-07-14T19:29:00.617078+0000","last_unstale":"2023-07-14T19:29:10.296968+0000","last_undegraded":"2023-07-14T19:29:10.296968+0000","last_fullsized":"2023-07-14T19:29:10.296968+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:28:59.600128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:28:59.600128+0000","last_clean_scrub_stamp":"2023-07-14T19:28:59.600128+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T21:38:43.985609+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":12,"reported_epoch":83,"state":"active+clean","last_fresh":"2023-07-14T19:29:02.299351+0000","last_change":"2023-07-14T19:29:01.307282+0000","last_active":"2023-07-14T19:29:02.299351+0000","last_peered":"2023-07-14T19:29:02.299351+0000","last_clean":"2023-07-14T19:29:02.299351+0000","last_became_active":"2023-07-14T19:29:01.306966+0000","last_became_peered":"2023-07-14T19:29:01.306966+0000","last_unstale":"2023-07-14T19:29:02.299351+0000","last_undegraded":"2023-07-14T19:29:02.299351+0000","last_fullsized":"2023-07-14T19:29:02.299351+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:29:00.604983+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:29:00.604983+0000","last_clean_scrub_stamp":"2023-07-14T19:29:00.604983+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T23:06:00.918796+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"83'4","reported_seq":18,"reported_epoch":83,"state":"active+clean","last_fresh":"2023-07-14T19:29:04.671123+0000","last_change":"2023-07-14T19:29:00.624599+0000","last_active":"2023-07-14T19:29:04.671123+0000","last_peered":"2023-07-14T19:29:04.671123+0000","last_clean":"2023-07-14T19:29:04.671123+0000","last_became_active":"2023-07-14T19:29:00.624272+0000","last_became_peered":"2023-07-14T19:29:00.624272+0000","last_unstale":"2023-07-14T19:29:04.671123+0000","last_undegraded":"2023-07-14T19:29:04.671123+0000","last_fullsized":"2023-07-14T19:29:04.671123+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:28:59.600128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:28:59.600128+0000","last_clean_scrub_stamp":"2023-07-14T19:28:59.600128+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T20:00:31.406685+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":12,"reported_epoch":83,"state":"active+clean","last_fresh":"2023-07-14T19:29:02.603663+0000","last_change":"2023-07-14T19:29:01.309411+0000","last_active":"2023-07-14T19:29:02.603663+0000","last_peered":"2023-07-14T19:29:02.603663+0000","last_clean":"2023-07-14T19:29:02.603663+0000","last_became_active":"2023-07-14T19:29:01.309113+0000","last_became_peered":"2023-07-14T19:29:01.309113+0000","last_unstale":"2023-07-14T19:29:02.603663+0000","last_undegraded":"2023-07-14T19:29:02.603663+0000","last_fullsized":"2023-07-14T19:29:02.603663+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:29:00.604983+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:29:00.604983+0000","last_clean_scrub_stamp":"2023-07-14T19:29:00.604983+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T07:09:16.642605+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"83'2","reported_seq":16,"reported_epoch":83,"state":"active+clean","last_fresh":"2023-07-14T19:29:04.671782+0000","last_change":"2023-07-14T19:29:00.622758+0000","last_active":"2023-07-14T19:29:04.671782+0000","last_peered":"2023-07-14T19:29:04.671782+0000","last_clean":"2023-07-14T19:29:04.671782+0000","last_became_active":"2023-07-14T19:29:00.622460+0000","last_became_peered":"2023-07-14T19:29:00.622460+0000","last_unstale":"2023-07-14T19:29:04.671782+0000","last_undegraded":"2023-07-14T19:29:04.671782+0000","last_fullsized":"2023-07-14T19:29:04.671782+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:28:59.600128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:28:59.600128+0000","last_clean_scrub_stamp":"2023-07-14T19:28:59.600128+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T23:30:15.407153+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":12,"reported_epoch":83,"state":"active+clean","last_fresh":"2023-07-14T19:29:02.660717+0000","last_change":"2023-07-14T19:29:01.317046+0000","last_active":"2023-07-14T19:29:02.660717+0000","last_peered":"2023-07-14T19:29:02.660717+0000","last_clean":"2023-07-14T19:29:02.660717+0000","last_became_active":"2023-07-14T19:29:01.316653+0000","last_became_peered":"2023-07-14T19:29:01.316653+0000","last_unstale":"2023-07-14T19:29:02.660717+0000","last_undegraded":"2023-07-14T19:29:02.660717+0000","last_fullsized":"2023-07-14T19:29:02.660717+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:29:00.604983+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:29:00.604983+0000","last_clean_scrub_stamp":"2023-07-14T19:29:00.604983+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T04:50:19.726545+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.9","version":"83'3","reported_seq":17,"reported_epoch":83,"state":"active+clean","last_fresh":"2023-07-14T19:29:04.670955+0000","last_change":"2023-07-14T19:29:00.619474+0000","last_active":"2023-07-14T19:29:04.670955+0000","last_peered":"2023-07-14T19:29:04.670955+0000","last_clean":"2023-07-14T19:29:04.670955+0000","last_became_active":"2023-07-14T19:29:00.619240+0000","last_became_peered":"2023-07-14T19:29:00.619240+0000","last_unstale":"2023-07-14T19:29:04.670955+0000","last_undegraded":"2023-07-14T19:29:04.670955+0000","last_fullsized":"2023-07-14T19:29:04.670955+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:28:59.600128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:28:59.600128+0000","last_clean_scrub_stamp":"2023-07-14T19:28:59.600128+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T02:51:09.423733+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":12,"reported_epoch":83,"state":"active+clean","last_fresh":"2023-07-14T19:29:02.573459+0000","last_change":"2023-07-14T19:29:01.308964+0000","last_active":"2023-07-14T19:29:02.573459+0000","last_peered":"2023-07-14T19:29:02.573459+0000","last_clean":"2023-07-14T19:29:02.573459+0000","last_became_active":"2023-07-14T19:29:01.308638+0000","last_became_peered":"2023-07-14T19:29:01.308638+0000","last_unstale":"2023-07-14T19:29:02.573459+0000","last_undegraded":"2023-07-14T19:29:02.573459+0000","last_fullsized":"2023-07-14T19:29:02.573459+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:29:00.604983+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:29:00.604983+0000","last_clean_scrub_stamp":"2023-07-14T19:29:00.604983+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T03:51:09.875511+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.a","version":"83'1","reported_seq":15,"reported_epoch":83,"state":"active+clean","last_fresh":"2023-07-14T19:29:03.960747+0000","last_change":"2023-07-14T19:29:00.617894+0000","last_active":"2023-07-14T19:29:03.960747+0000","last_peered":"2023-07-14T19:29:03.960747+0000","last_clean":"2023-07-14T19:29:03.960747+0000","last_became_active":"2023-07-14T19:29:00.617584+0000","last_became_peered":"2023-07-14T19:29:00.617584+0000","last_unstale":"2023-07-14T19:29:03.960747+0000","last_undegraded":"2023-07-14T19:29:03.960747+0000","last_fullsized":"2023-07-14T19:29:03.960747+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:28:59.600128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:28:59.600128+0000","last_clean_scrub_stamp":"2023-07-14T19:28:59.600128+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T23:24:30.467604+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":12,"reported_epoch":83,"state":"active+clean","last_fresh":"2023-07-14T19:29:02.299239+0000","last_change":"2023-07-14T19:29:01.317153+0000","last_active":"2023-07-14T19:29:02.299239+0000","last_peered":"2023-07-14T19:29:02.299239+0000","last_clean":"2023-07-14T19:29:02.299239+0000","last_became_active":"2023-07-14T19:29:01.316800+0000","last_became_peered":"2023-07-14T19:29:01.316800+0000","last_unstale":"2023-07-14T19:29:02.299239+0000","last_undegraded":"2023-07-14T19:29:02.299239+0000","last_fullsized":"2023-07-14T19:29:02.299239+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:29:00.604983+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:29:00.604983+0000","last_clean_scrub_stamp":"2023-07-14T19:29:00.604983+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T04:58:24.726933+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"83'4","reported_seq":19,"reported_epoch":83,"state":"active+clean","last_fresh":"2023-07-14T19:29:04.672156+0000","last_change":"2023-07-14T19:29:00.618469+0000","last_active":"2023-07-14T19:29:04.672156+0000","last_peered":"2023-07-14T19:29:04.672156+0000","last_clean":"2023-07-14T19:29:04.672156+0000","last_became_active":"2023-07-14T19:29:00.618218+0000","last_became_peered":"2023-07-14T19:29:00.618218+0000","last_unstale":"2023-07-14T19:29:04.672156+0000","last_undegraded":"2023-07-14T19:29:04.672156+0000","last_fullsized":"2023-07-14T19:29:04.672156+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:28:59.600128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:28:59.600128+0000","last_clean_scrub_stamp":"2023-07-14T19:28:59.600128+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T22:41:36.925396+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":12,"reported_epoch":83,"state":"active+clean","last_fresh":"2023-07-14T19:29:03.339251+0000","last_change":"2023-07-14T19:29:01.319023+0000","last_active":"2023-07-14T19:29:03.339251+0000","last_peered":"2023-07-14T19:29:03.339251+0000","last_clean":"2023-07-14T19:29:03.339251+0000","last_became_active":"2023-07-14T19:29:01.311530+0000","last_became_peered":"2023-07-14T19:29:01.311530+0000","last_unstale":"2023-07-14T19:29:03.339251+0000","last_undegraded":"2023-07-14T19:29:03.339251+0000","last_fullsized":"2023-07-14T19:29:03.339251+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:29:00.604983+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:29:00.604983+0000","last_clean_scrub_stamp":"2023-07-14T19:29:00.604983+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T01:46:30.175019+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"83'3","reported_seq":18,"reported_epoch":83,"state":"active+clean","last_fresh":"2023-07-14T19:29:05.678777+0000","last_change":"2023-07-14T19:29:00.624227+0000","last_active":"2023-07-14T19:29:05.678777+0000","last_peered":"2023-07-14T19:29:05.678777+0000","last_clean":"2023-07-14T19:29:05.678777+0000","last_became_active":"2023-07-14T19:29:00.623953+0000","last_became_peered":"2023-07-14T19:29:00.623953+0000","last_unstale":"2023-07-14T19:29:05.678777+0000","last_undegraded":"2023-07-14T19:29:05.678777+0000","last_fullsized":"2023-07-14T19:29:05.678777+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:28:59.600128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:28:59.600128+0000","last_clean_scrub_stamp":"2023-07-14T19:28:59.600128+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T04:31:39.691751+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":12,"reported_epoch":83,"state":"active+clean","last_fresh":"2023-07-14T19:29:03.340024+0000","last_change":"2023-07-14T19:29:01.319936+0000","last_active":"2023-07-14T19:29:03.340024+0000","last_peered":"2023-07-14T19:29:03.340024+0000","last_clean":"2023-07-14T19:29:03.340024+0000","last_became_active":"2023-07-14T19:29:01.319437+0000","last_became_peered":"2023-07-14T19:29:01.319437+0000","last_unstale":"2023-07-14T19:29:03.340024+0000","last_undegraded":"2023-07-14T19:29:03.340024+0000","last_fullsized":"2023-07-14T19:29:03.340024+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:29:00.604983+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:29:00.604983+0000","last_clean_scrub_stamp":"2023-07-14T19:29:00.604983+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T07:14:58.343588+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"82'1","reported_seq":16,"reported_epoch":83,"state":"active+clean","last_fresh":"2023-07-14T19:29:02.299764+0000","last_change":"2023-07-14T19:29:00.628103+0000","last_active":"2023-07-14T19:29:02.299764+0000","last_peered":"2023-07-14T19:29:02.299764+0000","last_clean":"2023-07-14T19:29:02.299764+0000","last_became_active":"2023-07-14T19:29:00.627689+0000","last_became_peered":"2023-07-14T19:29:00.627689+0000","last_unstale":"2023-07-14T19:29:02.299764+0000","last_undegraded":"2023-07-14T19:29:02.299764+0000","last_fullsized":"2023-07-14T19:29:02.299764+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:28:59.600128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:28:59.600128+0000","last_clean_scrub_stamp":"2023-07-14T19:28:59.600128+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T23:24:22.985508+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"83'1","reported_seq":13,"reported_epoch":83,"state":"active+clean","last_fresh":"2023-07-14T19:29:05.363002+0000","last_change":"2023-07-14T19:29:01.319519+0000","last_active":"2023-07-14T19:29:05.363002+0000","last_peered":"2023-07-14T19:29:05.363002+0000","last_clean":"2023-07-14T19:29:05.363002+0000","last_became_active":"2023-07-14T19:29:01.318900+0000","last_became_peered":"2023-07-14T19:29:01.318900+0000","last_unstale":"2023-07-14T19:29:05.363002+0000","last_undegraded":"2023-07-14T19:29:05.363002+0000","last_fullsized":"2023-07-14T19:29:05.363002+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:29:00.604983+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:29:00.604983+0000","last_clean_scrub_stamp":"2023-07-14T19:29:00.604983+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T20:31:55.947593+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":123,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"82'2","reported_seq":16,"reported_epoch":83,"state":"active+clean","last_fresh":"2023-07-14T19:29:02.444432+0000","last_change":"2023-07-14T19:29:00.619280+0000","last_active":"2023-07-14T19:29:02.444432+0000","last_peered":"2023-07-14T19:29:02.444432+0000","last_clean":"2023-07-14T19:29:02.444432+0000","last_became_active":"2023-07-14T19:29:00.618980+0000","last_became_peered":"2023-07-14T19:29:00.618980+0000","last_unstale":"2023-07-14T19:29:02.444432+0000","last_undegraded":"2023-07-14T19:29:02.444432+0000","last_fullsized":"2023-07-14T19:29:02.444432+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:28:59.600128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:28:59.600128+0000","last_clean_scrub_stamp":"2023-07-14T19:28:59.600128+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T22:09:48.530415+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":12,"reported_epoch":83,"state":"active+clean","last_fresh":"2023-07-14T19:29:03.339356+0000","last_change":"2023-07-14T19:29:01.320160+0000","last_active":"2023-07-14T19:29:03.339356+0000","last_peered":"2023-07-14T19:29:03.339356+0000","last_clean":"2023-07-14T19:29:03.339356+0000","last_became_active":"2023-07-14T19:29:01.319850+0000","last_became_peered":"2023-07-14T19:29:01.319850+0000","last_unstale":"2023-07-14T19:29:03.339356+0000","last_undegraded":"2023-07-14T19:29:03.339356+0000","last_fullsized":"2023-07-14T19:29:03.339356+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:29:00.604983+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:29:00.604983+0000","last_clean_scrub_stamp":"2023-07-14T19:29:00.604983+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T04:25:40.648850+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":12,"reported_epoch":83,"state":"active+clean","last_fresh":"2023-07-14T19:29:02.573560+0000","last_change":"2023-07-14T19:29:01.309770+0000","last_active":"2023-07-14T19:29:02.573560+0000","last_peered":"2023-07-14T19:29:02.573560+0000","last_clean":"2023-07-14T19:29:02.573560+0000","last_became_active":"2023-07-14T19:29:01.308677+0000","last_became_peered":"2023-07-14T19:29:01.308677+0000","last_unstale":"2023-07-14T19:29:02.573560+0000","last_undegraded":"2023-07-14T19:29:02.573560+0000","last_fullsized":"2023-07-14T19:29:02.573560+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:29:00.604983+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:29:00.604983+0000","last_clean_scrub_stamp":"2023-07-14T19:29:00.604983+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T21:27:03.743070+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":12,"reported_epoch":83,"state":"active+clean","last_fresh":"2023-07-14T19:29:03.221821+0000","last_change":"2023-07-14T19:29:01.319447+0000","last_active":"2023-07-14T19:29:03.221821+0000","last_peered":"2023-07-14T19:29:03.221821+0000","last_clean":"2023-07-14T19:29:03.221821+0000","last_became_active":"2023-07-14T19:29:01.319120+0000","last_became_peered":"2023-07-14T19:29:01.319120+0000","last_unstale":"2023-07-14T19:29:03.221821+0000","last_undegraded":"2023-07-14T19:29:03.221821+0000","last_fullsized":"2023-07-14T19:29:03.221821+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:29:00.604983+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:29:00.604983+0000","last_clean_scrub_stamp":"2023-07-14T19:29:00.604983+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T07:03:28.055942+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":12,"reported_epoch":83,"state":"active+clean","last_fresh":"2023-07-14T19:29:03.221929+0000","last_change":"2023-07-14T19:29:01.310554+0000","last_active":"2023-07-14T19:29:03.221929+0000","last_peered":"2023-07-14T19:29:03.221929+0000","last_clean":"2023-07-14T19:29:03.221929+0000","last_became_active":"2023-07-14T19:29:01.310302+0000","last_became_peered":"2023-07-14T19:29:01.310302+0000","last_unstale":"2023-07-14T19:29:03.221929+0000","last_undegraded":"2023-07-14T19:29:03.221929+0000","last_fullsized":"2023-07-14T19:29:03.221929+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:29:00.604983+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:29:00.604983+0000","last_clean_scrub_stamp":"2023-07-14T19:29:00.604983+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T04:50:18.928722+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":12,"reported_epoch":83,"state":"active+clean","last_fresh":"2023-07-14T19:29:02.603764+0000","last_change":"2023-07-14T19:29:01.318753+0000","last_active":"2023-07-14T19:29:02.603764+0000","last_peered":"2023-07-14T19:29:02.603764+0000","last_clean":"2023-07-14T19:29:02.603764+0000","last_became_active":"2023-07-14T19:29:01.318476+0000","last_became_peered":"2023-07-14T19:29:01.318476+0000","last_unstale":"2023-07-14T19:29:02.603764+0000","last_undegraded":"2023-07-14T19:29:02.603764+0000","last_fullsized":"2023-07-14T19:29:02.603764+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:29:00.604983+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:29:00.604983+0000","last_clean_scrub_stamp":"2023-07-14T19:29:00.604983+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T07:27:40.189788+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":12,"reported_epoch":83,"state":"active+clean","last_fresh":"2023-07-14T19:29:02.444422+0000","last_change":"2023-07-14T19:29:01.317843+0000","last_active":"2023-07-14T19:29:02.444422+0000","last_peered":"2023-07-14T19:29:02.444422+0000","last_clean":"2023-07-14T19:29:02.444422+0000","last_became_active":"2023-07-14T19:29:01.317550+0000","last_became_peered":"2023-07-14T19:29:01.317550+0000","last_unstale":"2023-07-14T19:29:02.444422+0000","last_undegraded":"2023-07-14T19:29:02.444422+0000","last_fullsized":"2023-07-14T19:29:02.444422+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:29:00.604983+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:29:00.604983+0000","last_clean_scrub_stamp":"2023-07-14T19:29:00.604983+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T22:59:15.139142+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":12,"reported_epoch":83,"state":"active+clean","last_fresh":"2023-07-14T19:29:02.573731+0000","last_change":"2023-07-14T19:29:01.309103+0000","last_active":"2023-07-14T19:29:02.573731+0000","last_peered":"2023-07-14T19:29:02.573731+0000","last_clean":"2023-07-14T19:29:02.573731+0000","last_became_active":"2023-07-14T19:29:01.308786+0000","last_became_peered":"2023-07-14T19:29:01.308786+0000","last_unstale":"2023-07-14T19:29:02.573731+0000","last_undegraded":"2023-07-14T19:29:02.573731+0000","last_fullsized":"2023-07-14T19:29:02.573731+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:29:00.604983+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:29:00.604983+0000","last_clean_scrub_stamp":"2023-07-14T19:29:00.604983+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T02:55:34.529448+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":12,"reported_epoch":83,"state":"active+clean","last_fresh":"2023-07-14T19:29:03.222213+0000","last_change":"2023-07-14T19:29:01.309940+0000","last_active":"2023-07-14T19:29:03.222213+0000","last_peered":"2023-07-14T19:29:03.222213+0000","last_clean":"2023-07-14T19:29:03.222213+0000","last_became_active":"2023-07-14T19:29:01.309638+0000","last_became_peered":"2023-07-14T19:29:01.309638+0000","last_unstale":"2023-07-14T19:29:03.222213+0000","last_undegraded":"2023-07-14T19:29:03.222213+0000","last_fullsized":"2023-07-14T19:29:03.222213+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:29:00.604983+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:29:00.604983+0000","last_clean_scrub_stamp":"2023-07-14T19:29:00.604983+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T03:28:26.563593+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":12,"reported_epoch":83,"state":"active+clean","last_fresh":"2023-07-14T19:29:02.299283+0000","last_change":"2023-07-14T19:29:01.305957+0000","last_active":"2023-07-14T19:29:02.299283+0000","last_peered":"2023-07-14T19:29:02.299283+0000","last_clean":"2023-07-14T19:29:02.299283+0000","last_became_active":"2023-07-14T19:29:01.305564+0000","last_became_peered":"2023-07-14T19:29:01.305564+0000","last_unstale":"2023-07-14T19:29:02.299283+0000","last_undegraded":"2023-07-14T19:29:02.299283+0000","last_fullsized":"2023-07-14T19:29:02.299283+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:29:00.604983+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:29:00.604983+0000","last_clean_scrub_stamp":"2023-07-14T19:29:00.604983+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T21:47:06.297622+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":12,"reported_epoch":83,"state":"active+clean","last_fresh":"2023-07-14T19:29:02.299375+0000","last_change":"2023-07-14T19:29:01.310521+0000","last_active":"2023-07-14T19:29:02.299375+0000","last_peered":"2023-07-14T19:29:02.299375+0000","last_clean":"2023-07-14T19:29:02.299375+0000","last_became_active":"2023-07-14T19:29:01.310211+0000","last_became_peered":"2023-07-14T19:29:01.310211+0000","last_unstale":"2023-07-14T19:29:02.299375+0000","last_undegraded":"2023-07-14T19:29:02.299375+0000","last_fullsized":"2023-07-14T19:29:02.299375+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:29:00.604983+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:29:00.604983+0000","last_clean_scrub_stamp":"2023-07-14T19:29:00.604983+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T19:41:09.317667+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":12,"reported_epoch":83,"state":"active+clean","last_fresh":"2023-07-14T19:29:02.298983+0000","last_change":"2023-07-14T19:29:01.308033+0000","last_active":"2023-07-14T19:29:02.298983+0000","last_peered":"2023-07-14T19:29:02.298983+0000","last_clean":"2023-07-14T19:29:02.298983+0000","last_became_active":"2023-07-14T19:29:01.307166+0000","last_became_peered":"2023-07-14T19:29:01.307166+0000","last_unstale":"2023-07-14T19:29:02.298983+0000","last_undegraded":"2023-07-14T19:29:02.298983+0000","last_fullsized":"2023-07-14T19:29:02.298983+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:29:00.604983+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:29:00.604983+0000","last_clean_scrub_stamp":"2023-07-14T19:29:00.604983+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T06:29:57.949181+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":12,"reported_epoch":83,"state":"active+clean","last_fresh":"2023-07-14T19:29:02.299844+0000","last_change":"2023-07-14T19:29:01.308122+0000","last_active":"2023-07-14T19:29:02.299844+0000","last_peered":"2023-07-14T19:29:02.299844+0000","last_clean":"2023-07-14T19:29:02.299844+0000","last_became_active":"2023-07-14T19:29:01.307568+0000","last_became_peered":"2023-07-14T19:29:01.307568+0000","last_unstale":"2023-07-14T19:29:02.299844+0000","last_undegraded":"2023-07-14T19:29:02.299844+0000","last_fullsized":"2023-07-14T19:29:02.299844+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:29:00.604983+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:29:00.604983+0000","last_clean_scrub_stamp":"2023-07-14T19:29:00.604983+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T04:08:56.378075+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":12,"reported_epoch":83,"state":"active+clean","last_fresh":"2023-07-14T19:29:02.444527+0000","last_change":"2023-07-14T19:29:01.314261+0000","last_active":"2023-07-14T19:29:02.444527+0000","last_peered":"2023-07-14T19:29:02.444527+0000","last_clean":"2023-07-14T19:29:02.444527+0000","last_became_active":"2023-07-14T19:29:01.313967+0000","last_became_peered":"2023-07-14T19:29:01.313967+0000","last_unstale":"2023-07-14T19:29:02.444527+0000","last_undegraded":"2023-07-14T19:29:02.444527+0000","last_fullsized":"2023-07-14T19:29:02.444527+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:29:00.604983+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:29:00.604983+0000","last_clean_scrub_stamp":"2023-07-14T19:29:00.604983+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T04:19:07.923742+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":12,"reported_epoch":83,"state":"active+clean","last_fresh":"2023-07-14T19:29:02.573578+0000","last_change":"2023-07-14T19:29:01.319605+0000","last_active":"2023-07-14T19:29:02.573578+0000","last_peered":"2023-07-14T19:29:02.573578+0000","last_clean":"2023-07-14T19:29:02.573578+0000","last_became_active":"2023-07-14T19:29:01.319310+0000","last_became_peered":"2023-07-14T19:29:01.319310+0000","last_unstale":"2023-07-14T19:29:02.573578+0000","last_undegraded":"2023-07-14T19:29:02.573578+0000","last_fullsized":"2023-07-14T19:29:02.573578+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:29:00.604983+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:29:00.604983+0000","last_clean_scrub_stamp":"2023-07-14T19:29:00.604983+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T20:14:29.714925+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":246,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":738,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":4751360,"data_stored":4722944,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":58954,"num_objects":60,"num_object_clones":0,"num_object_copies":180,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":60,"num_whiteouts":0,"num_read":261,"num_read_kb":191,"num_write":149,"num_write_kb":100,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":405504,"data_stored":176862,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":131,"ondisk_log_size":131,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":78,"seq":335007449108,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27892,"kb_used_data":876,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717260,"statfs":{"total":95995035648,"available":95966474240,"internally_reserved":0,"allocated":897024,"data_stored":429039,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":27654411},"hb_peers":[0,1,2,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jul 14 19:29:19 2023","interfaces":[{"interface":"back","average":{"1min":0.52800000000000002,"5min":0.52800000000000002,"15min":0.52800000000000002},"min":{"1min":0.33400000000000002,"5min":0.33400000000000002,"15min":0.33400000000000002},"max":{"1min":0.82999999999999996,"5min":0.82999999999999996,"15min":0.82999999999999996},"last":0.51000000000000001},{"interface":"front","average":{"1min":0.56100000000000005,"5min":0.56100000000000005,"15min":0.56100000000000005},"min":{"1min":0.38800000000000001,"5min":0.38800000000000001,"15min":0.38800000000000001},"max":{"1min":0.95599999999999996,"5min":0.95599999999999996,"15min":0.95599999999999996},"last":0.47399999999999998}]},{"osd":1,"last update":"Fri Jul 14 19:29:19 2023","interfaces":[{"interface":"back","average":{"1min":0.65800000000000003,"5min":0.65800000000000003,"15min":0.65800000000000003},"min":{"1min":0.41499999999999998,"5min":0.41499999999999998,"15min":0.41499999999999998},"max":{"1min":0.99399999999999999,"5min":0.99399999999999999,"15min":0.99399999999999999},"last":0.76500000000000001},{"interface":"front","average":{"1min":0.60699999999999998,"5min":0.60699999999999998,"15min":0.60699999999999998},"min":{"1min":0.42099999999999999,"5min":0.42099999999999999,"15min":0.42099999999999999},"max":{"1min":1.5640000000000001,"5min":1.5640000000000001,"15min":1.5640000000000001},"last":0.64600000000000002}]},{"osd":2,"last update":"Fri Jul 14 19:29:19 2023","interfaces":[{"interface":"back","average":{"1min":0.52800000000000002,"5min":0.52800000000000002,"15min":0.52800000000000002},"min":{"1min":0.30299999999999999,"5min":0.30299999999999999,"15min":0.30299999999999999},"max":{"1min":1.097,"5min":1.097,"15min":1.097},"last":0.69099999999999995},{"interface":"front","average":{"1min":0.51800000000000002,"5min":0.51800000000000002,"15min":0.51800000000000002},"min":{"1min":0.30099999999999999,"5min":0.30099999999999999,"15min":0.30099999999999999},"max":{"1min":0.86399999999999999,"5min":0.86399999999999999,"15min":0.86399999999999999},"last":0.66800000000000004}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.37}]},{"osd":4,"last update":"Fri Jul 14 19:29:19 2023","interfaces":[{"interface":"back","average":{"1min":0.68600000000000005,"5min":0.68600000000000005,"15min":0.68600000000000005},"min":{"1min":0.39300000000000002,"5min":0.39300000000000002,"15min":0.39300000000000002},"max":{"1min":1.0389999999999999,"5min":1.0389999999999999,"15min":1.0389999999999999},"last":0.80000000000000004},{"interface":"front","average":{"1min":0.63700000000000001,"5min":0.63700000000000001,"15min":0.63700000000000001},"min":{"1min":0.41699999999999998,"5min":0.41699999999999998,"15min":0.41699999999999998},"max":{"1min":0.91800000000000004,"5min":0.91800000000000004,"15min":0.91800000000000004},"last":0.83399999999999996}]},{"osd":5,"last update":"Fri Jul 14 19:29:19 2023","interfaces":[{"interface":"back","average":{"1min":0.61299999999999999,"5min":0.61299999999999999,"15min":0.61299999999999999},"min":{"1min":0.32500000000000001,"5min":0.32500000000000001,"15min":0.32500000000000001},"max":{"1min":1.1930000000000001,"5min":1.1930000000000001,"15min":1.1930000000000001},"last":0.628},{"interface":"front","average":{"1min":0.61599999999999999,"5min":0.61599999999999999,"15min":0.61599999999999999},"min":{"1min":0.36699999999999999,"5min":0.36699999999999999,"15min":0.36699999999999999},"max":{"1min":1.018,"5min":1.018,"15min":1.018},"last":0.71899999999999997}]},{"osd":6,"last update":"Fri Jul 14 19:29:19 2023","interfaces":[{"interface":"back","average":{"1min":0.62,"5min":0.62,"15min":0.62},"min":{"1min":0.40100000000000002,"5min":0.40100000000000002,"15min":0.40100000000000002},"max":{"1min":0.97599999999999998,"5min":0.97599999999999998,"15min":0.97599999999999998},"last":0.56599999999999995},{"interface":"front","average":{"1min":0.70899999999999996,"5min":0.70899999999999996,"15min":0.70899999999999996},"min":{"1min":0.436,"5min":0.436,"15min":0.436},"max":{"1min":1,"5min":1,"15min":1},"last":0.75}]},{"osd":7,"last update":"Fri Jul 14 19:29:19 2023","interfaces":[{"interface":"back","average":{"1min":0.76900000000000002,"5min":0.76900000000000002,"15min":0.76900000000000002},"min":{"1min":0.52400000000000002,"5min":0.52400000000000002,"15min":0.52400000000000002},"max":{"1min":1.032,"5min":1.032,"15min":1.032},"last":0.85399999999999998},{"interface":"front","average":{"1min":0.71299999999999997,"5min":0.71299999999999997,"15min":0.71299999999999997},"min":{"1min":0.54100000000000004,"5min":0.54100000000000004,"15min":0.54100000000000004},"max":{"1min":1.0269999999999999,"5min":1.0269999999999999,"15min":1.0269999999999999},"last":0.73999999999999999}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.77700000000000002}]},{"osd":9,"last update":"Fri Jul 14 19:29:19 2023","interfaces":[{"interface":"back","average":{"1min":0.77600000000000002,"5min":0.77600000000000002,"15min":0.77600000000000002},"min":{"1min":0.52600000000000002,"5min":0.52600000000000002,"15min":0.52600000000000002},"max":{"1min":1.2230000000000001,"5min":1.2230000000000001,"15min":1.2230000000000001},"last":0.81299999999999994},{"interface":"front","average":{"1min":0.78000000000000003,"5min":0.78000000000000003,"15min":0.78000000000000003},"min":{"1min":0.51700000000000002,"5min":0.51700000000000002,"15min":0.51700000000000002},"max":{"1min":1.0700000000000001,"5min":1.0700000000000001,"15min":1.0700000000000001},"last":0.88500000000000001}]},{"osd":10,"last update":"Fri Jul 14 19:29:19 2023","interfaces":[{"interface":"back","average":{"1min":0.74099999999999999,"5min":0.74099999999999999,"15min":0.74099999999999999},"min":{"1min":0.56299999999999994,"5min":0.56299999999999994,"15min":0.56299999999999994},"max":{"1min":0.93100000000000005,"5min":0.93100000000000005,"15min":0.93100000000000005},"last":0.78900000000000003},{"interface":"front","average":{"1min":0.81200000000000006,"5min":0.81200000000000006,"15min":0.81200000000000006},"min":{"1min":0.56799999999999995,"5min":0.56799999999999995,"15min":0.56799999999999995},"max":{"1min":1.2609999999999999,"5min":1.2609999999999999,"15min":1.2609999999999999},"last":0.87}]}]},{"osd":10,"up_from":70,"seq":300647710745,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28564,"kb_used_data":1548,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716588,"statfs":{"total":95995035648,"available":95965786112,"internally_reserved":0,"allocated":1585152,"data_stored":1075197,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":27654411},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.55700000000000005}]},{"osd":1,"last update":"Fri Jul 14 19:29:00 2023","interfaces":[{"interface":"back","average":{"1min":0.44800000000000001,"5min":0.44800000000000001,"15min":0.44800000000000001},"min":{"1min":0.27000000000000002,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":0.73299999999999998,"5min":0.73299999999999998,"15min":0.73299999999999998},"last":0.40000000000000002},{"interface":"front","average":{"1min":0.46800000000000003,"5min":0.46800000000000003,"15min":0.46800000000000003},"min":{"1min":0.28599999999999998,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":0.80800000000000005,"5min":0.80800000000000005,"15min":0.80800000000000005},"last":0.53700000000000003}]},{"osd":2,"last update":"Fri Jul 14 19:29:00 2023","interfaces":[{"interface":"back","average":{"1min":0.59899999999999998,"5min":0.59899999999999998,"15min":0.59899999999999998},"min":{"1min":0.40200000000000002,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":0.83699999999999997,"5min":0.83699999999999997,"15min":0.83699999999999997},"last":0.68000000000000005},{"interface":"front","average":{"1min":0.64300000000000002,"5min":0.64300000000000002,"15min":0.64300000000000002},"min":{"1min":0.46800000000000003,"5min":0.46800000000000003,"15min":0.46800000000000003},"max":{"1min":0.89100000000000001,"5min":0.89100000000000001,"15min":0.89100000000000001},"last":0.58299999999999996}]},{"osd":3,"last update":"Fri Jul 14 19:29:00 2023","interfaces":[{"interface":"back","average":{"1min":0.58199999999999996,"5min":0.58199999999999996,"15min":0.58199999999999996},"min":{"1min":0.35999999999999999,"5min":0.35999999999999999,"15min":0.35999999999999999},"max":{"1min":0.77800000000000002,"5min":0.77800000000000002,"15min":0.77800000000000002},"last":0.504},{"interface":"front","average":{"1min":0.63200000000000001,"5min":0.63200000000000001,"15min":0.63200000000000001},"min":{"1min":0.45200000000000001,"5min":0.45200000000000001,"15min":0.45200000000000001},"max":{"1min":0.89700000000000002,"5min":0.89700000000000002,"15min":0.89700000000000002},"last":0.42299999999999999}]},{"osd":4,"last update":"Fri Jul 14 19:29:00 2023","interfaces":[{"interface":"back","average":{"1min":0.55600000000000005,"5min":0.55600000000000005,"15min":0.55600000000000005},"min":{"1min":0.316,"5min":0.316,"15min":0.316},"max":{"1min":0.99399999999999999,"5min":0.99399999999999999,"15min":0.99399999999999999},"last":0.59599999999999997},{"interface":"front","average":{"1min":0.54800000000000004,"5min":0.54800000000000004,"15min":0.54800000000000004},"min":{"1min":0.29299999999999998,"5min":0.29299999999999998,"15min":0.29299999999999998},"max":{"1min":0.94999999999999996,"5min":0.94999999999999996,"15min":0.94999999999999996},"last":0.63300000000000001}]},{"osd":5,"last update":"Fri Jul 14 19:29:00 2023","interfaces":[{"interface":"back","average":{"1min":0.68600000000000005,"5min":0.68600000000000005,"15min":0.68600000000000005},"min":{"1min":0.45300000000000001,"5min":0.45300000000000001,"15min":0.45300000000000001},"max":{"1min":1.0129999999999999,"5min":1.0129999999999999,"15min":1.0129999999999999},"last":0.72299999999999998},{"interface":"front","average":{"1min":0.67100000000000004,"5min":0.67100000000000004,"15min":0.67100000000000004},"min":{"1min":0.374,"5min":0.374,"15min":0.374},"max":{"1min":0.96499999999999997,"5min":0.96499999999999997,"15min":0.96499999999999997},"last":0.69999999999999996}]},{"osd":6,"last update":"Fri Jul 14 19:29:00 2023","interfaces":[{"interface":"back","average":{"1min":0.65700000000000003,"5min":0.65700000000000003,"15min":0.65700000000000003},"min":{"1min":0.40400000000000003,"5min":0.40400000000000003,"15min":0.40400000000000003},"max":{"1min":1.032,"5min":1.032,"15min":1.032},"last":0.61899999999999999},{"interface":"front","average":{"1min":0.68799999999999994,"5min":0.68799999999999994,"15min":0.68799999999999994},"min":{"1min":0.45700000000000002,"5min":0.45700000000000002,"15min":0.45700000000000002},"max":{"1min":1.0649999999999999,"5min":1.0649999999999999,"15min":1.0649999999999999},"last":0.71099999999999997}]},{"osd":7,"last update":"Fri Jul 14 19:29:00 2023","interfaces":[{"interface":"back","average":{"1min":0.66000000000000003,"5min":0.66000000000000003,"15min":0.66000000000000003},"min":{"1min":0.34200000000000003,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":1.0960000000000001,"5min":1.0960000000000001,"15min":1.0960000000000001},"last":0.46000000000000002},{"interface":"front","average":{"1min":0.69399999999999995,"5min":0.69399999999999995,"15min":0.69399999999999995},"min":{"1min":0.44600000000000001,"5min":0.44600000000000001,"15min":0.44600000000000001},"max":{"1min":0.97899999999999998,"5min":0.97899999999999998,"15min":0.97899999999999998},"last":0.44600000000000001}]},{"osd":8,"last update":"Fri Jul 14 19:29:00 2023","interfaces":[{"interface":"back","average":{"1min":0.75900000000000001,"5min":0.75900000000000001,"15min":0.75900000000000001},"min":{"1min":0.52800000000000002,"5min":0.52800000000000002,"15min":0.52800000000000002},"max":{"1min":1.143,"5min":1.143,"15min":1.143},"last":0.77700000000000002},{"interface":"front","average":{"1min":0.77600000000000002,"5min":0.77600000000000002,"15min":0.77600000000000002},"min":{"1min":0.47599999999999998,"5min":0.47599999999999998,"15min":0.47599999999999998},"max":{"1min":1.1060000000000001,"5min":1.1060000000000001,"15min":1.1060000000000001},"last":0.81100000000000005}]},{"osd":9,"last update":"Fri Jul 14 19:29:00 2023","interfaces":[{"interface":"back","average":{"1min":0.754,"5min":0.754,"15min":0.754},"min":{"1min":0.54500000000000004,"5min":0.54500000000000004,"15min":0.54500000000000004},"max":{"1min":1.115,"5min":1.115,"15min":1.115},"last":0.65600000000000003},{"interface":"front","average":{"1min":0.77000000000000002,"5min":0.77000000000000002,"15min":0.77000000000000002},"min":{"1min":0.49299999999999999,"5min":0.49299999999999999,"15min":0.49299999999999999},"max":{"1min":1.127,"5min":1.127,"15min":1.127},"last":0.76000000000000001}]},{"osd":11,"last update":"Fri Jul 14 19:29:25 2023","interfaces":[{"interface":"back","average":{"1min":0.75,"5min":0.75,"15min":0.75},"min":{"1min":0.54000000000000004,"5min":0.54000000000000004,"15min":0.54000000000000004},"max":{"1min":1.0489999999999999,"5min":1.0489999999999999,"15min":1.0489999999999999},"last":0.79800000000000004},{"interface":"front","average":{"1min":0.75800000000000001,"5min":0.75800000000000001,"15min":0.75800000000000001},"min":{"1min":0.54900000000000004,"5min":0.54900000000000004,"15min":0.54900000000000004},"max":{"1min":0.99299999999999999,"5min":0.99299999999999999,"15min":0.99299999999999999},"last":0.74199999999999999}]}]},{"osd":9,"up_from":65,"seq":279172874269,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27908,"kb_used_data":892,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717244,"statfs":{"total":95995035648,"available":95966457856,"internally_reserved":0,"allocated":913408,"data_stored":429275,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":27654411},"hb_peers":[0,1,2,3,4,5,6,7,8,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jul 14 19:29:38 2023","interfaces":[{"interface":"back","average":{"1min":0.34699999999999998,"5min":0.41599999999999998,"15min":0.42799999999999999},"min":{"1min":0.218,"5min":0.218,"15min":0.218},"max":{"1min":0.55000000000000004,"5min":0.55000000000000004,"15min":0.55000000000000004},"last":0.28699999999999998},{"interface":"front","average":{"1min":0.53200000000000003,"5min":0.48399999999999999,"15min":0.47599999999999998},"min":{"1min":0.28699999999999998,"5min":0.26100000000000001,"15min":0.26100000000000001},"max":{"1min":2.0619999999999998,"5min":2.0619999999999998,"15min":2.0619999999999998},"last":0.26100000000000001}]},{"osd":1,"last update":"Fri Jul 14 19:29:38 2023","interfaces":[{"interface":"back","average":{"1min":0.50900000000000001,"5min":0.60699999999999998,"15min":0.623},"min":{"1min":0.35099999999999998,"5min":0.35099999999999998,"15min":0.35099999999999998},"max":{"1min":0.73599999999999999,"5min":0.82199999999999995,"15min":0.82199999999999995},"last":0.41499999999999998},{"interface":"front","average":{"1min":0.53600000000000003,"5min":0.54500000000000004,"15min":0.54700000000000004},"min":{"1min":0.34100000000000003,"5min":0.34100000000000003,"15min":0.34100000000000003},"max":{"1min":0.86699999999999999,"5min":0.86699999999999999,"15min":0.86699999999999999},"last":0.27500000000000002}]},{"osd":2,"last update":"Fri Jul 14 19:29:38 2023","interfaces":[{"interface":"back","average":{"1min":0.53600000000000003,"5min":0.61099999999999999,"15min":0.623},"min":{"1min":0.28899999999999998,"5min":0.28899999999999998,"15min":0.28899999999999998},"max":{"1min":0.66100000000000003,"5min":0.75800000000000001,"15min":0.75800000000000001},"last":0.44700000000000001},{"interface":"front","average":{"1min":0.57099999999999995,"5min":0.58499999999999996,"15min":0.58699999999999997},"min":{"1min":0.36499999999999999,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":0.72899999999999998,"5min":0.72899999999999998,"15min":0.72899999999999998},"last":0.499}]},{"osd":3,"last update":"Fri Jul 14 19:29:38 2023","interfaces":[{"interface":"back","average":{"1min":0.57399999999999995,"5min":0.53000000000000003,"15min":0.52200000000000002},"min":{"1min":0.249,"5min":0.249,"15min":0.249},"max":{"1min":2.4260000000000002,"5min":2.4260000000000002,"15min":2.4260000000000002},"last":0.33300000000000002},{"interface":"front","average":{"1min":0.42399999999999999,"5min":0.55200000000000005,"15min":0.57299999999999995},"min":{"1min":0.29799999999999999,"5min":0.29799999999999999,"15min":0.29799999999999999},"max":{"1min":0.84299999999999997,"5min":0.85699999999999998,"15min":0.85699999999999998},"last":0.376}]},{"osd":4,"last update":"Fri Jul 14 19:29:38 2023","interfaces":[{"interface":"back","average":{"1min":0.54900000000000004,"5min":0.58599999999999997,"15min":0.59199999999999997},"min":{"1min":0.34100000000000003,"5min":0.34100000000000003,"15min":0.34100000000000003},"max":{"1min":0.76700000000000002,"5min":0.77200000000000002,"15min":0.77200000000000002},"last":0.34300000000000003},{"interface":"front","average":{"1min":0.72599999999999998,"5min":0.67300000000000004,"15min":0.66400000000000003},"min":{"1min":0.42599999999999999,"5min":0.42599999999999999,"15min":0.42599999999999999},"max":{"1min":2.3450000000000002,"5min":2.3450000000000002,"15min":2.3450000000000002},"last":0.35899999999999999}]},{"osd":5,"last update":"Fri Jul 14 19:29:38 2023","interfaces":[{"interface":"back","average":{"1min":0.76100000000000001,"5min":0.68899999999999995,"15min":0.67700000000000005},"min":{"1min":0.39100000000000001,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":2.274,"5min":2.274,"15min":2.274},"last":0.42799999999999999},{"interface":"front","average":{"1min":0.61199999999999999,"5min":0.71299999999999997,"15min":0.72999999999999998},"min":{"1min":0.35499999999999998,"5min":0.35499999999999998,"15min":0.35499999999999998},"max":{"1min":0.88900000000000001,"5min":0.90600000000000003,"15min":0.90600000000000003},"last":0.53500000000000003}]},{"osd":6,"last update":"Fri Jul 14 19:29:38 2023","interfaces":[{"interface":"back","average":{"1min":0.49099999999999999,"5min":0.65300000000000002,"15min":0.68000000000000005},"min":{"1min":0.26800000000000002,"5min":0.26800000000000002,"15min":0.26800000000000002},"max":{"1min":0.98899999999999999,"5min":0.98899999999999999,"15min":0.98899999999999999},"last":0.48399999999999999},{"interface":"front","average":{"1min":0.90800000000000003,"5min":0.66400000000000003,"15min":0.624},"min":{"1min":0.29799999999999999,"5min":0.29799999999999999,"15min":0.29799999999999999},"max":{"1min":8.0980000000000008,"5min":8.0980000000000008,"15min":8.0980000000000008},"last":0.316}]},{"osd":7,"last update":"Fri Jul 14 19:29:38 2023","interfaces":[{"interface":"back","average":{"1min":0.79400000000000004,"5min":0.76500000000000001,"15min":0.76000000000000001},"min":{"1min":0.45500000000000002,"5min":0.45500000000000002,"15min":0.45500000000000002},"max":{"1min":2.3980000000000001,"5min":2.3980000000000001,"15min":2.3980000000000001},"last":0.47599999999999998},{"interface":"front","average":{"1min":0.61099999999999999,"5min":0.74199999999999999,"15min":0.76400000000000001},"min":{"1min":0.46999999999999997,"5min":0.46999999999999997,"15min":0.46999999999999997},"max":{"1min":0.93799999999999994,"5min":0.95299999999999996,"15min":0.95299999999999996},"last":0.51600000000000001}]},{"osd":8,"last update":"Fri Jul 14 19:29:38 2023","interfaces":[{"interface":"back","average":{"1min":1.0549999999999999,"5min":0.78400000000000003,"15min":0.73899999999999999},"min":{"1min":0.439,"5min":0.439,"15min":0.439},"max":{"1min":8.0129999999999999,"5min":8.0129999999999999,"15min":8.0129999999999999},"last":0.52600000000000002},{"interface":"front","average":{"1min":0.82799999999999996,"5min":0.75900000000000001,"15min":0.747},"min":{"1min":0.41199999999999998,"5min":0.41199999999999998,"15min":0.41199999999999998},"max":{"1min":2.3130000000000002,"5min":2.3130000000000002,"15min":2.3130000000000002},"last":0.50900000000000001}]},{"osd":10,"last update":"Fri Jul 14 19:29:00 2023","interfaces":[{"interface":"back","average":{"1min":0.70299999999999996,"5min":0.70299999999999996,"15min":0.70299999999999996},"min":{"1min":0.439,"5min":0.439,"15min":0.439},"max":{"1min":0.95199999999999996,"5min":0.95199999999999996,"15min":0.95199999999999996},"last":0.46800000000000003},{"interface":"front","average":{"1min":0.73799999999999999,"5min":0.73799999999999999,"15min":0.73799999999999999},"min":{"1min":0.47799999999999998,"5min":0.47799999999999998,"15min":0.47799999999999998},"max":{"1min":1.016,"5min":1.016,"15min":1.016},"last":0.45800000000000002}]}]},{"osd":8,"up_from":57,"seq":244813135905,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27904,"kb_used_data":888,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717248,"statfs":{"total":95995035648,"available":95966461952,"internally_reserved":0,"allocated":909312,"data_stored":428914,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":27654411},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jul 14 19:29:19 2023","interfaces":[{"interface":"back","average":{"1min":0.51100000000000001,"5min":0.54600000000000004,"15min":0.55200000000000005},"min":{"1min":0.36899999999999999,"5min":0.30099999999999999,"15min":0.30099999999999999},"max":{"1min":0.82099999999999995,"5min":0.82099999999999995,"15min":0.82099999999999995},"last":0.56000000000000005},{"interface":"front","average":{"1min":0.61199999999999999,"5min":0.55800000000000005,"15min":0.54900000000000004},"min":{"1min":0.33400000000000002,"5min":0.32100000000000001,"15min":0.32100000000000001},"max":{"1min":2.0270000000000001,"5min":2.0270000000000001,"15min":2.0270000000000001},"last":0.58299999999999996}]},{"osd":1,"last update":"Fri Jul 14 19:29:19 2023","interfaces":[{"interface":"back","average":{"1min":0.623,"5min":0.58999999999999997,"15min":0.58399999999999996},"min":{"1min":0.372,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":0.83599999999999997,"5min":0.85499999999999998,"15min":0.85499999999999998},"last":0.89900000000000002},{"interface":"front","average":{"1min":0.61799999999999999,"5min":0.60599999999999998,"15min":0.60399999999999998},"min":{"1min":0.42099999999999999,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":0.81999999999999995,"5min":0.93400000000000005,"15min":0.93400000000000005},"last":0.69399999999999995}]},{"osd":2,"last update":"Fri Jul 14 19:29:19 2023","interfaces":[{"interface":"back","average":{"1min":0.49399999999999999,"5min":0.47599999999999998,"15min":0.47299999999999998},"min":{"1min":0.311,"5min":0.27700000000000002,"15min":0.27700000000000002},"max":{"1min":0.63900000000000001,"5min":0.67200000000000004,"15min":0.67200000000000004},"last":0.61799999999999999},{"interface":"front","average":{"1min":0.48999999999999999,"5min":0.46200000000000002,"15min":0.45700000000000002},"min":{"1min":0.29899999999999999,"5min":0.25600000000000001,"15min":0.25600000000000001},"max":{"1min":0.80700000000000005,"5min":0.80700000000000005,"15min":0.80700000000000005},"last":0.65800000000000003}]},{"osd":3,"last update":"Fri Jul 14 19:29:19 2023","interfaces":[{"interface":"back","average":{"1min":0.60199999999999998,"5min":0.57799999999999996,"15min":0.57399999999999995},"min":{"1min":0.38700000000000001,"5min":0.36199999999999999,"15min":0.36199999999999999},"max":{"1min":1.1120000000000001,"5min":1.1120000000000001,"15min":1.1120000000000001},"last":0.78300000000000003},{"interface":"front","average":{"1min":0.65500000000000003,"5min":0.59799999999999998,"15min":0.58799999999999997},"min":{"1min":0.32500000000000001,"5min":0.32500000000000001,"15min":0.32500000000000001},"max":{"1min":1.1890000000000001,"5min":1.1890000000000001,"15min":1.1890000000000001},"last":0.75700000000000001}]},{"osd":4,"last update":"Fri Jul 14 19:29:19 2023","interfaces":[{"interface":"back","average":{"1min":0.64400000000000002,"5min":0.61199999999999999,"15min":0.60599999999999998},"min":{"1min":0.34300000000000003,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":0.84799999999999998,"5min":0.86399999999999999,"15min":0.86399999999999999},"last":0.86299999999999999},{"interface":"front","average":{"1min":0.67400000000000004,"5min":0.64400000000000002,"15min":0.63900000000000001},"min":{"1min":0.41899999999999998,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":0.88100000000000001,"5min":0.88100000000000001,"15min":0.88100000000000001},"last":0.93000000000000005}]},{"osd":5,"last update":"Fri Jul 14 19:29:19 2023","interfaces":[{"interface":"back","average":{"1min":0.58999999999999997,"5min":0.55800000000000005,"15min":0.55300000000000005},"min":{"1min":0.39700000000000002,"5min":0.311,"15min":0.311},"max":{"1min":0.88100000000000001,"5min":0.88100000000000001,"15min":0.88100000000000001},"last":0.88100000000000001},{"interface":"front","average":{"1min":0.61499999999999999,"5min":0.57499999999999996,"15min":0.56899999999999995},"min":{"1min":0.433,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":0.84399999999999997,"5min":0.90900000000000003,"15min":0.90900000000000003},"last":0.91500000000000004}]},{"osd":6,"last update":"Fri Jul 14 19:29:19 2023","interfaces":[{"interface":"back","average":{"1min":0.77500000000000002,"5min":0.67800000000000005,"15min":0.66200000000000003},"min":{"1min":0.501,"5min":0.436,"15min":0.436},"max":{"1min":2.012,"5min":2.012,"15min":2.012},"last":0.83699999999999997},{"interface":"front","average":{"1min":0.67500000000000004,"5min":0.67900000000000005,"15min":0.67900000000000005},"min":{"1min":0.48699999999999999,"5min":0.45600000000000002,"15min":0.45600000000000002},"max":{"1min":0.92200000000000004,"5min":1.1479999999999999,"15min":1.1479999999999999},"last":0.81000000000000005}]},{"osd":7,"last update":"Fri Jul 14 19:29:19 2023","interfaces":[{"interface":"back","average":{"1min":0.71899999999999997,"5min":0.72399999999999998,"15min":0.72499999999999998},"min":{"1min":0.54600000000000004,"5min":0.53400000000000003,"15min":0.53400000000000003},"max":{"1min":0.96299999999999997,"5min":1.2070000000000001,"15min":1.2070000000000001},"last":0.94299999999999995},{"interface":"front","average":{"1min":0.75700000000000001,"5min":0.72099999999999997,"15min":0.71499999999999997},"min":{"1min":0.56299999999999994,"5min":0.47499999999999998,"15min":0.47499999999999998},"max":{"1min":0.90900000000000003,"5min":1.127,"15min":1.127},"last":0.96999999999999997}]},{"osd":9,"last update":"Fri Jul 14 19:29:44 2023","interfaces":[{"interface":"back","average":{"1min":0.75600000000000001,"5min":0.69499999999999995,"15min":0.68500000000000005},"min":{"1min":0.49099999999999999,"5min":0.49099999999999999,"15min":0.49099999999999999},"max":{"1min":0.96699999999999997,"5min":0.997,"15min":0.997},"last":0.995},{"interface":"front","average":{"1min":0.873,"5min":0.72099999999999997,"15min":0.69599999999999995},"min":{"1min":0.53300000000000003,"5min":0.48299999999999998,"15min":0.48299999999999998},"max":{"1min":2.5720000000000001,"5min":2.5720000000000001,"15min":2.5720000000000001},"last":0.98099999999999998}]},{"osd":10,"last update":"Fri Jul 14 19:28:58 2023","interfaces":[{"interface":"back","average":{"1min":0.69599999999999995,"5min":0.69599999999999995,"15min":0.69599999999999995},"min":{"1min":0.55800000000000005,"5min":0.55800000000000005,"15min":0.55800000000000005},"max":{"1min":0.94799999999999995,"5min":0.94799999999999995,"15min":0.94799999999999995},"last":1.044},{"interface":"front","average":{"1min":0.72299999999999998,"5min":0.72299999999999998,"15min":0.72299999999999998},"min":{"1min":0.55200000000000005,"5min":0.55200000000000005,"15min":0.55200000000000005},"max":{"1min":0.91100000000000003,"5min":0.91100000000000003,"15min":0.91100000000000003},"last":0.95899999999999996}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.02}]}]},{"osd":7,"up_from":50,"seq":214748364837,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28488,"kb_used_data":1472,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716664,"statfs":{"total":95995035648,"available":95965863936,"internally_reserved":0,"allocated":1507328,"data_stored":1021424,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":27654411},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jul 14 19:28:59 2023","interfaces":[{"interface":"back","average":{"1min":0.60799999999999998,"5min":0.57299999999999995,"15min":0.56699999999999995},"min":{"1min":0.41599999999999998,"5min":0.40400000000000003,"15min":0.40400000000000003},"max":{"1min":0.95899999999999996,"5min":1.0349999999999999,"15min":1.0349999999999999},"last":0.45100000000000001},{"interface":"front","average":{"1min":0.56499999999999995,"5min":0.54200000000000004,"15min":0.53800000000000003},"min":{"1min":0.34899999999999998,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":1.0089999999999999,"5min":1.0089999999999999,"15min":1.0089999999999999},"last":0.40999999999999998}]},{"osd":1,"last update":"Fri Jul 14 19:28:59 2023","interfaces":[{"interface":"back","average":{"1min":0.47299999999999998,"5min":0.45600000000000002,"15min":0.45300000000000001},"min":{"1min":0.27600000000000002,"5min":0.27600000000000002,"15min":0.27600000000000002},"max":{"1min":0.76000000000000001,"5min":0.76000000000000001,"15min":0.76000000000000001},"last":0.34699999999999998},{"interface":"front","average":{"1min":0.48899999999999999,"5min":0.47699999999999998,"15min":0.47499999999999998},"min":{"1min":0.251,"5min":0.251,"15min":0.251},"max":{"1min":0.81999999999999995,"5min":0.81999999999999995,"15min":0.81999999999999995},"last":0.375}]},{"osd":2,"last update":"Fri Jul 14 19:28:59 2023","interfaces":[{"interface":"back","average":{"1min":0.64700000000000002,"5min":0.621,"15min":0.61699999999999999},"min":{"1min":0.41399999999999998,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":1.091,"5min":1.091,"15min":1.091},"last":0.51300000000000001},{"interface":"front","average":{"1min":0.63800000000000001,"5min":0.61799999999999999,"15min":0.61499999999999999},"min":{"1min":0.35199999999999998,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":1.1040000000000001,"5min":1.1040000000000001,"15min":1.1040000000000001},"last":0.55900000000000005}]},{"osd":3,"last update":"Fri Jul 14 19:28:59 2023","interfaces":[{"interface":"back","average":{"1min":0.624,"5min":0.57999999999999996,"15min":0.57299999999999995},"min":{"1min":0.45100000000000001,"5min":0.38800000000000001,"15min":0.38800000000000001},"max":{"1min":0.90200000000000002,"5min":1.0109999999999999,"15min":1.0109999999999999},"last":0.44600000000000001},{"interface":"front","average":{"1min":0.64900000000000002,"5min":0.621,"15min":0.61599999999999999},"min":{"1min":0.495,"5min":0.47299999999999998,"15min":0.47299999999999998},"max":{"1min":0.88500000000000001,"5min":0.88600000000000001,"15min":0.88600000000000001},"last":0.33000000000000002}]},{"osd":4,"last update":"Fri Jul 14 19:28:59 2023","interfaces":[{"interface":"back","average":{"1min":0.60799999999999998,"5min":0.58899999999999997,"15min":0.58599999999999997},"min":{"1min":0.36099999999999999,"5min":0.36099999999999999,"15min":0.36099999999999999},"max":{"1min":1.125,"5min":1.125,"15min":1.125},"last":0.54800000000000004},{"interface":"front","average":{"1min":0.58799999999999997,"5min":0.54200000000000004,"15min":0.53400000000000003},"min":{"1min":0.33600000000000002,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":0.879,"5min":0.879,"15min":0.879},"last":0.41899999999999998}]},{"osd":5,"last update":"Fri Jul 14 19:28:59 2023","interfaces":[{"interface":"back","average":{"1min":0.69899999999999995,"5min":0.68300000000000005,"15min":0.68000000000000005},"min":{"1min":0.38300000000000001,"5min":0.38300000000000001,"15min":0.38300000000000001},"max":{"1min":1.173,"5min":1.173,"15min":1.173},"last":0.61899999999999999},{"interface":"front","average":{"1min":0.69299999999999995,"5min":0.69299999999999995,"15min":0.69299999999999995},"min":{"1min":0.40200000000000002,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":1.0329999999999999,"5min":1.0329999999999999,"15min":1.0329999999999999},"last":0.60299999999999998}]},{"osd":6,"last update":"Fri Jul 14 19:28:59 2023","interfaces":[{"interface":"back","average":{"1min":0.72699999999999998,"5min":0.68799999999999994,"15min":0.68200000000000005},"min":{"1min":0.48499999999999999,"5min":0.42799999999999999,"15min":0.42799999999999999},"max":{"1min":1.0309999999999999,"5min":1.0309999999999999,"15min":1.0309999999999999},"last":0.48799999999999999},{"interface":"front","average":{"1min":0.71899999999999997,"5min":0.69299999999999995,"15min":0.68899999999999995},"min":{"1min":0.50800000000000001,"5min":0.443,"15min":0.443},"max":{"1min":0.93999999999999995,"5min":1.0209999999999999,"15min":1.0209999999999999},"last":0.47399999999999998}]},{"osd":8,"last update":"Fri Jul 14 19:29:22 2023","interfaces":[{"interface":"back","average":{"1min":0.72399999999999998,"5min":0.748,"15min":0.752},"min":{"1min":0.44600000000000001,"5min":0.44600000000000001,"15min":0.44600000000000001},"max":{"1min":0.92700000000000005,"5min":1.0720000000000001,"15min":1.0720000000000001},"last":0.69799999999999995},{"interface":"front","average":{"1min":0.71399999999999997,"5min":0.745,"15min":0.75},"min":{"1min":0.42899999999999999,"5min":0.42899999999999999,"15min":0.42899999999999999},"max":{"1min":0.96699999999999997,"5min":1.141,"15min":1.141},"last":0.59199999999999997}]},{"osd":9,"last update":"Fri Jul 14 19:29:43 2023","interfaces":[{"interface":"back","average":{"1min":0.65200000000000002,"5min":0.72299999999999998,"15min":0.73499999999999999},"min":{"1min":0.505,"5min":0.505,"15min":0.505},"max":{"1min":0.99099999999999999,"5min":1.0489999999999999,"15min":1.0489999999999999},"last":0.53600000000000003},{"interface":"front","average":{"1min":0.69999999999999996,"5min":0.747,"15min":0.755},"min":{"1min":0.52000000000000002,"5min":0.52000000000000002,"15min":0.52000000000000002},"max":{"1min":0.95399999999999996,"5min":1.1519999999999999,"15min":1.1519999999999999},"last":0.65500000000000003}]},{"osd":10,"last update":"Fri Jul 14 19:28:59 2023","interfaces":[{"interface":"back","average":{"1min":0.72999999999999998,"5min":0.72999999999999998,"15min":0.72999999999999998},"min":{"1min":0.38700000000000001,"5min":0.38700000000000001,"15min":0.38700000000000001},"max":{"1min":1.077,"5min":1.077,"15min":1.077},"last":0.57899999999999996},{"interface":"front","average":{"1min":0.74199999999999999,"5min":0.74199999999999999,"15min":0.74199999999999999},"min":{"1min":0.38500000000000001,"5min":0.38500000000000001,"15min":0.38500000000000001},"max":{"1min":1.117,"5min":1.117,"15min":1.117},"last":0.72499999999999998}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.751}]}]},{"osd":6,"up_from":45,"seq":193273528362,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27896,"kb_used_data":880,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717256,"statfs":{"total":95995035648,"available":95966470144,"internally_reserved":0,"allocated":901120,"data_stored":428670,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":27654411},"hb_peers":[0,1,2,3,4,5,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jul 14 19:29:40 2023","interfaces":[{"interface":"back","average":{"1min":0.36499999999999999,"5min":0.38100000000000001,"15min":0.375},"min":{"1min":0.27000000000000002,"5min":0.23400000000000001,"15min":0.23400000000000001},"max":{"1min":0.50800000000000001,"5min":0.63,"15min":0.63},"last":0.33900000000000002},{"interface":"front","average":{"1min":0.35399999999999998,"5min":0.38400000000000001,"15min":0.38100000000000001},"min":{"1min":0.248,"5min":0.248,"15min":0.248},"max":{"1min":0.47399999999999998,"5min":0.63900000000000001,"15min":0.63900000000000001},"last":0.26800000000000002}]},{"osd":1,"last update":"Fri Jul 14 19:29:40 2023","interfaces":[{"interface":"back","average":{"1min":0.58899999999999997,"5min":0.55300000000000005,"15min":0.52000000000000002},"min":{"1min":0.41699999999999998,"5min":0.29899999999999999,"15min":0.29899999999999999},"max":{"1min":0.74199999999999999,"5min":0.83999999999999997,"15min":0.83999999999999997},"last":0.54400000000000004},{"interface":"front","average":{"1min":0.52800000000000002,"5min":0.51300000000000001,"15min":0.495},"min":{"1min":0.33300000000000002,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":0.755,"5min":0.872,"15min":0.872},"last":0.36399999999999999}]},{"osd":2,"last update":"Fri Jul 14 19:29:40 2023","interfaces":[{"interface":"back","average":{"1min":0.50800000000000001,"5min":0.57199999999999995,"15min":0.58399999999999996},"min":{"1min":0.39600000000000002,"5min":0.27600000000000002,"15min":0.27600000000000002},"max":{"1min":0.627,"5min":0.76800000000000002,"15min":0.76800000000000002},"last":0.50900000000000001},{"interface":"front","average":{"1min":0.55400000000000005,"5min":0.60099999999999998,"15min":0.60499999999999998},"min":{"1min":0.38600000000000001,"5min":0.378,"15min":0.378},"max":{"1min":0.80500000000000005,"5min":0.83299999999999996,"15min":0.83299999999999996},"last":0.63200000000000001}]},{"osd":3,"last update":"Fri Jul 14 19:29:40 2023","interfaces":[{"interface":"back","average":{"1min":0.434,"5min":0.46700000000000003,"15min":0.47299999999999998},"min":{"1min":0.29299999999999998,"5min":0.29299999999999998,"15min":0.29299999999999998},"max":{"1min":0.66800000000000004,"5min":0.78500000000000003,"15min":0.78500000000000003},"last":0.40300000000000002},{"interface":"front","average":{"1min":0.378,"5min":0.42899999999999999,"15min":0.435},"min":{"1min":0.28499999999999998,"5min":0.28499999999999998,"15min":0.28499999999999998},"max":{"1min":0.53100000000000003,"5min":0.75700000000000001,"15min":0.75700000000000001},"last":0.30099999999999999}]},{"osd":4,"last update":"Fri Jul 14 19:29:40 2023","interfaces":[{"interface":"back","average":{"1min":0.53100000000000003,"5min":0.53800000000000003,"15min":0.52300000000000002},"min":{"1min":0.36699999999999999,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":0.76700000000000002,"5min":0.82899999999999996,"15min":0.82899999999999996},"last":0.41599999999999998},{"interface":"front","average":{"1min":0.58799999999999997,"5min":0.61599999999999999,"15min":0.60999999999999999},"min":{"1min":0.45000000000000001,"5min":0.38700000000000001,"15min":0.38700000000000001},"max":{"1min":0.78100000000000003,"5min":0.96899999999999997,"15min":0.96899999999999997},"last":0.435}]},{"osd":5,"last update":"Fri Jul 14 19:29:40 2023","interfaces":[{"interface":"back","average":{"1min":0.53600000000000003,"5min":0.56599999999999995,"15min":0.57199999999999995},"min":{"1min":0.36899999999999999,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":0.66400000000000003,"5min":0.81799999999999995,"15min":0.81799999999999995},"last":0.48199999999999998},{"interface":"front","average":{"1min":0.56100000000000005,"5min":0.62,"15min":0.628},"min":{"1min":0.437,"5min":0.437,"15min":0.437},"max":{"1min":0.69299999999999995,"5min":0.88500000000000001,"15min":0.88500000000000001},"last":0.56100000000000005}]},{"osd":7,"last update":"Fri Jul 14 19:28:59 2023","interfaces":[{"interface":"back","average":{"1min":0.68600000000000005,"5min":0.68100000000000005,"15min":0.68000000000000005},"min":{"1min":0.47599999999999998,"5min":0.44500000000000001,"15min":0.44500000000000001},"max":{"1min":1.0029999999999999,"5min":1.0029999999999999,"15min":1.0029999999999999},"last":0.57399999999999995},{"interface":"front","average":{"1min":0.69699999999999995,"5min":0.69699999999999995,"15min":0.69699999999999995},"min":{"1min":0.50900000000000001,"5min":0.46700000000000003,"15min":0.46700000000000003},"max":{"1min":0.86599999999999999,"5min":1.0189999999999999,"15min":1.0189999999999999},"last":0.46999999999999997}]},{"osd":8,"last update":"Fri Jul 14 19:29:17 2023","interfaces":[{"interface":"back","average":{"1min":0.66500000000000004,"5min":0.70099999999999996,"15min":0.70699999999999996},"min":{"1min":0.48899999999999999,"5min":0.48899999999999999,"15min":0.48899999999999999},"max":{"1min":0.90200000000000002,"5min":1.1000000000000001,"15min":1.1000000000000001},"last":0.60999999999999999},{"interface":"front","average":{"1min":0.60799999999999998,"5min":0.59999999999999998,"15min":0.59799999999999998},"min":{"1min":0.46999999999999997,"5min":0.434,"15min":0.434},"max":{"1min":0.83299999999999996,"5min":0.83299999999999996,"15min":0.83299999999999996},"last":0.66100000000000003}]},{"osd":9,"last update":"Fri Jul 14 19:29:43 2023","interfaces":[{"interface":"back","average":{"1min":0.54100000000000004,"5min":0.629,"15min":0.64300000000000002},"min":{"1min":0.40600000000000003,"5min":0.40600000000000003,"15min":0.40600000000000003},"max":{"1min":0.68200000000000005,"5min":0.90600000000000003,"15min":0.90600000000000003},"last":0.44600000000000001},{"interface":"front","average":{"1min":0.58299999999999996,"5min":0.73099999999999998,"15min":0.75600000000000001},"min":{"1min":0.42399999999999999,"5min":0.42399999999999999,"15min":0.42399999999999999},"max":{"1min":0.77000000000000002,"5min":1.1379999999999999,"15min":1.1379999999999999},"last":0.495}]},{"osd":10,"last update":"Fri Jul 14 19:28:59 2023","interfaces":[{"interface":"back","average":{"1min":0.63900000000000001,"5min":0.63900000000000001,"15min":0.63900000000000001},"min":{"1min":0.505,"5min":0.505,"15min":0.505},"max":{"1min":0.82999999999999996,"5min":0.82999999999999996,"15min":0.82999999999999996},"last":0.52200000000000002},{"interface":"front","average":{"1min":0.76800000000000002,"5min":0.76800000000000002,"15min":0.76800000000000002},"min":{"1min":0.58999999999999997,"5min":0.58999999999999997,"15min":0.58999999999999997},"max":{"1min":1.0329999999999999,"5min":1.0329999999999999,"15min":1.0329999999999999},"last":0.59299999999999997}]}]},{"osd":1,"up_from":13,"seq":55834574911,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27952,"kb_used_data":936,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717200,"statfs":{"total":95995035648,"available":95966412800,"internally_reserved":0,"allocated":958464,"data_stored":483117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":27654411},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jul 14 19:28:58 2023","interfaces":[{"interface":"back","average":{"1min":0.53800000000000003,"5min":0.498,"15min":0.47099999999999997},"min":{"1min":0.43099999999999999,"5min":0.215,"15min":0.215},"max":{"1min":0.82399999999999995,"5min":1.6970000000000001,"15min":1.6970000000000001},"last":0.71499999999999997},{"interface":"front","average":{"1min":0.58599999999999997,"5min":0.48699999999999999,"15min":0.46100000000000002},"min":{"1min":0.39100000000000001,"5min":0.23699999999999999,"15min":0.23699999999999999},"max":{"1min":0.82899999999999996,"5min":0.82899999999999996,"15min":0.82899999999999996},"last":0.55300000000000005}]},{"osd":2,"last update":"Fri Jul 14 19:29:15 2023","interfaces":[{"interface":"back","average":{"1min":0.625,"5min":0.52400000000000002,"15min":0.496},"min":{"1min":0.40400000000000003,"5min":0.31,"15min":0.31},"max":{"1min":0.93400000000000005,"5min":0.93400000000000005,"15min":0.93400000000000005},"last":0.73399999999999999},{"interface":"front","average":{"1min":0.61199999999999999,"5min":0.54900000000000004,"15min":0.50600000000000001},"min":{"1min":0.434,"5min":0.317,"15min":0.317},"max":{"1min":0.84199999999999997,"5min":1.74,"15min":1.74},"last":0.69299999999999995}]},{"osd":3,"last update":"Fri Jul 14 19:29:38 2023","interfaces":[{"interface":"back","average":{"1min":0.61599999999999999,"5min":0.57199999999999995,"15min":0.54200000000000004},"min":{"1min":0.38600000000000001,"5min":0.32200000000000001,"15min":0.32200000000000001},"max":{"1min":0.78400000000000003,"5min":1.8460000000000001,"15min":1.8460000000000001},"last":0.58099999999999996},{"interface":"front","average":{"1min":0.59999999999999998,"5min":0.56999999999999995,"15min":0.55200000000000005},"min":{"1min":0.38400000000000001,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.752,"5min":0.81499999999999995,"15min":0.81499999999999995},"last":0.78200000000000003}]},{"osd":4,"last update":"Fri Jul 14 19:28:58 2023","interfaces":[{"interface":"back","average":{"1min":0.51400000000000001,"5min":0.49199999999999999,"15min":0.502},"min":{"1min":0.314,"5min":0.29899999999999999,"15min":0.29899999999999999},"max":{"1min":0.78100000000000003,"5min":2.2029999999999998,"15min":2.2029999999999998},"last":0.60199999999999998},{"interface":"front","average":{"1min":0.53700000000000003,"5min":0.495,"15min":0.49199999999999999},"min":{"1min":0.309,"5min":0.26400000000000001,"15min":0.26400000000000001},"max":{"1min":0.90400000000000003,"5min":1.901,"15min":1.901},"last":0.50700000000000001}]},{"osd":5,"last update":"Fri Jul 14 19:29:15 2023","interfaces":[{"interface":"back","average":{"1min":0.68400000000000005,"5min":0.59799999999999998,"15min":0.57299999999999995},"min":{"1min":0.373,"5min":0.308,"15min":0.308},"max":{"1min":0.88100000000000001,"5min":0.88100000000000001,"15min":0.88100000000000001},"last":0.65300000000000002},{"interface":"front","average":{"1min":0.77700000000000002,"5min":0.68100000000000005,"15min":0.66100000000000003},"min":{"1min":0.57999999999999996,"5min":0.42699999999999999,"15min":0.42699999999999999},"max":{"1min":0.94899999999999995,"5min":1.8,"15min":1.8},"last":0.76400000000000001}]},{"osd":6,"last update":"Fri Jul 14 19:29:42 2023","interfaces":[{"interface":"back","average":{"1min":0.69299999999999995,"5min":0.64700000000000002,"15min":0.63700000000000001},"min":{"1min":0.48599999999999999,"5min":0.40999999999999998,"15min":0.40999999999999998},"max":{"1min":0.94699999999999995,"5min":1.8300000000000001,"15min":1.8300000000000001},"last":0.94699999999999995},{"interface":"front","average":{"1min":0.69099999999999995,"5min":0.66800000000000004,"15min":0.65900000000000003},"min":{"1min":0.54800000000000004,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":0.83299999999999996,"5min":1.875,"15min":1.875},"last":0.75}]},{"osd":7,"last update":"Fri Jul 14 19:28:58 2023","interfaces":[{"interface":"back","average":{"1min":0.67400000000000004,"5min":0.56599999999999995,"15min":0.54800000000000004},"min":{"1min":0.36199999999999999,"5min":0.36199999999999999,"15min":0.36199999999999999},"max":{"1min":0.95399999999999996,"5min":0.95399999999999996,"15min":0.95399999999999996},"last":0.81000000000000005},{"interface":"front","average":{"1min":0.70199999999999996,"5min":0.60799999999999998,"15min":0.59199999999999997},"min":{"1min":0.39200000000000002,"5min":0.38400000000000001,"15min":0.38400000000000001},"max":{"1min":1.24,"5min":1.24,"15min":1.24},"last":0.67500000000000004}]},{"osd":8,"last update":"Fri Jul 14 19:29:21 2023","interfaces":[{"interface":"back","average":{"1min":0.753,"5min":0.70399999999999996,"15min":0.69599999999999995},"min":{"1min":0.45500000000000002,"5min":0.42199999999999999,"15min":0.42199999999999999},"max":{"1min":1.091,"5min":1.091,"15min":1.091},"last":0.84799999999999998},{"interface":"front","average":{"1min":0.83599999999999997,"5min":0.75900000000000001,"15min":0.746},"min":{"1min":0.52400000000000002,"5min":0.505,"15min":0.505},"max":{"1min":1.2130000000000001,"5min":1.2130000000000001,"15min":1.2130000000000001},"last":0.89800000000000002}]},{"osd":9,"last update":"Fri Jul 14 19:29:42 2023","interfaces":[{"interface":"back","average":{"1min":0.96799999999999997,"5min":0.81399999999999995,"15min":0.78800000000000003},"min":{"1min":0.65100000000000002,"5min":0.52100000000000002,"15min":0.52100000000000002},"max":{"1min":3.0489999999999999,"5min":3.0489999999999999,"15min":3.0489999999999999},"last":0.92300000000000004},{"interface":"front","average":{"1min":0.92700000000000005,"5min":0.79400000000000004,"15min":0.77200000000000002},"min":{"1min":0.64400000000000002,"5min":0.51200000000000001,"15min":0.51200000000000001},"max":{"1min":3.0630000000000002,"5min":3.0630000000000002,"15min":3.0630000000000002},"last":0.79800000000000004}]},{"osd":10,"last update":"Fri Jul 14 19:28:58 2023","interfaces":[{"interface":"back","average":{"1min":0.73699999999999999,"5min":0.73699999999999999,"15min":0.73699999999999999},"min":{"1min":0.44800000000000001,"5min":0.44800000000000001,"15min":0.44800000000000001},"max":{"1min":1.079,"5min":1.079,"15min":1.079},"last":0.82599999999999996},{"interface":"front","average":{"1min":0.81799999999999995,"5min":0.81799999999999995,"15min":0.81799999999999995},"min":{"1min":0.58799999999999997,"5min":0.58799999999999997,"15min":0.58799999999999997},"max":{"1min":1.127,"5min":1.127,"15min":1.127},"last":0.85999999999999999}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.88300000000000001}]}]},{"osd":0,"up_from":8,"seq":34359738435,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27916,"kb_used_data":900,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717236,"statfs":{"total":95995035648,"available":95966449664,"internally_reserved":0,"allocated":921600,"data_stored":429935,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":27654411},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Jul 14 19:28:52 2023","interfaces":[{"interface":"back","average":{"1min":0.629,"5min":0.52600000000000002,"15min":0.503},"min":{"1min":0.35999999999999999,"5min":0.28000000000000003,"15min":0.28000000000000003},"max":{"1min":1.79,"5min":1.79,"15min":1.79},"last":0.53300000000000003},{"interface":"front","average":{"1min":0.57199999999999995,"5min":0.51700000000000002,"15min":0.505},"min":{"1min":0.32000000000000001,"5min":0.313,"15min":0.313},"max":{"1min":1.605,"5min":1.605,"15min":1.605},"last":0.56799999999999995}]},{"osd":2,"last update":"Fri Jul 14 19:29:18 2023","interfaces":[{"interface":"back","average":{"1min":0.52200000000000002,"5min":0.54900000000000004,"15min":0.51900000000000002},"min":{"1min":0.33900000000000002,"5min":0.33900000000000002,"15min":0.33900000000000002},"max":{"1min":0.67500000000000004,"5min":0.77700000000000002,"15min":0.77700000000000002},"last":0.44600000000000001},{"interface":"front","average":{"1min":0.55200000000000005,"5min":0.54900000000000004,"15min":0.52400000000000002},"min":{"1min":0.34499999999999997,"5min":0.29999999999999999,"15min":0.29999999999999999},"max":{"1min":0.79900000000000004,"5min":0.81399999999999995,"15min":0.81399999999999995},"last":0.46100000000000002}]},{"osd":3,"last update":"Fri Jul 14 19:27:35 2023","interfaces":[{"interface":"back","average":{"1min":0.439,"5min":0.41099999999999998,"15min":0.40600000000000003},"min":{"1min":0.29399999999999998,"5min":0.26400000000000001,"15min":0.26400000000000001},"max":{"1min":0.72299999999999998,"5min":0.72299999999999998,"15min":0.72299999999999998},"last":0.32900000000000001},{"interface":"front","average":{"1min":0.44900000000000001,"5min":0.437,"15min":0.435},"min":{"1min":0.245,"5min":0.24099999999999999,"15min":0.24099999999999999},"max":{"1min":0.61499999999999999,"5min":0.61499999999999999,"15min":0.61499999999999999},"last":0.41299999999999998}]},{"osd":4,"last update":"Fri Jul 14 19:29:02 2023","interfaces":[{"interface":"back","average":{"1min":0.57099999999999995,"5min":0.53900000000000003,"15min":0.52500000000000002},"min":{"1min":0.38800000000000001,"5min":0.377,"15min":0.377},"max":{"1min":1.139,"5min":1.139,"15min":1.139},"last":0.52300000000000002},{"interface":"front","average":{"1min":0.58499999999999996,"5min":0.55000000000000004,"15min":0.53700000000000003},"min":{"1min":0.34799999999999998,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":1.2190000000000001,"5min":1.2190000000000001,"15min":1.2190000000000001},"last":0.58599999999999997}]},{"osd":5,"last update":"Fri Jul 14 19:29:18 2023","interfaces":[{"interface":"back","average":{"1min":0.56899999999999995,"5min":0.59899999999999998,"15min":0.59899999999999998},"min":{"1min":0.41699999999999998,"5min":0.41699999999999998,"15min":0.41699999999999998},"max":{"1min":0.71399999999999997,"5min":0.81799999999999995,"15min":0.81799999999999995},"last":0.51100000000000001},{"interface":"front","average":{"1min":0.57999999999999996,"5min":0.60999999999999999,"15min":0.60899999999999999},"min":{"1min":0.38,"5min":0.38,"15min":0.38},"max":{"1min":0.76200000000000001,"5min":0.85499999999999998,"15min":0.85499999999999998},"last":0.59599999999999997}]},{"osd":6,"last update":"Fri Jul 14 19:29:38 2023","interfaces":[{"interface":"back","average":{"1min":0.56000000000000005,"5min":0.50800000000000001,"15min":0.5},"min":{"1min":0.30099999999999999,"5min":0.25900000000000001,"15min":0.25900000000000001},"max":{"1min":3.2650000000000001,"5min":3.2650000000000001,"15min":3.2650000000000001},"last":0.34799999999999998},{"interface":"front","average":{"1min":0.56200000000000006,"5min":0.52600000000000002,"15min":0.51200000000000001},"min":{"1min":0.32400000000000001,"5min":0.32400000000000001,"15min":0.32400000000000001},"max":{"1min":2.9129999999999998,"5min":2.9129999999999998,"15min":2.9129999999999998},"last":0.373}]},{"osd":7,"last update":"Fri Jul 14 19:27:57 2023","interfaces":[{"interface":"back","average":{"1min":0.60299999999999998,"5min":0.60299999999999998,"15min":0.60299999999999998},"min":{"1min":0.41999999999999998,"5min":0.41999999999999998,"15min":0.41999999999999998},"max":{"1min":0.83599999999999997,"5min":0.83599999999999997,"15min":0.83599999999999997},"last":0.52100000000000002},{"interface":"front","average":{"1min":0.65200000000000002,"5min":0.65200000000000002,"15min":0.65200000000000002},"min":{"1min":0.47299999999999998,"5min":0.47299999999999998,"15min":0.47299999999999998},"max":{"1min":0.89500000000000002,"5min":0.89500000000000002,"15min":0.89500000000000002},"last":0.40799999999999997}]},{"osd":8,"last update":"Fri Jul 14 19:29:18 2023","interfaces":[{"interface":"back","average":{"1min":0.58699999999999997,"5min":0.67100000000000004,"15min":0.68500000000000005},"min":{"1min":0.36599999999999999,"5min":0.36599999999999999,"15min":0.36599999999999999},"max":{"1min":0.73899999999999999,"5min":0.80700000000000005,"15min":0.80700000000000005},"last":0.47299999999999998},{"interface":"front","average":{"1min":0.63200000000000001,"5min":0.69699999999999995,"15min":0.70799999999999996},"min":{"1min":0.40699999999999997,"5min":0.40699999999999997,"15min":0.40699999999999997},"max":{"1min":0.81899999999999995,"5min":0.86499999999999999,"15min":0.86499999999999999},"last":0.54400000000000004}]},{"osd":9,"last update":"Fri Jul 14 19:29:40 2023","interfaces":[{"interface":"back","average":{"1min":0.56999999999999995,"5min":0.622,"15min":0.63},"min":{"1min":0.34100000000000003,"5min":0.34100000000000003,"15min":0.34100000000000003},"max":{"1min":1.254,"5min":1.254,"15min":1.254},"last":0.39800000000000002},{"interface":"front","average":{"1min":0.54800000000000004,"5min":0.623,"15min":0.63500000000000001},"min":{"1min":0.35799999999999998,"5min":0.32300000000000001,"15min":0.32300000000000001},"max":{"1min":0.78300000000000003,"5min":0.96099999999999997,"15min":0.96099999999999997},"last":0.48999999999999999}]},{"osd":10,"last update":"Fri Jul 14 19:29:02 2023","interfaces":[{"interface":"back","average":{"1min":0.754,"5min":0.754,"15min":0.754},"min":{"1min":0.45900000000000002,"5min":0.45900000000000002,"15min":0.45900000000000002},"max":{"1min":1.776,"5min":1.776,"15min":1.776},"last":0.625},{"interface":"front","average":{"1min":0.72499999999999998,"5min":0.72499999999999998,"15min":0.72499999999999998},"min":{"1min":0.502,"5min":0.502,"15min":0.502},"max":{"1min":1.522,"5min":1.522,"15min":1.522},"last":0.65200000000000002}]},{"osd":11,"last update":"Fri Jul 14 19:29:23 2023","interfaces":[{"interface":"back","average":{"1min":0.65900000000000003,"5min":0.65900000000000003,"15min":0.65900000000000003},"min":{"1min":0.48299999999999998,"5min":0.48299999999999998,"15min":0.48299999999999998},"max":{"1min":0.85999999999999999,"5min":0.85999999999999999,"15min":0.85999999999999999},"last":0.61199999999999999},{"interface":"front","average":{"1min":0.77300000000000002,"5min":0.77300000000000002,"15min":0.77300000000000002},"min":{"1min":0.53600000000000003,"5min":0.53600000000000003,"15min":0.53600000000000003},"max":{"1min":2.73,"5min":2.73,"15min":2.73},"last":0.64500000000000002}]}]},{"osd":2,"up_from":18,"seq":77309411386,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27976,"kb_used_data":960,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717176,"statfs":{"total":95995035648,"available":95966388224,"internally_reserved":0,"allocated":983040,"data_stored":484809,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":27654411},"hb_peers":[0,1,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jul 14 19:29:16 2023","interfaces":[{"interface":"back","average":{"1min":0.58199999999999996,"5min":0.52200000000000002,"15min":0.498},"min":{"1min":0.34599999999999997,"5min":0.312,"15min":0.312},"max":{"1min":0.99199999999999999,"5min":0.99199999999999999,"15min":0.99199999999999999},"last":0.38200000000000001},{"interface":"front","average":{"1min":0.496,"5min":0.49299999999999999,"15min":0.48799999999999999},"min":{"1min":0.33600000000000002,"5min":0.31900000000000001,"15min":0.31900000000000001},"max":{"1min":0.66800000000000004,"5min":0.69699999999999995,"15min":0.69699999999999995},"last":0.58599999999999997}]},{"osd":1,"last update":"Fri Jul 14 19:29:16 2023","interfaces":[{"interface":"back","average":{"1min":0.626,"5min":0.55700000000000005,"15min":0.52300000000000002},"min":{"1min":0.42699999999999999,"5min":0.316,"15min":0.316},"max":{"1min":0.93100000000000005,"5min":0.93100000000000005,"15min":0.93100000000000005},"last":0.437},{"interface":"front","average":{"1min":0.66600000000000004,"5min":0.57699999999999996,"15min":0.53300000000000003},"min":{"1min":0.38300000000000001,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":0.86499999999999999,"5min":0.91100000000000003,"15min":0.91100000000000003},"last":0.68700000000000006}]},{"osd":3,"last update":"Fri Jul 14 19:29:39 2023","interfaces":[{"interface":"back","average":{"1min":0.65000000000000002,"5min":0.59499999999999997,"15min":0.56100000000000005},"min":{"1min":0.39300000000000002,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":0.82999999999999996,"5min":0.91700000000000004,"15min":0.91700000000000004},"last":0.52500000000000002},{"interface":"front","average":{"1min":0.58699999999999997,"5min":0.55500000000000005,"15min":0.53700000000000003},"min":{"1min":0.41999999999999998,"5min":0.34699999999999998,"15min":0.34699999999999998},"max":{"1min":0.82399999999999995,"5min":0.82399999999999995,"15min":0.82399999999999995},"last":0.54400000000000004}]},{"osd":4,"last update":"Fri Jul 14 19:27:59 2023","interfaces":[{"interface":"back","average":{"1min":0.53800000000000003,"5min":0.52200000000000002,"15min":0.52000000000000002},"min":{"1min":0.38800000000000001,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":0.68400000000000005,"5min":0.70399999999999996,"15min":0.70399999999999996},"last":0.72999999999999998},{"interface":"front","average":{"1min":0.63600000000000001,"5min":0.58399999999999996,"15min":0.57599999999999996},"min":{"1min":0.40100000000000002,"5min":0.40100000000000002,"15min":0.40100000000000002},"max":{"1min":0.81699999999999995,"5min":0.81699999999999995,"15min":0.81699999999999995},"last":0.46200000000000002}]},{"osd":5,"last update":"Fri Jul 14 19:29:16 2023","interfaces":[{"interface":"back","average":{"1min":0.50600000000000001,"5min":0.44900000000000001,"15min":0.43099999999999999},"min":{"1min":0.33100000000000002,"5min":0.26500000000000001,"15min":0.26500000000000001},"max":{"1min":0.82199999999999995,"5min":0.82199999999999995,"15min":0.82199999999999995},"last":0.622},{"interface":"front","average":{"1min":0.56799999999999995,"5min":0.499,"15min":0.47299999999999998},"min":{"1min":0.36199999999999999,"5min":0.313,"15min":0.313},"max":{"1min":0.84999999999999998,"5min":0.84999999999999998,"15min":0.84999999999999998},"last":0.60599999999999998}]},{"osd":6,"last update":"Fri Jul 14 19:28:43 2023","interfaces":[{"interface":"back","average":{"1min":0.59899999999999998,"5min":0.55500000000000005,"15min":0.54800000000000004},"min":{"1min":0.33500000000000002,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":0.93400000000000005,"5min":0.93400000000000005,"15min":0.93400000000000005},"last":0.57199999999999995},{"interface":"front","average":{"1min":0.69199999999999995,"5min":0.60899999999999999,"15min":0.59499999999999997},"min":{"1min":0.46999999999999997,"5min":0.371,"15min":0.371},"max":{"1min":0.92000000000000004,"5min":0.92000000000000004,"15min":0.92000000000000004},"last":0.76600000000000001}]},{"osd":7,"last update":"Fri Jul 14 19:27:59 2023","interfaces":[{"interface":"back","average":{"1min":0.622,"5min":0.622,"15min":0.622},"min":{"1min":0.44600000000000001,"5min":0.44600000000000001,"15min":0.44600000000000001},"max":{"1min":0.91900000000000004,"5min":0.91900000000000004,"15min":0.91900000000000004},"last":0.59899999999999998},{"interface":"front","average":{"1min":0.67500000000000004,"5min":0.67500000000000004,"15min":0.67500000000000004},"min":{"1min":0.433,"5min":0.433,"15min":0.433},"max":{"1min":0.96399999999999997,"5min":0.96399999999999997,"15min":0.96399999999999997},"last":0.66000000000000003}]},{"osd":8,"last update":"Fri Jul 14 19:29:16 2023","interfaces":[{"interface":"back","average":{"1min":0.63400000000000001,"5min":0.628,"15min":0.627},"min":{"1min":0.35499999999999998,"5min":0.35499999999999998,"15min":0.35499999999999998},"max":{"1min":0.91600000000000004,"5min":0.91600000000000004,"15min":0.91600000000000004},"last":0.80100000000000005},{"interface":"front","average":{"1min":0.60099999999999998,"5min":0.62,"15min":0.623},"min":{"1min":0.374,"5min":0.374,"15min":0.374},"max":{"1min":0.78600000000000003,"5min":0.94199999999999995,"15min":0.94199999999999995},"last":0.65600000000000003}]},{"osd":9,"last update":"Fri Jul 14 19:28:43 2023","interfaces":[{"interface":"back","average":{"1min":0.67800000000000005,"5min":0.67800000000000005,"15min":0.67800000000000005},"min":{"1min":0.45300000000000001,"5min":0.45300000000000001,"15min":0.45300000000000001},"max":{"1min":0.90700000000000003,"5min":0.90700000000000003,"15min":0.90700000000000003},"last":0.74399999999999999},{"interface":"front","average":{"1min":0.65300000000000002,"5min":0.65300000000000002,"15min":0.65300000000000002},"min":{"1min":0.55200000000000005,"5min":0.55200000000000005,"15min":0.55200000000000005},"max":{"1min":0.79700000000000004,"5min":0.79700000000000004,"15min":0.79700000000000004},"last":0.67300000000000004}]},{"osd":10,"last update":"Fri Jul 14 19:28:59 2023","interfaces":[{"interface":"back","average":{"1min":0.63200000000000001,"5min":0.63200000000000001,"15min":0.63200000000000001},"min":{"1min":0.46800000000000003,"5min":0.46800000000000003,"15min":0.46800000000000003},"max":{"1min":0.79700000000000004,"5min":0.79700000000000004,"15min":0.79700000000000004},"last":0.70299999999999996},{"interface":"front","average":{"1min":0.745,"5min":0.745,"15min":0.745},"min":{"1min":0.61899999999999999,"5min":0.61899999999999999,"15min":0.61899999999999999},"max":{"1min":0.88900000000000001,"5min":0.88900000000000001,"15min":0.88900000000000001},"last":0.86799999999999999}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.83399999999999996}]}]},{"osd":3,"up_from":25,"seq":107374182454,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27908,"kb_used_data":892,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717244,"statfs":{"total":95995035648,"available":95966457856,"internally_reserved":0,"allocated":913408,"data_stored":428656,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":27654411},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jul 14 19:29:36 2023","interfaces":[{"interface":"back","average":{"1min":0.35799999999999998,"5min":0.34899999999999998,"15min":0.32500000000000001},"min":{"1min":0.251,"5min":0.221,"15min":0.221},"max":{"1min":0.57599999999999996,"5min":0.61299999999999999,"15min":0.61299999999999999},"last":0.80900000000000005},{"interface":"front","average":{"1min":0.36499999999999999,"5min":0.378,"15min":0.35799999999999998},"min":{"1min":0.26600000000000001,"5min":0.23899999999999999,"15min":0.23899999999999999},"max":{"1min":0.55100000000000005,"5min":0.63600000000000001,"15min":0.63600000000000001},"last":0.34100000000000003}]},{"osd":1,"last update":"Fri Jul 14 19:29:36 2023","interfaces":[{"interface":"back","average":{"1min":0.56399999999999995,"5min":0.52400000000000002,"15min":0.499},"min":{"1min":0.34200000000000003,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":0.79600000000000004,"5min":0.84199999999999997,"15min":0.84199999999999997},"last":0.76500000000000001},{"interface":"front","average":{"1min":0.59299999999999997,"5min":0.54000000000000004,"15min":0.50900000000000001},"min":{"1min":0.40699999999999997,"5min":0.30399999999999999,"15min":0.30399999999999999},"max":{"1min":0.76200000000000001,"5min":0.76200000000000001,"15min":0.76200000000000001},"last":0.43099999999999999}]},{"osd":2,"last update":"Fri Jul 14 19:29:36 2023","interfaces":[{"interface":"back","average":{"1min":0.58299999999999996,"5min":0.57999999999999996,"15min":0.54500000000000004},"min":{"1min":0.40899999999999997,"5min":0.36199999999999999,"15min":0.36199999999999999},"max":{"1min":0.81299999999999994,"5min":0.93899999999999995,"15min":0.93899999999999995},"last":0.67800000000000005},{"interface":"front","average":{"1min":0.59599999999999997,"5min":0.56000000000000005,"15min":0.53200000000000003},"min":{"1min":0.33800000000000002,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":0.85399999999999998,"5min":0.85399999999999998,"15min":0.85399999999999998},"last":0.89900000000000002}]},{"osd":4,"last update":"Fri Jul 14 19:28:51 2023","interfaces":[{"interface":"back","average":{"1min":0.66100000000000003,"5min":0.55600000000000005,"15min":0.52200000000000002},"min":{"1min":0.44800000000000001,"5min":0.32300000000000001,"15min":0.32300000000000001},"max":{"1min":0.91100000000000003,"5min":0.98499999999999999,"15min":0.98499999999999999},"last":0.72599999999999998},{"interface":"front","average":{"1min":0.60499999999999998,"5min":0.55000000000000004,"15min":0.52900000000000003},"min":{"1min":0.45500000000000002,"5min":0.39300000000000002,"15min":0.39300000000000002},"max":{"1min":0.90200000000000002,"5min":0.90200000000000002,"15min":0.90200000000000002},"last":0.874}]},{"osd":5,"last update":"Fri Jul 14 19:29:13 2023","interfaces":[{"interface":"back","average":{"1min":0.56999999999999995,"5min":0.57999999999999996,"15min":0.57099999999999995},"min":{"1min":0.38200000000000001,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":0.79800000000000004,"5min":0.81299999999999994,"15min":0.81299999999999994},"last":0.52700000000000002},{"interface":"front","average":{"1min":0.67600000000000005,"5min":0.628,"15min":0.60199999999999998},"min":{"1min":0.44800000000000001,"5min":0.39300000000000002,"15min":0.39300000000000002},"max":{"1min":0.92200000000000004,"5min":0.98199999999999998,"15min":0.98199999999999998},"last":0.82299999999999995}]},{"osd":6,"last update":"Fri Jul 14 19:29:41 2023","interfaces":[{"interface":"back","average":{"1min":0.48699999999999999,"5min":0.503,"15min":0.48299999999999998},"min":{"1min":0.27700000000000002,"5min":0.27700000000000002,"15min":0.27700000000000002},"max":{"1min":0.88600000000000001,"5min":0.88600000000000001,"15min":0.88600000000000001},"last":0.56399999999999995},{"interface":"front","average":{"1min":0.46500000000000002,"5min":0.50900000000000001,"15min":0.51700000000000002},"min":{"1min":0.28799999999999998,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":0.79600000000000004,"5min":0.79600000000000004,"15min":0.79600000000000004},"last":0.79600000000000004}]},{"osd":7,"last update":"Fri Jul 14 19:28:57 2023","interfaces":[{"interface":"back","average":{"1min":0.69699999999999995,"5min":0.68000000000000005,"15min":0.67700000000000005},"min":{"1min":0.48999999999999999,"5min":0.41699999999999998,"15min":0.41699999999999998},"max":{"1min":0.96399999999999997,"5min":1.196,"15min":1.196},"last":0.59699999999999998},{"interface":"front","average":{"1min":0.78700000000000003,"5min":0.69199999999999995,"15min":0.67600000000000005},"min":{"1min":0.58499999999999996,"5min":0.46100000000000002,"15min":0.46100000000000002},"max":{"1min":1.0289999999999999,"5min":1.0289999999999999,"15min":1.0289999999999999},"last":0.78100000000000003}]},{"osd":8,"last update":"Fri Jul 14 19:28:17 2023","interfaces":[{"interface":"back","average":{"1min":0.69899999999999995,"5min":0.69899999999999995,"15min":0.69899999999999995},"min":{"1min":0.48199999999999998,"5min":0.48199999999999998,"15min":0.48199999999999998},"max":{"1min":0.98799999999999999,"5min":0.98799999999999999,"15min":0.98799999999999999},"last":0.751},{"interface":"front","average":{"1min":0.749,"5min":0.749,"15min":0.749},"min":{"1min":0.51800000000000002,"5min":0.51800000000000002,"15min":0.51800000000000002},"max":{"1min":0.92500000000000004,"5min":0.92500000000000004,"15min":0.92500000000000004},"last":0.68100000000000005}]},{"osd":9,"last update":"Fri Jul 14 19:29:41 2023","interfaces":[{"interface":"back","average":{"1min":0.57799999999999996,"5min":0.72799999999999998,"15min":0.753},"min":{"1min":0.36699999999999999,"5min":0.36699999999999999,"15min":0.36699999999999999},"max":{"1min":0.97299999999999998,"5min":1.0549999999999999,"15min":1.0549999999999999},"last":0.61099999999999999},{"interface":"front","average":{"1min":0.52600000000000002,"5min":0.624,"15min":0.64000000000000001},"min":{"1min":0.35499999999999998,"5min":0.35499999999999998,"15min":0.35499999999999998},"max":{"1min":0.92400000000000004,"5min":0.95099999999999996,"15min":0.95099999999999996},"last":0.92400000000000004}]},{"osd":10,"last update":"Fri Jul 14 19:29:02 2023","interfaces":[{"interface":"back","average":{"1min":0.72699999999999998,"5min":0.72699999999999998,"15min":0.72699999999999998},"min":{"1min":0.56499999999999995,"5min":0.56499999999999995,"15min":0.56499999999999995},"max":{"1min":0.92700000000000005,"5min":0.92700000000000005,"15min":0.92700000000000005},"last":0.626},{"interface":"front","average":{"1min":0.82099999999999995,"5min":0.82099999999999995,"15min":0.82099999999999995},"min":{"1min":0.59399999999999997,"5min":0.59399999999999997,"15min":0.59399999999999997},"max":{"1min":1.0880000000000001,"5min":1.0880000000000001,"15min":1.0880000000000001},"last":0.85799999999999998}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.69599999999999995}]}]},{"osd":4,"up_from":32,"seq":137438953522,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27900,"kb_used_data":884,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717252,"statfs":{"total":95995035648,"available":95966466048,"internally_reserved":0,"allocated":905216,"data_stored":429443,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":27654411},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jul 14 19:28:54 2023","interfaces":[{"interface":"back","average":{"1min":0.57599999999999996,"5min":0.52300000000000002,"15min":0.50900000000000001},"min":{"1min":0.39200000000000002,"5min":0.27400000000000002,"15min":0.27400000000000002},"max":{"1min":0.85299999999999998,"5min":0.85299999999999998,"15min":0.85299999999999998},"last":0.496},{"interface":"front","average":{"1min":0.56799999999999995,"5min":0.51700000000000002,"15min":0.497},"min":{"1min":0.35599999999999998,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":0.77600000000000002,"5min":0.90800000000000003,"15min":0.90800000000000003},"last":0.40699999999999997}]},{"osd":1,"last update":"Fri Jul 14 19:28:54 2023","interfaces":[{"interface":"back","average":{"1min":0.42799999999999999,"5min":0.378,"15min":0.35899999999999999},"min":{"1min":0.28699999999999998,"5min":0.224,"15min":0.224},"max":{"1min":0.77200000000000002,"5min":0.77200000000000002,"15min":0.77200000000000002},"last":0.30599999999999999},{"interface":"front","average":{"1min":0.48699999999999999,"5min":0.40500000000000003,"15min":0.38700000000000001},"min":{"1min":0.33800000000000002,"5min":0.247,"15min":0.247},"max":{"1min":0.72999999999999998,"5min":0.72999999999999998,"15min":0.72999999999999998},"last":0.35299999999999998}]},{"osd":2,"last update":"Fri Jul 14 19:28:54 2023","interfaces":[{"interface":"back","average":{"1min":0.61099999999999999,"5min":0.59799999999999998,"15min":0.59599999999999997},"min":{"1min":0.373,"5min":0.32800000000000001,"15min":0.32800000000000001},"max":{"1min":0.83799999999999997,"5min":0.86399999999999999,"15min":0.86399999999999999},"last":0.68700000000000006},{"interface":"front","average":{"1min":0.65200000000000002,"5min":0.59699999999999998,"15min":0.58299999999999996},"min":{"1min":0.43099999999999999,"5min":0.43099999999999999,"15min":0.43099999999999999},"max":{"1min":0.97099999999999997,"5min":0.97099999999999997,"15min":0.97099999999999997},"last":0.46500000000000002}]},{"osd":3,"last update":"Fri Jul 14 19:28:54 2023","interfaces":[{"interface":"back","average":{"1min":0.61299999999999999,"5min":0.55700000000000005,"15min":0.54400000000000004},"min":{"1min":0.442,"5min":0.36599999999999999,"15min":0.36599999999999999},"max":{"1min":0.98599999999999999,"5min":0.98599999999999999,"15min":0.98599999999999999},"last":0.65400000000000003},{"interface":"front","average":{"1min":0.59399999999999997,"5min":0.54600000000000004,"15min":0.53000000000000003},"min":{"1min":0.36399999999999999,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.876,"5min":0.876,"15min":0.876},"last":0.754}]},{"osd":5,"last update":"Fri Jul 14 19:29:16 2023","interfaces":[{"interface":"back","average":{"1min":0.68700000000000006,"5min":0.61699999999999999,"15min":0.59899999999999998},"min":{"1min":0.55000000000000004,"5min":0.34699999999999998,"15min":0.34699999999999998},"max":{"1min":0.93500000000000005,"5min":0.93500000000000005,"15min":0.93500000000000005},"last":0.52800000000000002},{"interface":"front","average":{"1min":0.67600000000000005,"5min":0.60799999999999998,"15min":0.59099999999999997},"min":{"1min":0.436,"5min":0.314,"15min":0.314},"max":{"1min":0.92100000000000004,"5min":0.97099999999999997,"15min":0.97099999999999997},"last":0.48199999999999998}]},{"osd":6,"last update":"Fri Jul 14 19:29:38 2023","interfaces":[{"interface":"back","average":{"1min":0.61899999999999999,"5min":0.59799999999999998,"15min":0.58999999999999997},"min":{"1min":0.42199999999999999,"5min":0.378,"15min":0.378},"max":{"1min":0.89500000000000002,"5min":0.93000000000000005,"15min":0.93000000000000005},"last":0.77800000000000002},{"interface":"front","average":{"1min":0.60999999999999999,"5min":0.625,"15min":0.621},"min":{"1min":0.44900000000000001,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":0.76400000000000001,"5min":0.89500000000000002,"15min":0.89500000000000002},"last":0.60099999999999998}]},{"osd":7,"last update":"Fri Jul 14 19:28:59 2023","interfaces":[{"interface":"back","average":{"1min":0.61199999999999999,"5min":0.58299999999999996,"15min":0.57799999999999996},"min":{"1min":0.373,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":0.86499999999999999,"5min":0.89800000000000002,"15min":0.89800000000000002},"last":0.42699999999999999},{"interface":"front","average":{"1min":0.59199999999999997,"5min":0.55600000000000005,"15min":0.55000000000000004},"min":{"1min":0.38400000000000001,"5min":0.36199999999999999,"15min":0.36199999999999999},"max":{"1min":0.96899999999999997,"5min":0.96899999999999997,"15min":0.96899999999999997},"last":0.56699999999999995}]},{"osd":8,"last update":"Fri Jul 14 19:29:21 2023","interfaces":[{"interface":"back","average":{"1min":0.71999999999999997,"5min":0.67600000000000005,"15min":0.66800000000000004},"min":{"1min":0.51000000000000001,"5min":0.51000000000000001,"15min":0.51000000000000001},"max":{"1min":0.92500000000000004,"5min":0.92500000000000004,"15min":0.92500000000000004},"last":0.64000000000000001},{"interface":"front","average":{"1min":0.76100000000000001,"5min":0.71899999999999997,"15min":0.71199999999999997},"min":{"1min":0.622,"5min":0.49099999999999999,"15min":0.49099999999999999},"max":{"1min":0.86899999999999999,"5min":0.95899999999999996,"15min":0.95899999999999996},"last":0.66500000000000004}]},{"osd":9,"last update":"Fri Jul 14 19:29:38 2023","interfaces":[{"interface":"back","average":{"1min":0.68700000000000006,"5min":0.69499999999999995,"15min":0.69699999999999995},"min":{"1min":0.45200000000000001,"5min":0.45200000000000001,"15min":0.45200000000000001},"max":{"1min":1.036,"5min":1.036,"15min":1.036},"last":0.55100000000000005},{"interface":"front","average":{"1min":0.751,"5min":0.74299999999999999,"15min":0.74199999999999999},"min":{"1min":0.53900000000000003,"5min":0.52000000000000002,"15min":0.52000000000000002},"max":{"1min":0.88,"5min":1.04,"15min":1.04},"last":0.70999999999999996}]},{"osd":10,"last update":"Fri Jul 14 19:28:59 2023","interfaces":[{"interface":"back","average":{"1min":0.68300000000000005,"5min":0.68300000000000005,"15min":0.68300000000000005},"min":{"1min":0.47299999999999998,"5min":0.47299999999999998,"15min":0.47299999999999998},"max":{"1min":0.95299999999999996,"5min":0.95299999999999996,"15min":0.95299999999999996},"last":0.61899999999999999},{"interface":"front","average":{"1min":0.68000000000000005,"5min":0.68000000000000005,"15min":0.68000000000000005},"min":{"1min":0.443,"5min":0.443,"15min":0.443},"max":{"1min":0.995,"5min":0.995,"15min":0.995},"last":0.45000000000000001}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.70299999999999996}]}]},{"osd":5,"up_from":37,"seq":158913789998,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28500,"kb_used_data":1484,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716652,"statfs":{"total":95995035648,"available":95965851648,"internally_reserved":0,"allocated":1519616,"data_stored":1020766,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":27654411},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jul 14 19:29:18 2023","interfaces":[{"interface":"back","average":{"1min":0.51300000000000001,"5min":0.48599999999999999,"15min":0.46800000000000003},"min":{"1min":0.30499999999999999,"5min":0.28499999999999998,"15min":0.28499999999999998},"max":{"1min":0.75900000000000001,"5min":0.86399999999999999,"15min":0.86399999999999999},"last":0.46000000000000002},{"interface":"front","average":{"1min":0.48199999999999998,"5min":0.50700000000000001,"15min":0.502},"min":{"1min":0.32300000000000001,"5min":0.32300000000000001,"15min":0.32300000000000001},"max":{"1min":0.71899999999999997,"5min":0.84099999999999997,"15min":0.84099999999999997},"last":0.59599999999999997}]},{"osd":1,"last update":"Fri Jul 14 19:29:18 2023","interfaces":[{"interface":"back","average":{"1min":0.57699999999999996,"5min":0.55000000000000004,"15min":0.54000000000000004},"min":{"1min":0.36199999999999999,"5min":0.36199999999999999,"15min":0.36199999999999999},"max":{"1min":0.81599999999999995,"5min":1.0620000000000001,"15min":1.0620000000000001},"last":0.71899999999999997},{"interface":"front","average":{"1min":0.61499999999999999,"5min":0.56999999999999995,"15min":0.56000000000000005},"min":{"1min":0.48399999999999999,"5min":0.36099999999999999,"15min":0.36099999999999999},"max":{"1min":0.76900000000000002,"5min":0.93899999999999995,"15min":0.93899999999999995},"last":0.66800000000000004}]},{"osd":2,"last update":"Fri Jul 14 19:29:18 2023","interfaces":[{"interface":"back","average":{"1min":0.44800000000000001,"5min":0.443,"15min":0.44},"min":{"1min":0.28199999999999997,"5min":0.217,"15min":0.217},"max":{"1min":0.66500000000000004,"5min":0.67400000000000004,"15min":0.67400000000000004},"last":0.41799999999999998},{"interface":"front","average":{"1min":0.441,"5min":0.45400000000000001,"15min":0.44900000000000001},"min":{"1min":0.30099999999999999,"5min":0.23699999999999999,"15min":0.23699999999999999},"max":{"1min":0.68700000000000006,"5min":0.82499999999999996,"15min":0.82499999999999996},"last":0.38400000000000001}]},{"osd":3,"last update":"Fri Jul 14 19:29:18 2023","interfaces":[{"interface":"back","average":{"1min":0.52900000000000003,"5min":0.56200000000000006,"15min":0.56699999999999995},"min":{"1min":0.39800000000000002,"5min":0.32200000000000001,"15min":0.32200000000000001},"max":{"1min":0.72199999999999998,"5min":0.90800000000000003,"15min":0.90800000000000003},"last":0.49199999999999999},{"interface":"front","average":{"1min":0.55000000000000004,"5min":0.57299999999999995,"15min":0.57099999999999995},"min":{"1min":0.39300000000000002,"5min":0.33900000000000002,"15min":0.33900000000000002},"max":{"1min":0.74399999999999999,"5min":0.94399999999999995,"15min":0.94399999999999995},"last":0.53700000000000003}]},{"osd":4,"last update":"Fri Jul 14 19:29:18 2023","interfaces":[{"interface":"back","average":{"1min":0.63800000000000001,"5min":0.61099999999999999,"15min":0.60099999999999998},"min":{"1min":0.43099999999999999,"5min":0.43099999999999999,"15min":0.43099999999999999},"max":{"1min":0.82899999999999996,"5min":0.86899999999999999,"15min":0.86899999999999999},"last":0.621},{"interface":"front","average":{"1min":0.61199999999999999,"5min":0.59999999999999998,"15min":0.60299999999999998},"min":{"1min":0.38900000000000001,"5min":0.38,"15min":0.38},"max":{"1min":0.81399999999999995,"5min":0.81399999999999995,"15min":0.81399999999999995},"last":0.74399999999999999}]},{"osd":6,"last update":"Fri Jul 14 19:29:43 2023","interfaces":[{"interface":"back","average":{"1min":0.58899999999999997,"5min":0.60399999999999998,"15min":0.59699999999999998},"min":{"1min":0.38700000000000001,"5min":0.38300000000000001,"15min":0.38300000000000001},"max":{"1min":0.81399999999999995,"5min":0.85799999999999998,"15min":0.85799999999999998},"last":0.63600000000000001},{"interface":"front","average":{"1min":0.64600000000000002,"5min":0.64800000000000002,"15min":0.63600000000000001},"min":{"1min":0.40799999999999997,"5min":0.36099999999999999,"15min":0.36099999999999999},"max":{"1min":0.84099999999999997,"5min":1.0189999999999999,"15min":1.0189999999999999},"last":0.58299999999999996}]},{"osd":7,"last update":"Fri Jul 14 19:28:59 2023","interfaces":[{"interface":"back","average":{"1min":0.65600000000000003,"5min":0.66400000000000003,"15min":0.66600000000000004},"min":{"1min":0.49099999999999999,"5min":0.49099999999999999,"15min":0.49099999999999999},"max":{"1min":0.88,"5min":0.95399999999999996,"15min":0.95399999999999996},"last":0.70099999999999996},{"interface":"front","average":{"1min":0.66800000000000004,"5min":0.68500000000000005,"15min":0.68799999999999994},"min":{"1min":0.46999999999999997,"5min":0.45800000000000002,"15min":0.45800000000000002},"max":{"1min":0.89500000000000002,"5min":0.96099999999999997,"15min":0.96099999999999997},"last":0.81699999999999995}]},{"osd":8,"last update":"Fri Jul 14 19:29:18 2023","interfaces":[{"interface":"back","average":{"1min":0.62,"5min":0.64500000000000002,"15min":0.64900000000000002},"min":{"1min":0.34200000000000003,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":0.93600000000000005,"5min":0.997,"15min":0.997},"last":0.56399999999999995},{"interface":"front","average":{"1min":0.58699999999999997,"5min":0.60199999999999998,"15min":0.60399999999999998},"min":{"1min":0.36099999999999999,"5min":0.36099999999999999,"15min":0.36099999999999999},"max":{"1min":0.89600000000000002,"5min":0.89600000000000002,"15min":0.89600000000000002},"last":0.68500000000000005}]},{"osd":9,"last update":"Fri Jul 14 19:29:43 2023","interfaces":[{"interface":"back","average":{"1min":0.65800000000000003,"5min":0.69399999999999995,"15min":0.69999999999999996},"min":{"1min":0.47299999999999998,"5min":0.47299999999999998,"15min":0.47299999999999998},"max":{"1min":0.82599999999999996,"5min":0.88700000000000001,"15min":0.88700000000000001},"last":0.65200000000000002},{"interface":"front","average":{"1min":0.72299999999999998,"5min":0.76700000000000002,"15min":0.77400000000000002},"min":{"1min":0.42899999999999999,"5min":0.42899999999999999,"15min":0.42899999999999999},"max":{"1min":0.877,"5min":1.111,"15min":1.111},"last":0.77900000000000003}]},{"osd":10,"last update":"Fri Jul 14 19:28:59 2023","interfaces":[{"interface":"back","average":{"1min":0.71699999999999997,"5min":0.71699999999999997,"15min":0.71699999999999997},"min":{"1min":0.46899999999999997,"5min":0.46899999999999997,"15min":0.46899999999999997},"max":{"1min":0.96599999999999997,"5min":0.96599999999999997,"15min":0.96599999999999997},"last":0.76600000000000001},{"interface":"front","average":{"1min":0.73199999999999998,"5min":0.73199999999999998,"15min":0.73199999999999998},"min":{"1min":0.54000000000000004,"5min":0.54000000000000004,"15min":0.54000000000000004},"max":{"1min":0.95099999999999996,"5min":0.95099999999999996,"15min":0.95099999999999996},"last":0.84099999999999997}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1478,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":54660,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":90112,"data_stored":56352,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":284,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":986,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1818,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":2476,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":738,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":818,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":98304,"data_stored":56580,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":246,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":123,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":123,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":123,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":123,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-07-14T19:29:48.212 INFO:tasks.cephadm.ceph_manager.ceph:clean! 2023-07-14T19:29:48.212 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2023-07-14T19:29:48.213 INFO:tasks.cephadm.ceph_manager.ceph:wait_until_healthy 2023-07-14T19:29:48.213 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81c9387d2e7576da065747ec9c16d1c7d69016ec shell --fsid 9adea648-227b-11ee-9b33-001a4aab830c -- ceph health --format=json 2023-07-14T19:29:48.765 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/9adea648-227b-11ee-9b33-001a4aab830c/mon.a/config 2023-07-14T19:29:48.941 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:29:48 smithi101 ceph-mon[128807]: pgmap v276: 97 pgs: 97 active+clean; 634 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2023-07-14T19:29:48.941 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:29:48 smithi101 ceph-mon[128807]: from='client.14970 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-07-14T19:29:48.962 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:29:48 smithi098 ceph-mon[127486]: pgmap v276: 97 pgs: 97 active+clean; 634 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2023-07-14T19:29:48.962 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:29:48 smithi098 ceph-mon[127486]: from='client.14970 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-07-14T19:29:48.970 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:29:48 smithi139 ceph-mon[129293]: pgmap v276: 97 pgs: 97 active+clean; 634 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2023-07-14T19:29:48.970 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:29:48 smithi139 ceph-mon[129293]: from='client.14970 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-07-14T19:29:49.874 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:49.873+0000 7f247d54a700 1 -- 172.21.15.98:0/249955855 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f24781040d0 msgr2=0x7f2478104540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:29:49.874 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:49.873+0000 7f247d54a700 1 --2- 172.21.15.98:0/249955855 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f24781040d0 0x7f2478104540 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7f246c0099c0 tx=0x7f246c009cd0 comp rx=0 tx=0).stop 2023-07-14T19:29:49.874 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:49.873+0000 7f247d54a700 1 -- 172.21.15.98:0/249955855 shutdown_connections 2023-07-14T19:29:49.875 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:49.873+0000 7f247d54a700 1 --2- 172.21.15.98:0/249955855 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f2478103370 0x7f2478103740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:29:49.875 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:49.873+0000 7f247d54a700 1 --2- 172.21.15.98:0/249955855 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f2478104a80 0x7f247810b270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:29:49.875 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:49.873+0000 7f247d54a700 1 --2- 172.21.15.98:0/249955855 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f24781040d0 0x7f2478104540 unknown :-1 s=CLOSED pgs=293 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:29:49.875 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:49.873+0000 7f247d54a700 1 -- 172.21.15.98:0/249955855 >> 172.21.15.98:0/249955855 conn(0x7f2478074550 msgr2=0x7f2478074950 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:29:49.875 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:49.873+0000 7f247d54a700 1 -- 172.21.15.98:0/249955855 shutdown_connections 2023-07-14T19:29:49.875 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:49.873+0000 7f247d54a700 1 -- 172.21.15.98:0/249955855 wait complete. 2023-07-14T19:29:49.875 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:49.874+0000 7f247d54a700 1 Processor -- start 2023-07-14T19:29:49.877 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:49.874+0000 7f247d54a700 1 -- start start 2023-07-14T19:29:49.877 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:49.874+0000 7f247d54a700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2478103370 0x7f247819aa60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:29:49.877 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:49.874+0000 7f247d54a700 1 --2- >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f24781040d0 0x7f247819afa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:29:49.877 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:49.874+0000 7f247d54a700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f2478104a80 0x7f2478194ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:29:49.877 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:49.874+0000 7f247d54a700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2478195150 con 0x7f2478103370 2023-07-14T19:29:49.878 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:49.874+0000 7f247d54a700 1 -- --> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f24781952c0 con 0x7f24781040d0 2023-07-14T19:29:49.878 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:49.874+0000 7f247d54a700 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2478195480 con 0x7f2478104a80 2023-07-14T19:29:49.878 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:49.874+0000 7f2476ffd700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2478103370 0x7f247819aa60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:29:49.878 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:49.874+0000 7f2476ffd700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2478103370 0x7f247819aa60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:46300/0 (socket says 172.21.15.98:46300) 2023-07-14T19:29:49.878 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:49.874+0000 7f2476ffd700 1 -- 172.21.15.98:0/1232190463 learned_addr learned my addr 172.21.15.98:0/1232190463 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-07-14T19:29:49.878 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:49.874+0000 7f24767fc700 1 --2- 172.21.15.98:0/1232190463 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f24781040d0 0x7f247819afa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:29:49.878 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:49.875+0000 7f2476ffd700 1 -- 172.21.15.98:0/1232190463 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f2478104a80 msgr2=0x7f2478194ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:29:49.879 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:49.875+0000 7f24777fe700 1 --2- 172.21.15.98:0/1232190463 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f2478104a80 0x7f2478194ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:29:49.879 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:49.875+0000 7f2476ffd700 1 --2- 172.21.15.98:0/1232190463 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f2478104a80 0x7f2478194ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:29:49.879 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:49.875+0000 7f2476ffd700 1 -- 172.21.15.98:0/1232190463 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f24781040d0 msgr2=0x7f247819afa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:29:49.879 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:49.875+0000 7f2476ffd700 1 --2- 172.21.15.98:0/1232190463 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f24781040d0 0x7f247819afa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:29:49.879 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:49.875+0000 7f2476ffd700 1 -- 172.21.15.98:0/1232190463 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f246c0095f0 con 0x7f2478103370 2023-07-14T19:29:49.879 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:49.875+0000 7f24777fe700 1 --2- 172.21.15.98:0/1232190463 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f2478104a80 0x7f2478194ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-07-14T19:29:49.879 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:49.875+0000 7f2476ffd700 1 --2- 172.21.15.98:0/1232190463 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2478103370 0x7f247819aa60 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7f246800c920 tx=0x7f246800cc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:29:49.880 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:49.875+0000 7f245ffff700 1 -- 172.21.15.98:0/1232190463 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2468007b00 con 0x7f2478103370 2023-07-14T19:29:49.880 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:49.875+0000 7f247d54a700 1 -- 172.21.15.98:0/1232190463 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2478195760 con 0x7f2478103370 2023-07-14T19:29:49.880 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:49.875+0000 7f245ffff700 1 -- 172.21.15.98:0/1232190463 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f246800ce70 con 0x7f2478103370 2023-07-14T19:29:49.880 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:49.875+0000 7f245ffff700 1 -- 172.21.15.98:0/1232190463 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2468005460 con 0x7f2478103370 2023-07-14T19:29:49.880 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:49.875+0000 7f247d54a700 1 -- 172.21.15.98:0/1232190463 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f247810bd40 con 0x7f2478103370 2023-07-14T19:29:49.880 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:49.876+0000 7f247d54a700 1 -- 172.21.15.98:0/1232190463 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f247804e340 con 0x7f2478103370 2023-07-14T19:29:49.881 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:49.877+0000 7f245ffff700 1 -- 172.21.15.98:0/1232190463 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 17) v1 ==== 135766+0+0 (secure 0 0 0) 0x7f24680055c0 con 0x7f2478103370 2023-07-14T19:29:49.881 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:49.878+0000 7f245ffff700 1 --2- 172.21.15.98:0/1232190463 >> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] conn(0x7f24600a11e0 0x7f24600a3690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:29:49.881 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:49.878+0000 7f24767fc700 1 --2- 172.21.15.98:0/1232190463 >> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] conn(0x7f24600a11e0 0x7f24600a3690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:29:49.882 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:49.878+0000 7f245ffff700 1 -- 172.21.15.98:0/1232190463 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 7979+0+0 (secure 0 0 0) 0x7f24680cca00 con 0x7f2478103370 2023-07-14T19:29:49.882 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:49.878+0000 7f24767fc700 1 --2- 172.21.15.98:0/1232190463 >> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] conn(0x7f24600a11e0 0x7f24600a3690 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f246c006010 tx=0x7f246c009f10 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:29:49.882 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:49.881+0000 7f245ffff700 1 -- 172.21.15.98:0/1232190463 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f24680960a0 con 0x7f2478103370 2023-07-14T19:29:50.110 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:50.109+0000 7f247d54a700 1 -- 172.21.15.98:0/1232190463 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f2478195ff0 con 0x7f2478103370 2023-07-14T19:29:50.110 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:50.109+0000 7f245ffff700 1 -- 172.21.15.98:0/1232190463 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+46 (secure 0 0 0) 0x7f246801d080 con 0x7f2478103370 2023-07-14T19:29:50.110 INFO:teuthology.orchestra.run.smithi098.stdout: 2023-07-14T19:29:50.110 INFO:teuthology.orchestra.run.smithi098.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2023-07-14T19:29:50.112 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:50.111+0000 7f247d54a700 1 -- 172.21.15.98:0/1232190463 >> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] conn(0x7f24600a11e0 msgr2=0x7f24600a3690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:29:50.112 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:50.111+0000 7f247d54a700 1 --2- 172.21.15.98:0/1232190463 >> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] conn(0x7f24600a11e0 0x7f24600a3690 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f246c006010 tx=0x7f246c009f10 comp rx=0 tx=0).stop 2023-07-14T19:29:50.112 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:50.111+0000 7f247d54a700 1 -- 172.21.15.98:0/1232190463 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2478103370 msgr2=0x7f247819aa60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:29:50.113 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:50.111+0000 7f247d54a700 1 --2- 172.21.15.98:0/1232190463 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2478103370 0x7f247819aa60 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7f246800c920 tx=0x7f246800cc30 comp rx=0 tx=0).stop 2023-07-14T19:29:50.113 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:50.112+0000 7f247d54a700 1 -- 172.21.15.98:0/1232190463 shutdown_connections 2023-07-14T19:29:50.113 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:50.112+0000 7f247d54a700 1 --2- 172.21.15.98:0/1232190463 >> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] conn(0x7f24600a11e0 0x7f24600a3690 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:29:50.113 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:50.112+0000 7f247d54a700 1 --2- 172.21.15.98:0/1232190463 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f2478104a80 0x7f2478194ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:29:50.113 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:50.112+0000 7f247d54a700 1 --2- 172.21.15.98:0/1232190463 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2478103370 0x7f247819aa60 unknown :-1 s=CLOSED pgs=294 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:29:50.113 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:50.112+0000 7f247d54a700 1 --2- 172.21.15.98:0/1232190463 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f24781040d0 0x7f247819afa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:29:50.113 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:50.112+0000 7f247d54a700 1 -- 172.21.15.98:0/1232190463 >> 172.21.15.98:0/1232190463 conn(0x7f2478074550 msgr2=0x7f24780fdbd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:29:50.113 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:50.112+0000 7f247d54a700 1 -- 172.21.15.98:0/1232190463 shutdown_connections 2023-07-14T19:29:50.114 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:50.112+0000 7f247d54a700 1 -- 172.21.15.98:0/1232190463 wait complete. 2023-07-14T19:29:50.520 INFO:tasks.cephadm.ceph_manager.ceph:wait_until_healthy done 2023-07-14T19:29:50.521 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-07-14T19:29:50.646 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:50.646+0000 7f12f719b700 1 -- 172.21.15.98:0/1999996459 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f12f0103020 msgr2=0x7f12f0105400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:29:50.647 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:50.646+0000 7f12f719b700 1 --2- 172.21.15.98:0/1999996459 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f12f0103020 0x7f12f0105400 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7f12dc009990 tx=0x7f12dc009ca0 comp rx=0 tx=0).stop 2023-07-14T19:29:50.647 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:50.646+0000 7f12f719b700 1 -- 172.21.15.98:0/1999996459 shutdown_connections 2023-07-14T19:29:50.647 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:50.646+0000 7f12f719b700 1 --2- 172.21.15.98:0/1999996459 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f12f0100700 0x7f12f0102ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:29:50.647 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:50.646+0000 7f12f719b700 1 --2- 172.21.15.98:0/1999996459 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f12f0105940 0x7f12f0107d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:29:50.647 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:50.646+0000 7f12f719b700 1 --2- 172.21.15.98:0/1999996459 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f12f0103020 0x7f12f0105400 unknown :-1 s=CLOSED pgs=295 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:29:50.648 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:50.646+0000 7f12f719b700 1 -- 172.21.15.98:0/1999996459 >> 172.21.15.98:0/1999996459 conn(0x7f12f00fa100 msgr2=0x7f12f00fc550 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:29:50.648 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:50.646+0000 7f12f719b700 1 -- 172.21.15.98:0/1999996459 shutdown_connections 2023-07-14T19:29:50.648 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:50.646+0000 7f12f719b700 1 -- 172.21.15.98:0/1999996459 wait complete. 2023-07-14T19:29:50.650 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:50.647+0000 7f12f719b700 1 Processor -- start 2023-07-14T19:29:50.650 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:50.647+0000 7f12f719b700 1 -- start start 2023-07-14T19:29:50.650 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:50.647+0000 7f12f719b700 1 --2- >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f12f0100700 0x7f12f01958a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:29:50.650 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:50.647+0000 7f12f719b700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f12f0103020 0x7f12f0195de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:29:50.650 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:50.647+0000 7f12f719b700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f12f0105940 0x7f12f019ce50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:29:50.650 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:50.647+0000 7f12f719b700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f12f019d420 con 0x7f12f0105940 2023-07-14T19:29:50.651 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:50.647+0000 7f12f719b700 1 -- --> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f12f019d560 con 0x7f12f0100700 2023-07-14T19:29:50.651 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:50.647+0000 7f12f719b700 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f12f019d6f0 con 0x7f12f0103020 2023-07-14T19:29:50.651 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:50.647+0000 7f12f5738700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f12f0105940 0x7f12f019ce50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:29:50.651 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:50.647+0000 7f12f5738700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f12f0105940 0x7f12f019ce50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:46316/0 (socket says 172.21.15.98:46316) 2023-07-14T19:29:50.651 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:50.647+0000 7f12f5738700 1 -- 172.21.15.98:0/2749659362 learned_addr learned my addr 172.21.15.98:0/2749659362 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-07-14T19:29:50.651 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:50.647+0000 7f12effff700 1 --2- 172.21.15.98:0/2749659362 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f12f0103020 0x7f12f0195de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:29:50.651 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:50.647+0000 7f12f4f37700 1 --2- 172.21.15.98:0/2749659362 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f12f0100700 0x7f12f01958a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:29:50.651 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:50.647+0000 7f12f5738700 1 -- 172.21.15.98:0/2749659362 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f12f0103020 msgr2=0x7f12f0195de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:29:50.652 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:50.647+0000 7f12f5738700 1 --2- 172.21.15.98:0/2749659362 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f12f0103020 0x7f12f0195de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:29:50.652 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:50.647+0000 7f12f5738700 1 -- 172.21.15.98:0/2749659362 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f12f0100700 msgr2=0x7f12f01958a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:29:50.652 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:50.647+0000 7f12f5738700 1 --2- 172.21.15.98:0/2749659362 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f12f0100700 0x7f12f01958a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:29:50.652 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:50.647+0000 7f12f5738700 1 -- 172.21.15.98:0/2749659362 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f12dc0095f0 con 0x7f12f0105940 2023-07-14T19:29:50.652 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:50.647+0000 7f12f4f37700 1 --2- 172.21.15.98:0/2749659362 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f12f0100700 0x7f12f01958a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-07-14T19:29:50.652 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:50.647+0000 7f12f5738700 1 --2- 172.21.15.98:0/2749659362 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f12f0105940 0x7f12f019ce50 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7f12e800ef30 tx=0x7f12e800c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:29:50.653 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:50.647+0000 7f12edffb700 1 -- 172.21.15.98:0/2749659362 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f12e8018070 con 0x7f12f0105940 2023-07-14T19:29:50.653 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:50.648+0000 7f12edffb700 1 -- 172.21.15.98:0/2749659362 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f12e800f040 con 0x7f12f0105940 2023-07-14T19:29:50.653 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:50.648+0000 7f12f719b700 1 -- 172.21.15.98:0/2749659362 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f12f019d970 con 0x7f12f0105940 2023-07-14T19:29:50.653 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:50.648+0000 7f12edffb700 1 -- 172.21.15.98:0/2749659362 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f12e800cdb0 con 0x7f12f0105940 2023-07-14T19:29:50.653 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:50.648+0000 7f12f719b700 1 -- 172.21.15.98:0/2749659362 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f12f019de30 con 0x7f12f0105940 2023-07-14T19:29:50.654 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:50.649+0000 7f12f719b700 1 -- 172.21.15.98:0/2749659362 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f12f00fbc70 con 0x7f12f0105940 2023-07-14T19:29:50.654 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:50.650+0000 7f12edffb700 1 -- 172.21.15.98:0/2749659362 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 17) v1 ==== 135766+0+0 (secure 0 0 0) 0x7f12e8003a00 con 0x7f12f0105940 2023-07-14T19:29:50.654 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:50.650+0000 7f12edffb700 1 --2- 172.21.15.98:0/2749659362 >> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] conn(0x7f12e00a1220 0x7f12e00a36d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:29:50.655 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:50.650+0000 7f12edffb700 1 -- 172.21.15.98:0/2749659362 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 7979+0+0 (secure 0 0 0) 0x7f12e80cc520 con 0x7f12f0105940 2023-07-14T19:29:50.655 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:50.650+0000 7f12f4f37700 1 --2- 172.21.15.98:0/2749659362 >> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] conn(0x7f12e00a1220 0x7f12e00a36d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:29:50.655 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:50.651+0000 7f12f4f37700 1 --2- 172.21.15.98:0/2749659362 >> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] conn(0x7f12e00a1220 0x7f12e00a36d0 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f12e4005950 tx=0x7f12e40058e0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:29:50.655 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:50.654+0000 7f12edffb700 1 -- 172.21.15.98:0/2749659362 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f12e8095d50 con 0x7f12f0105940 2023-07-14T19:29:50.781 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:29:50 smithi098 ceph-mon[127486]: pgmap v277: 97 pgs: 97 active+clean; 634 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2023-07-14T19:29:50.781 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:29:50 smithi098 ceph-mon[127486]: from='client.? 172.21.15.98:0/1232190463' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2023-07-14T19:29:50.845 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:50.844+0000 7f12f719b700 1 -- 172.21.15.98:0/2749659362 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f12f002ca50 con 0x7f12f0105940 2023-07-14T19:29:50.846 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:50.846+0000 7f12edffb700 1 -- 172.21.15.98:0/2749659362 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v83) v1 ==== 74+0+22117 (secure 0 0 0) 0x7f12e8099370 con 0x7f12f0105940 2023-07-14T19:29:50.847 INFO:teuthology.orchestra.run.smithi098.stdout: 2023-07-14T19:29:50.847 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":83,"fsid":"9adea648-227b-11ee-9b33-001a4aab830c","created":"2023-07-14T19:22:07.178854+0000","modified":"2023-07-14T19:29:02.293099+0000","last_up_change":"2023-07-14T19:28:19.646827+0000","last_in_change":"2023-07-14T19:28:02.018325+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-07-14T19:25:09.845827+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2023-07-14T19:28:58.952921+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"82","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2023-07-14T19:29:00.228272+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"83","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"a3797e89-1754-41cf-9517-227e06ffdd72","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6802","nonce":120507228},{"type":"v1","addr":"172.21.15.98:6803","nonce":120507228}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6804","nonce":120507228},{"type":"v1","addr":"172.21.15.98:6805","nonce":120507228}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6808","nonce":120507228},{"type":"v1","addr":"172.21.15.98:6809","nonce":120507228}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6806","nonce":120507228},{"type":"v1","addr":"172.21.15.98:6807","nonce":120507228}]},"public_addr":"172.21.15.98:6803/120507228","cluster_addr":"172.21.15.98:6805/120507228","heartbeat_back_addr":"172.21.15.98:6809/120507228","heartbeat_front_addr":"172.21.15.98:6807/120507228","state":["exists","up"]},{"osd":1,"uuid":"2b001add-1aaa-4496-a403-7ef9725b80a0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.101:6800","nonce":2450584584},{"type":"v1","addr":"172.21.15.101:6801","nonce":2450584584}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.101:6802","nonce":2450584584},{"type":"v1","addr":"172.21.15.101:6803","nonce":2450584584}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.101:6806","nonce":2450584584},{"type":"v1","addr":"172.21.15.101:6807","nonce":2450584584}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.101:6804","nonce":2450584584},{"type":"v1","addr":"172.21.15.101:6805","nonce":2450584584}]},"public_addr":"172.21.15.101:6801/2450584584","cluster_addr":"172.21.15.101:6803/2450584584","heartbeat_back_addr":"172.21.15.101:6807/2450584584","heartbeat_front_addr":"172.21.15.101:6805/2450584584","state":["exists","up"]},{"osd":2,"uuid":"48e6d499-3ca9-424b-b0fb-3ba5f33b9fc9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":945038995},{"type":"v1","addr":"172.21.15.139:6801","nonce":945038995}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":945038995},{"type":"v1","addr":"172.21.15.139:6803","nonce":945038995}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":945038995},{"type":"v1","addr":"172.21.15.139:6807","nonce":945038995}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":945038995},{"type":"v1","addr":"172.21.15.139:6805","nonce":945038995}]},"public_addr":"172.21.15.139:6801/945038995","cluster_addr":"172.21.15.139:6803/945038995","heartbeat_back_addr":"172.21.15.139:6807/945038995","heartbeat_front_addr":"172.21.15.139:6805/945038995","state":["exists","up"]},{"osd":3,"uuid":"5530b265-31a5-447b-a1ea-17ebae9cfddf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6810","nonce":3095249665},{"type":"v1","addr":"172.21.15.98:6811","nonce":3095249665}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6812","nonce":3095249665},{"type":"v1","addr":"172.21.15.98:6813","nonce":3095249665}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6816","nonce":3095249665},{"type":"v1","addr":"172.21.15.98:6817","nonce":3095249665}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6814","nonce":3095249665},{"type":"v1","addr":"172.21.15.98:6815","nonce":3095249665}]},"public_addr":"172.21.15.98:6811/3095249665","cluster_addr":"172.21.15.98:6813/3095249665","heartbeat_back_addr":"172.21.15.98:6817/3095249665","heartbeat_front_addr":"172.21.15.98:6815/3095249665","state":["exists","up"]},{"osd":4,"uuid":"615d4642-a997-4f95-9e38-9fbdaa831400","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":32,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.101:6808","nonce":2844761406},{"type":"v1","addr":"172.21.15.101:6809","nonce":2844761406}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.101:6810","nonce":2844761406},{"type":"v1","addr":"172.21.15.101:6811","nonce":2844761406}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.101:6814","nonce":2844761406},{"type":"v1","addr":"172.21.15.101:6815","nonce":2844761406}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.101:6812","nonce":2844761406},{"type":"v1","addr":"172.21.15.101:6813","nonce":2844761406}]},"public_addr":"172.21.15.101:6809/2844761406","cluster_addr":"172.21.15.101:6811/2844761406","heartbeat_back_addr":"172.21.15.101:6815/2844761406","heartbeat_front_addr":"172.21.15.101:6813/2844761406","state":["exists","up"]},{"osd":5,"uuid":"415de5fa-607a-45a8-b732-28daf3be6dd3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":37,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":3532263042},{"type":"v1","addr":"172.21.15.139:6809","nonce":3532263042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6810","nonce":3532263042},{"type":"v1","addr":"172.21.15.139:6811","nonce":3532263042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6814","nonce":3532263042},{"type":"v1","addr":"172.21.15.139:6815","nonce":3532263042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":3532263042},{"type":"v1","addr":"172.21.15.139:6813","nonce":3532263042}]},"public_addr":"172.21.15.139:6809/3532263042","cluster_addr":"172.21.15.139:6811/3532263042","heartbeat_back_addr":"172.21.15.139:6815/3532263042","heartbeat_front_addr":"172.21.15.139:6813/3532263042","state":["exists","up"]},{"osd":6,"uuid":"d7f9d4ba-c309-4b75-a836-5a36384b528e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":45,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6818","nonce":658948743},{"type":"v1","addr":"172.21.15.98:6819","nonce":658948743}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6820","nonce":658948743},{"type":"v1","addr":"172.21.15.98:6821","nonce":658948743}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6824","nonce":658948743},{"type":"v1","addr":"172.21.15.98:6825","nonce":658948743}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6822","nonce":658948743},{"type":"v1","addr":"172.21.15.98:6823","nonce":658948743}]},"public_addr":"172.21.15.98:6819/658948743","cluster_addr":"172.21.15.98:6821/658948743","heartbeat_back_addr":"172.21.15.98:6825/658948743","heartbeat_front_addr":"172.21.15.98:6823/658948743","state":["exists","up"]},{"osd":7,"uuid":"eedb94d9-c5cd-4a93-8a8c-2648804b7195","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":50,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.101:6816","nonce":2881214066},{"type":"v1","addr":"172.21.15.101:6817","nonce":2881214066}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.101:6818","nonce":2881214066},{"type":"v1","addr":"172.21.15.101:6819","nonce":2881214066}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.101:6822","nonce":2881214066},{"type":"v1","addr":"172.21.15.101:6823","nonce":2881214066}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.101:6820","nonce":2881214066},{"type":"v1","addr":"172.21.15.101:6821","nonce":2881214066}]},"public_addr":"172.21.15.101:6817/2881214066","cluster_addr":"172.21.15.101:6819/2881214066","heartbeat_back_addr":"172.21.15.101:6823/2881214066","heartbeat_front_addr":"172.21.15.101:6821/2881214066","state":["exists","up"]},{"osd":8,"uuid":"6bfd77c2-5770-466e-aa6f-84b50c74ffb6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":57,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":3669514524},{"type":"v1","addr":"172.21.15.139:6817","nonce":3669514524}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":3669514524},{"type":"v1","addr":"172.21.15.139:6819","nonce":3669514524}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":3669514524},{"type":"v1","addr":"172.21.15.139:6823","nonce":3669514524}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6820","nonce":3669514524},{"type":"v1","addr":"172.21.15.139:6821","nonce":3669514524}]},"public_addr":"172.21.15.139:6817/3669514524","cluster_addr":"172.21.15.139:6819/3669514524","heartbeat_back_addr":"172.21.15.139:6823/3669514524","heartbeat_front_addr":"172.21.15.139:6821/3669514524","state":["exists","up"]},{"osd":9,"uuid":"19c5b54a-2242-4b23-b721-63be9559653d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":65,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6826","nonce":2893994080},{"type":"v1","addr":"172.21.15.98:6827","nonce":2893994080}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6828","nonce":2893994080},{"type":"v1","addr":"172.21.15.98:6829","nonce":2893994080}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6832","nonce":2893994080},{"type":"v1","addr":"172.21.15.98:6833","nonce":2893994080}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6830","nonce":2893994080},{"type":"v1","addr":"172.21.15.98:6831","nonce":2893994080}]},"public_addr":"172.21.15.98:6827/2893994080","cluster_addr":"172.21.15.98:6829/2893994080","heartbeat_back_addr":"172.21.15.98:6833/2893994080","heartbeat_front_addr":"172.21.15.98:6831/2893994080","state":["exists","up"]},{"osd":10,"uuid":"326f0209-ba9d-4b59-97cc-d61a758a8f29","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":70,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.101:6824","nonce":3545244225},{"type":"v1","addr":"172.21.15.101:6825","nonce":3545244225}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.101:6826","nonce":3545244225},{"type":"v1","addr":"172.21.15.101:6827","nonce":3545244225}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.101:6830","nonce":3545244225},{"type":"v1","addr":"172.21.15.101:6831","nonce":3545244225}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.101:6828","nonce":3545244225},{"type":"v1","addr":"172.21.15.101:6829","nonce":3545244225}]},"public_addr":"172.21.15.101:6825/3545244225","cluster_addr":"172.21.15.101:6827/3545244225","heartbeat_back_addr":"172.21.15.101:6831/3545244225","heartbeat_front_addr":"172.21.15.101:6829/3545244225","state":["exists","up"]},{"osd":11,"uuid":"c61c0320-1057-4f17-9bf1-7ad306ed7ed3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":78,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":2630023454},{"type":"v1","addr":"172.21.15.139:6825","nonce":2630023454}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6826","nonce":2630023454},{"type":"v1","addr":"172.21.15.139:6827","nonce":2630023454}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":2630023454},{"type":"v1","addr":"172.21.15.139:6831","nonce":2630023454}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":2630023454},{"type":"v1","addr":"172.21.15.139:6829","nonce":2630023454}]},"public_addr":"172.21.15.139:6825/2630023454","cluster_addr":"172.21.15.139:6827/2630023454","heartbeat_back_addr":"172.21.15.139:6831/2630023454","heartbeat_front_addr":"172.21.15.139:6829/2630023454","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T19:24:23.620028+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T19:24:45.779846+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T19:25:06.877949+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T19:25:27.994199+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T19:25:49.461139+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T19:26:10.409282+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T19:26:32.116390+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T19:26:53.536474+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T19:27:14.384839+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T19:27:35.635921+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T19:27:56.858744+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T19:28:18.342519+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.98:0/2164590253":"2023-07-15T19:23:01.520302+0000","172.21.15.98:0/2094734896":"2023-07-15T19:23:01.520302+0000","172.21.15.98:0/784745009":"2023-07-15T19:22:39.481502+0000","172.21.15.98:0/3657190818":"2023-07-15T19:22:39.481502+0000","172.21.15.98:6800/1732769087":"2023-07-15T19:23:01.520302+0000","172.21.15.98:0/3393542419":"2023-07-15T19:22:39.481502+0000","172.21.15.98:6800/749234665":"2023-07-15T19:22:39.481502+0000","172.21.15.98:6801/1732769087":"2023-07-15T19:23:01.520302+0000","172.21.15.98:6801/749234665":"2023-07-15T19:22:39.481502+0000","172.21.15.98:0/187822873":"2023-07-15T19:23:01.520302+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-07-14T19:29:50.849 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:50.848+0000 7f12f719b700 1 -- 172.21.15.98:0/2749659362 >> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] conn(0x7f12e00a1220 msgr2=0x7f12e00a36d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:29:50.849 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:50.848+0000 7f12f719b700 1 --2- 172.21.15.98:0/2749659362 >> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] conn(0x7f12e00a1220 0x7f12e00a36d0 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f12e4005950 tx=0x7f12e40058e0 comp rx=0 tx=0).stop 2023-07-14T19:29:50.849 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:50.848+0000 7f12f719b700 1 -- 172.21.15.98:0/2749659362 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f12f0105940 msgr2=0x7f12f019ce50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:29:50.849 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:50.848+0000 7f12f719b700 1 --2- 172.21.15.98:0/2749659362 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f12f0105940 0x7f12f019ce50 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7f12e800ef30 tx=0x7f12e800c3a0 comp rx=0 tx=0).stop 2023-07-14T19:29:50.849 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:50.848+0000 7f12f719b700 1 -- 172.21.15.98:0/2749659362 shutdown_connections 2023-07-14T19:29:50.849 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:50.848+0000 7f12f719b700 1 --2- 172.21.15.98:0/2749659362 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f12f0105940 0x7f12f019ce50 unknown :-1 s=CLOSED pgs=296 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:29:50.850 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:50.848+0000 7f12f719b700 1 --2- 172.21.15.98:0/2749659362 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f12f0100700 0x7f12f01958a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:29:50.850 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:50.848+0000 7f12f719b700 1 --2- 172.21.15.98:0/2749659362 >> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] conn(0x7f12e00a1220 0x7f12e00a36d0 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:29:50.850 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:50.848+0000 7f12f719b700 1 --2- 172.21.15.98:0/2749659362 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f12f0103020 0x7f12f0195de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:29:50.850 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:50.848+0000 7f12f719b700 1 -- 172.21.15.98:0/2749659362 >> 172.21.15.98:0/2749659362 conn(0x7f12f00fa100 msgr2=0x7f12f0101360 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:29:50.850 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:50.848+0000 7f12f719b700 1 -- 172.21.15.98:0/2749659362 shutdown_connections 2023-07-14T19:29:50.850 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:50.848+0000 7f12f719b700 1 -- 172.21.15.98:0/2749659362 wait complete. 2023-07-14T19:29:50.862 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-07-14T19:25:09.845827+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2023-07-14T19:28:58.952921+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '82', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 2.25, 'score_stable': 2.25, 'optimal_score': 1, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2023-07-14T19:29:00.228272+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '83', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-07-14T19:29:50.863 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-07-14T19:29:50.941 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:29:50 smithi101 ceph-mon[128807]: pgmap v277: 97 pgs: 97 active+clean; 634 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2023-07-14T19:29:50.941 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:29:50 smithi101 ceph-mon[128807]: from='client.? 172.21.15.98:0/1232190463' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2023-07-14T19:29:50.970 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:29:50 smithi139 ceph-mon[129293]: pgmap v277: 97 pgs: 97 active+clean; 634 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2023-07-14T19:29:50.971 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:29:50 smithi139 ceph-mon[129293]: from='client.? 172.21.15.98:0/1232190463' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2023-07-14T19:29:50.988 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:50.987+0000 7f5cd0e84700 1 -- 172.21.15.98:0/1466646344 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5ccc0ffd00 msgr2=0x7f5ccc100150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:29:50.989 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:50.987+0000 7f5cd0e84700 1 --2- 172.21.15.98:0/1466646344 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5ccc0ffd00 0x7f5ccc100150 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7f5cbc009990 tx=0x7f5cbc009ca0 comp rx=0 tx=0).stop 2023-07-14T19:29:50.989 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:50.988+0000 7f5cd0e84700 1 -- 172.21.15.98:0/1466646344 shutdown_connections 2023-07-14T19:29:50.989 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:50.988+0000 7f5cd0e84700 1 --2- 172.21.15.98:0/1466646344 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f5ccc100690 0x7f5ccc106f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:29:50.989 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:50.988+0000 7f5cd0e84700 1 --2- 172.21.15.98:0/1466646344 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f5ccc0feb10 0x7f5ccc0fef20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:29:50.989 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:50.988+0000 7f5cd0e84700 1 --2- 172.21.15.98:0/1466646344 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5ccc0ffd00 0x7f5ccc100150 unknown :-1 s=CLOSED pgs=297 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:29:50.989 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:50.988+0000 7f5cd0e84700 1 -- 172.21.15.98:0/1466646344 >> 172.21.15.98:0/1466646344 conn(0x7f5ccc0fa130 msgr2=0x7f5ccc0fc560 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:29:50.989 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:50.988+0000 7f5cd0e84700 1 -- 172.21.15.98:0/1466646344 shutdown_connections 2023-07-14T19:29:50.990 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:50.988+0000 7f5cd0e84700 1 -- 172.21.15.98:0/1466646344 wait complete. 2023-07-14T19:29:50.992 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:50.988+0000 7f5cd0e84700 1 Processor -- start 2023-07-14T19:29:50.992 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:50.988+0000 7f5cd0e84700 1 -- start start 2023-07-14T19:29:50.992 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:50.989+0000 7f5cd0e84700 1 --2- >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f5ccc0feb10 0x7f5ccc195a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:29:50.992 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:50.989+0000 7f5cd0e84700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f5ccc0ffd00 0x7f5ccc195f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:29:50.992 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:50.989+0000 7f5cd0e84700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5ccc100690 0x7f5ccc19cfc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:29:50.993 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:50.989+0000 7f5cd0e84700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5ccc19d590 con 0x7f5ccc100690 2023-07-14T19:29:50.993 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:50.989+0000 7f5cd0e84700 1 -- --> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5ccc19d6d0 con 0x7f5ccc0feb10 2023-07-14T19:29:50.993 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:50.989+0000 7f5cd0e84700 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5ccc19d860 con 0x7f5ccc0ffd00 2023-07-14T19:29:50.993 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:50.989+0000 7f5ccad9d700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5ccc100690 0x7f5ccc19cfc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:29:50.993 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:50.989+0000 7f5ccad9d700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5ccc100690 0x7f5ccc19cfc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:46330/0 (socket says 172.21.15.98:46330) 2023-07-14T19:29:50.993 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:50.989+0000 7f5ccad9d700 1 -- 172.21.15.98:0/1654970315 learned_addr learned my addr 172.21.15.98:0/1654970315 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-07-14T19:29:50.993 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:50.989+0000 7f5cc9d9b700 1 --2- 172.21.15.98:0/1654970315 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f5ccc0ffd00 0x7f5ccc195f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:29:50.994 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:50.989+0000 7f5cca59c700 1 --2- 172.21.15.98:0/1654970315 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f5ccc0feb10 0x7f5ccc195a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:29:50.994 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:50.989+0000 7f5ccad9d700 1 -- 172.21.15.98:0/1654970315 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f5ccc0ffd00 msgr2=0x7f5ccc195f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:29:50.994 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:50.989+0000 7f5ccad9d700 1 --2- 172.21.15.98:0/1654970315 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f5ccc0ffd00 0x7f5ccc195f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:29:50.994 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:50.989+0000 7f5ccad9d700 1 -- 172.21.15.98:0/1654970315 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f5ccc0feb10 msgr2=0x7f5ccc195a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:29:50.994 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:50.989+0000 7f5ccad9d700 1 --2- 172.21.15.98:0/1654970315 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f5ccc0feb10 0x7f5ccc195a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:29:50.994 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:50.989+0000 7f5ccad9d700 1 -- 172.21.15.98:0/1654970315 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5cbc0095f0 con 0x7f5ccc100690 2023-07-14T19:29:50.994 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:50.989+0000 7f5ccad9d700 1 --2- 172.21.15.98:0/1654970315 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5ccc100690 0x7f5ccc19cfc0 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7f5cc4007ee0 tx=0x7f5cc400d3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:29:50.994 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:50.989+0000 7f5cbb7fe700 1 -- 172.21.15.98:0/1654970315 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5cc4017070 con 0x7f5ccc100690 2023-07-14T19:29:50.995 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:50.989+0000 7f5cd0e84700 1 -- 172.21.15.98:0/1654970315 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5ccc19db40 con 0x7f5ccc100690 2023-07-14T19:29:50.995 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:50.989+0000 7f5cbb7fe700 1 -- 172.21.15.98:0/1654970315 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f5cc400f040 con 0x7f5ccc100690 2023-07-14T19:29:50.995 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:50.989+0000 7f5cbb7fe700 1 -- 172.21.15.98:0/1654970315 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5cc4005740 con 0x7f5ccc100690 2023-07-14T19:29:50.995 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:50.989+0000 7f5cd0e84700 1 -- 172.21.15.98:0/1654970315 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5ccc19e050 con 0x7f5ccc100690 2023-07-14T19:29:50.995 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:50.990+0000 7f5cd0e84700 1 -- 172.21.15.98:0/1654970315 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5ccc10a380 con 0x7f5ccc100690 2023-07-14T19:29:50.995 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:50.991+0000 7f5cbb7fe700 1 -- 172.21.15.98:0/1654970315 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 17) v1 ==== 135766+0+0 (secure 0 0 0) 0x7f5cc40058a0 con 0x7f5ccc100690 2023-07-14T19:29:50.996 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:50.992+0000 7f5cbb7fe700 1 --2- 172.21.15.98:0/1654970315 >> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] conn(0x7f5cb40a1260 0x7f5cb40a3710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:29:50.996 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:50.992+0000 7f5cca59c700 1 --2- 172.21.15.98:0/1654970315 >> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] conn(0x7f5cb40a1260 0x7f5cb40a3710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:29:50.996 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:50.992+0000 7f5cbb7fe700 1 -- 172.21.15.98:0/1654970315 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 7979+0+0 (secure 0 0 0) 0x7f5cc40cc8b0 con 0x7f5ccc100690 2023-07-14T19:29:50.997 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:50.992+0000 7f5cca59c700 1 --2- 172.21.15.98:0/1654970315 >> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] conn(0x7f5cb40a1260 0x7f5cb40a3710 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f5cc0006fd0 tx=0x7f5cc0008040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:29:50.997 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:50.995+0000 7f5cbb7fe700 1 -- 172.21.15.98:0/1654970315 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f5cc4095f50 con 0x7f5ccc100690 2023-07-14T19:29:51.184 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:51.183+0000 7f5cd0e84700 1 -- 172.21.15.98:0/1654970315 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f5ccc19e660 con 0x7f5ccc100690 2023-07-14T19:29:51.185 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:51.183+0000 7f5cbb7fe700 1 -- 172.21.15.98:0/1654970315 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v83) v1 ==== 93+0+10 (secure 0 0 0) 0x7f5cc4099570 con 0x7f5ccc100690 2023-07-14T19:29:51.185 INFO:teuthology.orchestra.run.smithi098.stdout:pg_num: 1 2023-07-14T19:29:51.186 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:51.185+0000 7f5cd0e84700 1 -- 172.21.15.98:0/1654970315 >> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] conn(0x7f5cb40a1260 msgr2=0x7f5cb40a3710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:29:51.186 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:51.185+0000 7f5cd0e84700 1 --2- 172.21.15.98:0/1654970315 >> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] conn(0x7f5cb40a1260 0x7f5cb40a3710 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f5cc0006fd0 tx=0x7f5cc0008040 comp rx=0 tx=0).stop 2023-07-14T19:29:51.186 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:51.185+0000 7f5cd0e84700 1 -- 172.21.15.98:0/1654970315 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5ccc100690 msgr2=0x7f5ccc19cfc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:29:51.186 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:51.185+0000 7f5cd0e84700 1 --2- 172.21.15.98:0/1654970315 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5ccc100690 0x7f5ccc19cfc0 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7f5cc4007ee0 tx=0x7f5cc400d3b0 comp rx=0 tx=0).stop 2023-07-14T19:29:51.187 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:51.185+0000 7f5cd0e84700 1 -- 172.21.15.98:0/1654970315 shutdown_connections 2023-07-14T19:29:51.187 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:51.185+0000 7f5cd0e84700 1 --2- 172.21.15.98:0/1654970315 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5ccc100690 0x7f5ccc19cfc0 unknown :-1 s=CLOSED pgs=298 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:29:51.187 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:51.185+0000 7f5cd0e84700 1 --2- 172.21.15.98:0/1654970315 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f5ccc0feb10 0x7f5ccc195a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:29:51.187 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:51.185+0000 7f5cd0e84700 1 --2- 172.21.15.98:0/1654970315 >> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] conn(0x7f5cb40a1260 0x7f5cb40a3710 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:29:51.187 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:51.185+0000 7f5cd0e84700 1 --2- 172.21.15.98:0/1654970315 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f5ccc0ffd00 0x7f5ccc195f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:29:51.187 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:51.185+0000 7f5cd0e84700 1 -- 172.21.15.98:0/1654970315 >> 172.21.15.98:0/1654970315 conn(0x7f5ccc0fa130 msgr2=0x7f5ccc068850 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:29:51.187 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:51.186+0000 7f5cd0e84700 1 -- 172.21.15.98:0/1654970315 shutdown_connections 2023-07-14T19:29:51.188 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:51.186+0000 7f5cd0e84700 1 -- 172.21.15.98:0/1654970315 wait complete. 2023-07-14T19:29:51.198 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-07-14T19:29:51.325 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:51.324+0000 7fd2a5f98700 1 -- 172.21.15.98:0/1105854953 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd2a0100e40 msgr2=0x7fd2a0101290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:29:51.325 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:51.324+0000 7fd2a5f98700 1 --2- 172.21.15.98:0/1105854953 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd2a0100e40 0x7fd2a0101290 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7fd290009990 tx=0x7fd290009ca0 comp rx=0 tx=0).stop 2023-07-14T19:29:51.325 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:51.324+0000 7fd2a5f98700 1 -- 172.21.15.98:0/1105854953 shutdown_connections 2023-07-14T19:29:51.325 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:51.324+0000 7fd2a5f98700 1 --2- 172.21.15.98:0/1105854953 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fd2a00ffc50 0x7fd2a0100060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:29:51.325 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:51.324+0000 7fd2a5f98700 1 --2- 172.21.15.98:0/1105854953 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7fd2a01017d0 0x7fd2a0108090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:29:51.326 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:51.324+0000 7fd2a5f98700 1 --2- 172.21.15.98:0/1105854953 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd2a0100e40 0x7fd2a0101290 unknown :-1 s=CLOSED pgs=299 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:29:51.326 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:51.324+0000 7fd2a5f98700 1 -- 172.21.15.98:0/1105854953 >> 172.21.15.98:0/1105854953 conn(0x7fd2a00fb290 msgr2=0x7fd2a00fd6a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:29:51.326 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:51.324+0000 7fd2a5f98700 1 -- 172.21.15.98:0/1105854953 shutdown_connections 2023-07-14T19:29:51.326 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:51.324+0000 7fd2a5f98700 1 -- 172.21.15.98:0/1105854953 wait complete. 2023-07-14T19:29:51.326 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:51.324+0000 7fd2a5f98700 1 Processor -- start 2023-07-14T19:29:51.328 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:51.325+0000 7fd2a5f98700 1 -- start start 2023-07-14T19:29:51.329 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:51.325+0000 7fd2a5f98700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fd2a00ffc50 0x7fd2a01959a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:29:51.329 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:51.325+0000 7fd2a5f98700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd2a0100e40 0x7fd2a0195ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:29:51.329 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:51.325+0000 7fd2a5f98700 1 --2- >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7fd2a01017d0 0x7fd2a019cf50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:29:51.329 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:51.325+0000 7fd2a5f98700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd2a019d520 con 0x7fd2a0100e40 2023-07-14T19:29:51.329 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:51.325+0000 7fd2a5f98700 1 -- --> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd2a019d660 con 0x7fd2a01017d0 2023-07-14T19:29:51.329 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:51.325+0000 7fd2a5f98700 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd2a019d7f0 con 0x7fd2a00ffc50 2023-07-14T19:29:51.330 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:51.325+0000 7fd29effd700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd2a0100e40 0x7fd2a0195ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:29:51.330 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:51.325+0000 7fd29effd700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd2a0100e40 0x7fd2a0195ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:46342/0 (socket says 172.21.15.98:46342) 2023-07-14T19:29:51.330 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:51.325+0000 7fd29effd700 1 -- 172.21.15.98:0/3187002787 learned_addr learned my addr 172.21.15.98:0/3187002787 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-07-14T19:29:51.330 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:51.325+0000 7fd29ffff700 1 --2- 172.21.15.98:0/3187002787 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7fd2a01017d0 0x7fd2a019cf50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:29:51.330 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:51.325+0000 7fd29f7fe700 1 --2- 172.21.15.98:0/3187002787 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fd2a00ffc50 0x7fd2a01959a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:29:51.330 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:51.325+0000 7fd29effd700 1 -- 172.21.15.98:0/3187002787 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fd2a00ffc50 msgr2=0x7fd2a01959a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:29:51.330 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:51.325+0000 7fd29effd700 1 --2- 172.21.15.98:0/3187002787 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fd2a00ffc50 0x7fd2a01959a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:29:51.331 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:51.325+0000 7fd29effd700 1 -- 172.21.15.98:0/3187002787 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7fd2a01017d0 msgr2=0x7fd2a019cf50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:29:51.331 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:51.325+0000 7fd29effd700 1 --2- 172.21.15.98:0/3187002787 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7fd2a01017d0 0x7fd2a019cf50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:29:51.331 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:51.325+0000 7fd29effd700 1 -- 172.21.15.98:0/3187002787 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd2900095f0 con 0x7fd2a0100e40 2023-07-14T19:29:51.331 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:51.325+0000 7fd29ffff700 1 --2- 172.21.15.98:0/3187002787 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7fd2a01017d0 0x7fd2a019cf50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-07-14T19:29:51.331 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:51.325+0000 7fd29effd700 1 --2- 172.21.15.98:0/3187002787 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd2a0100e40 0x7fd2a0195ee0 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7fd290005910 tx=0x7fd290005e80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:29:51.331 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:51.325+0000 7fd29cff9700 1 -- 172.21.15.98:0/3187002787 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd29001d070 con 0x7fd2a0100e40 2023-07-14T19:29:51.331 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:51.325+0000 7fd2a5f98700 1 -- 172.21.15.98:0/3187002787 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd2a019da70 con 0x7fd2a0100e40 2023-07-14T19:29:51.332 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:51.325+0000 7fd29cff9700 1 -- 172.21.15.98:0/3187002787 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fd290004060 con 0x7fd2a0100e40 2023-07-14T19:29:51.332 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:51.326+0000 7fd29cff9700 1 -- 172.21.15.98:0/3187002787 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd290022e10 con 0x7fd2a0100e40 2023-07-14T19:29:51.332 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:51.326+0000 7fd2a5f98700 1 -- 172.21.15.98:0/3187002787 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd2a019de90 con 0x7fd2a0100e40 2023-07-14T19:29:51.332 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:51.326+0000 7fd2a5f98700 1 -- 172.21.15.98:0/3187002787 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd2a010b470 con 0x7fd2a0100e40 2023-07-14T19:29:51.332 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:51.328+0000 7fd29cff9700 1 -- 172.21.15.98:0/3187002787 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 17) v1 ==== 135766+0+0 (secure 0 0 0) 0x7fd290022470 con 0x7fd2a0100e40 2023-07-14T19:29:51.332 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:51.328+0000 7fd29cff9700 1 --2- 172.21.15.98:0/3187002787 >> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] conn(0x7fd2880a11e0 0x7fd2880a3690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:29:51.332 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:51.328+0000 7fd29f7fe700 1 --2- 172.21.15.98:0/3187002787 >> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] conn(0x7fd2880a11e0 0x7fd2880a3690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:29:51.333 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:51.328+0000 7fd29cff9700 1 -- 172.21.15.98:0/3187002787 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 7979+0+0 (secure 0 0 0) 0x7fd2900cde20 con 0x7fd2a0100e40 2023-07-14T19:29:51.333 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:51.328+0000 7fd29f7fe700 1 --2- 172.21.15.98:0/3187002787 >> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] conn(0x7fd2880a11e0 0x7fd2880a3690 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7fd294005950 tx=0x7fd2940058e0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:29:51.333 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:51.331+0000 7fd29cff9700 1 -- 172.21.15.98:0/3187002787 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7fd2900975d0 con 0x7fd2a0100e40 2023-07-14T19:29:51.524 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:51.523+0000 7fd2a5f98700 1 -- 172.21.15.98:0/3187002787 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fd2a0106b00 con 0x7fd2a0100e40 2023-07-14T19:29:51.524 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:51.523+0000 7fd29cff9700 1 -- 172.21.15.98:0/3187002787 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v83) v1 ==== 104+0+11 (secure 0 0 0) 0x7fd29009abf0 con 0x7fd2a0100e40 2023-07-14T19:29:51.524 INFO:teuthology.orchestra.run.smithi098.stdout:pg_num: 32 2023-07-14T19:29:51.526 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:51.526+0000 7fd2a5f98700 1 -- 172.21.15.98:0/3187002787 >> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] conn(0x7fd2880a11e0 msgr2=0x7fd2880a3690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:29:51.527 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:51.526+0000 7fd2a5f98700 1 --2- 172.21.15.98:0/3187002787 >> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] conn(0x7fd2880a11e0 0x7fd2880a3690 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7fd294005950 tx=0x7fd2940058e0 comp rx=0 tx=0).stop 2023-07-14T19:29:51.527 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:51.526+0000 7fd2a5f98700 1 -- 172.21.15.98:0/3187002787 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd2a0100e40 msgr2=0x7fd2a0195ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:29:51.527 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:51.526+0000 7fd2a5f98700 1 --2- 172.21.15.98:0/3187002787 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd2a0100e40 0x7fd2a0195ee0 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7fd290005910 tx=0x7fd290005e80 comp rx=0 tx=0).stop 2023-07-14T19:29:51.527 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:51.526+0000 7fd2a5f98700 1 -- 172.21.15.98:0/3187002787 shutdown_connections 2023-07-14T19:29:51.528 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:51.526+0000 7fd2a5f98700 1 --2- 172.21.15.98:0/3187002787 >> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] conn(0x7fd2880a11e0 0x7fd2880a3690 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:29:51.528 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:51.526+0000 7fd2a5f98700 1 --2- 172.21.15.98:0/3187002787 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fd2a00ffc50 0x7fd2a01959a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:29:51.528 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:51.526+0000 7fd2a5f98700 1 --2- 172.21.15.98:0/3187002787 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7fd2a01017d0 0x7fd2a019cf50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:29:51.528 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:51.526+0000 7fd2a5f98700 1 --2- 172.21.15.98:0/3187002787 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd2a0100e40 0x7fd2a0195ee0 unknown :-1 s=CLOSED pgs=300 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:29:51.528 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:51.526+0000 7fd2a5f98700 1 -- 172.21.15.98:0/3187002787 >> 172.21.15.98:0/3187002787 conn(0x7fd2a00fb290 msgr2=0x7fd2a00fd470 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:29:51.529 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:51.526+0000 7fd2a5f98700 1 -- 172.21.15.98:0/3187002787 shutdown_connections 2023-07-14T19:29:51.529 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:51.526+0000 7fd2a5f98700 1 -- 172.21.15.98:0/3187002787 wait complete. 2023-07-14T19:29:51.538 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-07-14T19:29:51.663 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:51.662+0000 7fbc1ceb5700 1 -- 172.21.15.98:0/3462744378 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbc18103010 msgr2=0x7fbc181053f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:29:51.663 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:51.662+0000 7fbc1ceb5700 1 --2- 172.21.15.98:0/3462744378 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbc18103010 0x7fbc181053f0 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7fbc0c009990 tx=0x7fbc0c009ca0 comp rx=0 tx=0).stop 2023-07-14T19:29:51.664 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:51.663+0000 7fbc1ceb5700 1 -- 172.21.15.98:0/3462744378 shutdown_connections 2023-07-14T19:29:51.664 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:51.663+0000 7fbc1ceb5700 1 --2- 172.21.15.98:0/3462744378 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fbc18105930 0x7fbc18107d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:29:51.664 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:51.663+0000 7fbc1ceb5700 1 --2- 172.21.15.98:0/3462744378 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7fbc181006f0 0x7fbc18102ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:29:51.664 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:51.663+0000 7fbc1ceb5700 1 --2- 172.21.15.98:0/3462744378 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbc18103010 0x7fbc181053f0 unknown :-1 s=CLOSED pgs=301 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:29:51.664 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:51.663+0000 7fbc1ceb5700 1 -- 172.21.15.98:0/3462744378 >> 172.21.15.98:0/3462744378 conn(0x7fbc180fa130 msgr2=0x7fbc180fc560 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:29:51.664 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:51.663+0000 7fbc1ceb5700 1 -- 172.21.15.98:0/3462744378 shutdown_connections 2023-07-14T19:29:51.665 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:51.663+0000 7fbc1ceb5700 1 -- 172.21.15.98:0/3462744378 wait complete. 2023-07-14T19:29:51.666 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:51.663+0000 7fbc1ceb5700 1 Processor -- start 2023-07-14T19:29:51.666 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:51.663+0000 7fbc1ceb5700 1 -- start start 2023-07-14T19:29:51.667 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:51.664+0000 7fbc1ceb5700 1 --2- >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7fbc181006f0 0x7fbc18100060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:29:51.667 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:51.664+0000 7fbc1ceb5700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbc18103010 0x7fbc180fe6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:29:51.667 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:51.664+0000 7fbc1ceb5700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fbc18105930 0x7fbc180febe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:29:51.667 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:51.664+0000 7fbc1ceb5700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbc180ff380 con 0x7fbc18103010 2023-07-14T19:29:51.667 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:51.664+0000 7fbc1ceb5700 1 -- --> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbc180ff4f0 con 0x7fbc181006f0 2023-07-14T19:29:51.667 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:51.664+0000 7fbc1ceb5700 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbc1819f9c0 con 0x7fbc18105930 2023-07-14T19:29:51.667 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:51.664+0000 7fbc15d9b700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbc18103010 0x7fbc180fe6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:29:51.668 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:51.664+0000 7fbc15d9b700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbc18103010 0x7fbc180fe6a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:46356/0 (socket says 172.21.15.98:46356) 2023-07-14T19:29:51.668 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:51.664+0000 7fbc15d9b700 1 -- 172.21.15.98:0/1946735337 learned_addr learned my addr 172.21.15.98:0/1946735337 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-07-14T19:29:51.668 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:51.664+0000 7fbc1659c700 1 --2- 172.21.15.98:0/1946735337 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7fbc181006f0 0x7fbc18100060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:29:51.668 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:51.664+0000 7fbc16d9d700 1 --2- 172.21.15.98:0/1946735337 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fbc18105930 0x7fbc180febe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:29:51.668 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:51.664+0000 7fbc15d9b700 1 -- 172.21.15.98:0/1946735337 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fbc18105930 msgr2=0x7fbc180febe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:29:51.668 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:51.664+0000 7fbc15d9b700 1 --2- 172.21.15.98:0/1946735337 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fbc18105930 0x7fbc180febe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:29:51.668 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:51.664+0000 7fbc15d9b700 1 -- 172.21.15.98:0/1946735337 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7fbc181006f0 msgr2=0x7fbc18100060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:29:51.669 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:51.664+0000 7fbc15d9b700 1 --2- 172.21.15.98:0/1946735337 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7fbc181006f0 0x7fbc18100060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:29:51.669 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:51.664+0000 7fbc15d9b700 1 -- 172.21.15.98:0/1946735337 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbc0c0095f0 con 0x7fbc18103010 2023-07-14T19:29:51.669 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:51.664+0000 7fbc15d9b700 1 --2- 172.21.15.98:0/1946735337 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbc18103010 0x7fbc180fe6a0 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7fbc0c005910 tx=0x7fbc0c00f690 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:29:51.670 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:51.666+0000 7fbc077fe700 1 -- 172.21.15.98:0/1946735337 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbc0c01d070 con 0x7fbc18103010 2023-07-14T19:29:51.670 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:51.666+0000 7fbc1ceb5700 1 -- 172.21.15.98:0/1946735337 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbc1819fc40 con 0x7fbc18103010 2023-07-14T19:29:51.670 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:51.666+0000 7fbc077fe700 1 -- 172.21.15.98:0/1946735337 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fbc0c022470 con 0x7fbc18103010 2023-07-14T19:29:51.670 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:51.666+0000 7fbc077fe700 1 -- 172.21.15.98:0/1946735337 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbc0c017650 con 0x7fbc18103010 2023-07-14T19:29:51.670 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:51.666+0000 7fbc1ceb5700 1 -- 172.21.15.98:0/1946735337 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbc181a00c0 con 0x7fbc18103010 2023-07-14T19:29:51.671 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:51.667+0000 7fbc1ceb5700 1 -- 172.21.15.98:0/1946735337 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbc1810b420 con 0x7fbc18103010 2023-07-14T19:29:51.671 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:51.668+0000 7fbc077fe700 1 -- 172.21.15.98:0/1946735337 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 17) v1 ==== 135766+0+0 (secure 0 0 0) 0x7fbc0c00fad0 con 0x7fbc18103010 2023-07-14T19:29:51.671 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:51.669+0000 7fbc077fe700 1 --2- 172.21.15.98:0/1946735337 >> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] conn(0x7fbc000a11e0 0x7fbc000a3690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:29:51.671 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:51.669+0000 7fbc1659c700 1 --2- 172.21.15.98:0/1946735337 >> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] conn(0x7fbc000a11e0 0x7fbc000a3690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:29:51.671 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:51.669+0000 7fbc077fe700 1 -- 172.21.15.98:0/1946735337 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 7979+0+0 (secure 0 0 0) 0x7fbc0c0cdbd0 con 0x7fbc18103010 2023-07-14T19:29:51.672 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:51.669+0000 7fbc1659c700 1 --2- 172.21.15.98:0/1946735337 >> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] conn(0x7fbc000a11e0 0x7fbc000a3690 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7fbc08005950 tx=0x7fbc0800a300 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:29:51.673 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:51.673+0000 7fbc077fe700 1 -- 172.21.15.98:0/1946735337 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7fbc0c097400 con 0x7fbc18103010 2023-07-14T19:29:51.863 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:51.863+0000 7fbc1ceb5700 1 -- 172.21.15.98:0/1946735337 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7fbc181a0670 con 0x7fbc18103010 2023-07-14T19:29:51.864 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:51.863+0000 7fbc077fe700 1 -- 172.21.15.98:0/1946735337 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v83) v1 ==== 100+0+11 (secure 0 0 0) 0x7fbc0c027020 con 0x7fbc18103010 2023-07-14T19:29:51.864 INFO:teuthology.orchestra.run.smithi098.stdout:pg_num: 64 2023-07-14T19:29:51.866 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:51.866+0000 7fbc1ceb5700 1 -- 172.21.15.98:0/1946735337 >> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] conn(0x7fbc000a11e0 msgr2=0x7fbc000a3690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:29:51.866 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:51.866+0000 7fbc1ceb5700 1 --2- 172.21.15.98:0/1946735337 >> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] conn(0x7fbc000a11e0 0x7fbc000a3690 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7fbc08005950 tx=0x7fbc0800a300 comp rx=0 tx=0).stop 2023-07-14T19:29:51.867 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:51.866+0000 7fbc1ceb5700 1 -- 172.21.15.98:0/1946735337 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbc18103010 msgr2=0x7fbc180fe6a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:29:51.867 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:51.866+0000 7fbc1ceb5700 1 --2- 172.21.15.98:0/1946735337 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbc18103010 0x7fbc180fe6a0 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7fbc0c005910 tx=0x7fbc0c00f690 comp rx=0 tx=0).stop 2023-07-14T19:29:51.867 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:51.866+0000 7fbc1ceb5700 1 -- 172.21.15.98:0/1946735337 shutdown_connections 2023-07-14T19:29:51.867 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:51.866+0000 7fbc1ceb5700 1 --2- 172.21.15.98:0/1946735337 >> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] conn(0x7fbc000a11e0 0x7fbc000a3690 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:29:51.867 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:51.866+0000 7fbc1ceb5700 1 --2- 172.21.15.98:0/1946735337 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fbc18105930 0x7fbc180febe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:29:51.868 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:51.866+0000 7fbc1ceb5700 1 --2- 172.21.15.98:0/1946735337 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7fbc181006f0 0x7fbc18100060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:29:51.868 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:51.866+0000 7fbc1ceb5700 1 --2- 172.21.15.98:0/1946735337 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbc18103010 0x7fbc180fe6a0 unknown :-1 s=CLOSED pgs=302 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:29:51.868 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:51.866+0000 7fbc1ceb5700 1 -- 172.21.15.98:0/1946735337 >> 172.21.15.98:0/1946735337 conn(0x7fbc180fa130 msgr2=0x7fbc18101b60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:29:51.868 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:51.866+0000 7fbc1ceb5700 1 -- 172.21.15.98:0/1946735337 shutdown_connections 2023-07-14T19:29:51.868 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:51.866+0000 7fbc1ceb5700 1 -- 172.21.15.98:0/1946735337 wait complete. 2023-07-14T19:29:51.878 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:29:51 smithi098 ceph-mon[127486]: from='client.? 172.21.15.98:0/2749659362' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-07-14T19:29:51.879 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:29:51 smithi098 ceph-mon[127486]: from='client.? 172.21.15.98:0/1654970315' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2023-07-14T19:29:51.879 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:29:51 smithi098 ceph-mon[127486]: from='client.? 172.21.15.98:0/3187002787' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2023-07-14T19:29:51.880 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-07-14T19:29:51.941 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:29:51 smithi101 ceph-mon[128807]: from='client.? 172.21.15.98:0/2749659362' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-07-14T19:29:51.941 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:29:51 smithi101 ceph-mon[128807]: from='client.? 172.21.15.98:0/1654970315' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2023-07-14T19:29:51.942 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:29:51 smithi101 ceph-mon[128807]: from='client.? 172.21.15.98:0/3187002787' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2023-07-14T19:29:51.970 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:29:51 smithi139 ceph-mon[129293]: from='client.? 172.21.15.98:0/2749659362' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-07-14T19:29:51.970 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:29:51 smithi139 ceph-mon[129293]: from='client.? 172.21.15.98:0/1654970315' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2023-07-14T19:29:51.970 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:29:51 smithi139 ceph-mon[129293]: from='client.? 172.21.15.98:0/3187002787' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2023-07-14T19:29:52.001 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:52.001+0000 7fe9653b7700 1 -- 172.21.15.98:0/2823102490 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe9600fe2d0 msgr2=0x7fe9600fe6e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:29:52.002 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:52.001+0000 7fe9653b7700 1 --2- 172.21.15.98:0/2823102490 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe9600fe2d0 0x7fe9600fe6e0 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7fe950009990 tx=0x7fe950009ca0 comp rx=0 tx=0).stop 2023-07-14T19:29:52.002 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:52.001+0000 7fe9653b7700 1 -- 172.21.15.98:0/2823102490 shutdown_connections 2023-07-14T19:29:52.002 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:52.001+0000 7fe9653b7700 1 --2- 172.21.15.98:0/2823102490 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7fe9600ffe50 0x7fe960106710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:29:52.002 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:52.001+0000 7fe9653b7700 1 --2- 172.21.15.98:0/2823102490 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe9600fe2d0 0x7fe9600fe6e0 unknown :-1 s=CLOSED pgs=303 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:29:52.002 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:52.001+0000 7fe9653b7700 1 --2- 172.21.15.98:0/2823102490 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fe9600ff4c0 0x7fe9600ff910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:29:52.003 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:52.001+0000 7fe9653b7700 1 -- 172.21.15.98:0/2823102490 >> 172.21.15.98:0/2823102490 conn(0x7fe9600f98d0 msgr2=0x7fe9600fbd20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:29:52.003 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:52.001+0000 7fe9653b7700 1 -- 172.21.15.98:0/2823102490 shutdown_connections 2023-07-14T19:29:52.003 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:52.001+0000 7fe9653b7700 1 -- 172.21.15.98:0/2823102490 wait complete. 2023-07-14T19:29:52.003 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:52.001+0000 7fe9653b7700 1 Processor -- start 2023-07-14T19:29:52.005 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:52.002+0000 7fe9653b7700 1 -- start start 2023-07-14T19:29:52.005 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:52.002+0000 7fe9653b7700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fe9600fe2d0 0x7fe9600fbca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:29:52.005 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:52.002+0000 7fe9653b7700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe9600ff4c0 0x7fe96019a6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:29:52.005 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:52.002+0000 7fe9653b7700 1 --2- >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7fe9600ffe50 0x7fe96019cd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:29:52.005 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:52.002+0000 7fe9653b7700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe96019d4f0 con 0x7fe9600ff4c0 2023-07-14T19:29:52.006 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:52.002+0000 7fe9653b7700 1 -- --> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe96019d660 con 0x7fe9600ffe50 2023-07-14T19:29:52.006 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:52.002+0000 7fe9653b7700 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe96019d820 con 0x7fe9600fe2d0 2023-07-14T19:29:52.006 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:52.002+0000 7fe95e7fc700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe9600ff4c0 0x7fe96019a6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:29:52.006 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:52.002+0000 7fe95e7fc700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe9600ff4c0 0x7fe96019a6d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:46368/0 (socket says 172.21.15.98:46368) 2023-07-14T19:29:52.006 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:52.002+0000 7fe95e7fc700 1 -- 172.21.15.98:0/1153644383 learned_addr learned my addr 172.21.15.98:0/1153644383 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-07-14T19:29:52.006 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:52.002+0000 7fe95effd700 1 --2- 172.21.15.98:0/1153644383 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fe9600fe2d0 0x7fe9600fbca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:29:52.006 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:52.002+0000 7fe95e7fc700 1 -- 172.21.15.98:0/1153644383 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fe9600fe2d0 msgr2=0x7fe9600fbca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:29:52.007 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:52.002+0000 7fe95e7fc700 1 --2- 172.21.15.98:0/1153644383 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fe9600fe2d0 0x7fe9600fbca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:29:52.007 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:52.002+0000 7fe95e7fc700 1 -- 172.21.15.98:0/1153644383 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7fe9600ffe50 msgr2=0x7fe96019cd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:29:52.007 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:52.002+0000 7fe95f7fe700 1 --2- 172.21.15.98:0/1153644383 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7fe9600ffe50 0x7fe96019cd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:29:52.007 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:52.002+0000 7fe95e7fc700 1 --2- 172.21.15.98:0/1153644383 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7fe9600ffe50 0x7fe96019cd20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:29:52.007 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:52.002+0000 7fe95e7fc700 1 -- 172.21.15.98:0/1153644383 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe9500095f0 con 0x7fe9600ff4c0 2023-07-14T19:29:52.007 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:52.002+0000 7fe95effd700 1 --2- 172.21.15.98:0/1153644383 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fe9600fe2d0 0x7fe9600fbca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-07-14T19:29:52.008 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:52.002+0000 7fe95e7fc700 1 --2- 172.21.15.98:0/1153644383 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe9600ff4c0 0x7fe96019a6d0 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7fe94800d980 tx=0x7fe94800dd40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:29:52.008 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:52.002+0000 7fe957fff700 1 -- 172.21.15.98:0/1153644383 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe948009b40 con 0x7fe9600ff4c0 2023-07-14T19:29:52.008 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:52.003+0000 7fe957fff700 1 -- 172.21.15.98:0/1153644383 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fe94800de60 con 0x7fe9600ff4c0 2023-07-14T19:29:52.008 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:52.003+0000 7fe9653b7700 1 -- 172.21.15.98:0/1153644383 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe96019db00 con 0x7fe9600ff4c0 2023-07-14T19:29:52.008 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:52.003+0000 7fe957fff700 1 -- 172.21.15.98:0/1153644383 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe948005580 con 0x7fe9600ff4c0 2023-07-14T19:29:52.008 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:52.003+0000 7fe9653b7700 1 -- 172.21.15.98:0/1153644383 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe96019dfc0 con 0x7fe9600ff4c0 2023-07-14T19:29:52.008 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:52.003+0000 7fe9653b7700 1 -- 172.21.15.98:0/1153644383 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe960105e40 con 0x7fe9600ff4c0 2023-07-14T19:29:52.009 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:52.004+0000 7fe957fff700 1 -- 172.21.15.98:0/1153644383 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 17) v1 ==== 135766+0+0 (secure 0 0 0) 0x7fe948004750 con 0x7fe9600ff4c0 2023-07-14T19:29:52.009 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:52.005+0000 7fe957fff700 1 --2- 172.21.15.98:0/1153644383 >> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] conn(0x7fe94c0a1220 0x7fe94c0a36d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:29:52.010 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:52.005+0000 7fe957fff700 1 -- 172.21.15.98:0/1153644383 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 7979+0+0 (secure 0 0 0) 0x7fe9480ccce0 con 0x7fe9600ff4c0 2023-07-14T19:29:52.010 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:52.005+0000 7fe95effd700 1 --2- 172.21.15.98:0/1153644383 >> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] conn(0x7fe94c0a1220 0x7fe94c0a36d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:29:52.010 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:52.005+0000 7fe95effd700 1 --2- 172.21.15.98:0/1153644383 >> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] conn(0x7fe94c0a1220 0x7fe94c0a36d0 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7fe950006010 tx=0x7fe95000b540 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:29:52.010 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:52.008+0000 7fe957fff700 1 -- 172.21.15.98:0/1153644383 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7fe948096480 con 0x7fe9600ff4c0 2023-07-14T19:29:52.239 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:52.239+0000 7fe9653b7700 1 -- 172.21.15.98:0/1153644383 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fe96002c8e0 con 0x7fe9600ff4c0 2023-07-14T19:29:52.240 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:52.240+0000 7fe957fff700 1 -- 172.21.15.98:0/1153644383 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 22 v22) v1 ==== 94+0+10826 (secure 0 0 0) 0x7fe948017070 con 0x7fe9600ff4c0 2023-07-14T19:29:52.240 INFO:teuthology.orchestra.run.smithi098.stdout: 2023-07-14T19:29:52.241 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":22,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14565,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6835/586280675","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6834","nonce":586280675},{"type":"v1","addr":"172.21.15.98:6835","nonce":586280675}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":4},{"gid":14577,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6837/487577257","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6836","nonce":487577257},{"type":"v1","addr":"172.21.15.98:6837","nonce":487577257}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":5},{"gid":14634,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6839/4085731209","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6838","nonce":4085731209},{"type":"v1","addr":"172.21.15.98:6839","nonce":4085731209}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":11},{"gid":14646,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6841/1679318259","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6840","nonce":1679318259},{"type":"v1","addr":"172.21.15.98:6841","nonce":1679318259}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":12},{"gid":24421,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.139:6833/3505477820","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6832","nonce":3505477820},{"type":"v1","addr":"172.21.15.139:6833","nonce":3505477820}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":24437,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.101:6833/3474460171","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.101:6832","nonce":3474460171},{"type":"v1","addr":"172.21.15.101:6833","nonce":3474460171}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":22,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2023-07-14T19:29:01.033156+0000","modified":"2023-07-14T19:29:06.291916+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24521,"mds_1":24493,"mds_2":24479},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24455":{"gid":24455,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.101:6835/1583264648","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.101:6834","nonce":1583264648},{"type":"v1","addr":"172.21.15.101:6835","nonce":1583264648}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24460":{"gid":24460,"name":"f","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.139:6835/2315281724","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6834","nonce":2315281724},{"type":"v1","addr":"172.21.15.139:6835","nonce":2315281724}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24469":{"gid":24469,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.139:6837/3597660552","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6836","nonce":3597660552},{"type":"v1","addr":"172.21.15.139:6837","nonce":3597660552}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24479":{"gid":24479,"name":"h","rank":2,"incarnation":21,"state":"up:active","state_seq":6,"addr":"172.21.15.101:6837/485741528","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.101:6836","nonce":485741528},{"type":"v1","addr":"172.21.15.101:6837","nonce":485741528}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24493":{"gid":24493,"name":"i","rank":1,"incarnation":19,"state":"up:active","state_seq":5,"addr":"172.21.15.139:6839/1258854143","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6838","nonce":1258854143},{"type":"v1","addr":"172.21.15.139:6839","nonce":1258854143}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24521":{"gid":24521,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.101:6839/48393381","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.101:6838","nonce":48393381},{"type":"v1","addr":"172.21.15.101:6839","nonce":48393381}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-07-14T19:29:52.243 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:52.242+0000 7fe9653b7700 1 -- 172.21.15.98:0/1153644383 >> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] conn(0x7fe94c0a1220 msgr2=0x7fe94c0a36d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:29:52.243 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:52.242+0000 7fe9653b7700 1 --2- 172.21.15.98:0/1153644383 >> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] conn(0x7fe94c0a1220 0x7fe94c0a36d0 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7fe950006010 tx=0x7fe95000b540 comp rx=0 tx=0).stop 2023-07-14T19:29:52.243 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:52.242+0000 7fe9653b7700 1 -- 172.21.15.98:0/1153644383 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe9600ff4c0 msgr2=0x7fe96019a6d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:29:52.244 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:52.242+0000 7fe9653b7700 1 --2- 172.21.15.98:0/1153644383 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe9600ff4c0 0x7fe96019a6d0 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7fe94800d980 tx=0x7fe94800dd40 comp rx=0 tx=0).stop 2023-07-14T19:29:52.244 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:52.242+0000 7fe9653b7700 1 -- 172.21.15.98:0/1153644383 shutdown_connections 2023-07-14T19:29:52.244 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:52.242+0000 7fe9653b7700 1 --2- 172.21.15.98:0/1153644383 >> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] conn(0x7fe94c0a1220 0x7fe94c0a36d0 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:29:52.244 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:52.242+0000 7fe9653b7700 1 --2- 172.21.15.98:0/1153644383 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fe9600fe2d0 0x7fe9600fbca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:29:52.244 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:52.242+0000 7fe9653b7700 1 --2- 172.21.15.98:0/1153644383 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7fe9600ffe50 0x7fe96019cd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:29:52.245 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:52.242+0000 7fe9653b7700 1 --2- 172.21.15.98:0/1153644383 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe9600ff4c0 0x7fe96019a6d0 unknown :-1 s=CLOSED pgs=304 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:29:52.245 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:52.242+0000 7fe9653b7700 1 -- 172.21.15.98:0/1153644383 >> 172.21.15.98:0/1153644383 conn(0x7fe9600f98d0 msgr2=0x7fe960102f80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:29:52.245 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:52.242+0000 7fe9653b7700 1 -- 172.21.15.98:0/1153644383 shutdown_connections 2023-07-14T19:29:52.245 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:52.242+0000 7fe9653b7700 1 -- 172.21.15.98:0/1153644383 wait complete. 2023-07-14T19:29:52.245 INFO:teuthology.orchestra.run.smithi098.stderr:dumped fsmap epoch 22 2023-07-14T19:29:52.255 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-07-14T19:29:52.379 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:52.377+0000 7f94da676700 1 -- 172.21.15.98:0/2680712521 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f94d40feb20 msgr2=0x7f94d40fef30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:29:52.380 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:52.377+0000 7f94da676700 1 --2- 172.21.15.98:0/2680712521 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f94d40feb20 0x7f94d40fef30 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7f94c4009990 tx=0x7f94c4009ca0 comp rx=0 tx=0).stop 2023-07-14T19:29:52.380 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:52.377+0000 7f94da676700 1 -- 172.21.15.98:0/2680712521 shutdown_connections 2023-07-14T19:29:52.380 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:52.377+0000 7f94da676700 1 --2- 172.21.15.98:0/2680712521 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f94d41006a0 0x7f94d4106f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:29:52.380 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:52.377+0000 7f94da676700 1 --2- 172.21.15.98:0/2680712521 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f94d40feb20 0x7f94d40fef30 unknown :-1 s=CLOSED pgs=305 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:29:52.380 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:52.377+0000 7f94da676700 1 --2- 172.21.15.98:0/2680712521 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f94d40ffd10 0x7f94d4100160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:29:52.381 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:52.377+0000 7f94da676700 1 -- 172.21.15.98:0/2680712521 >> 172.21.15.98:0/2680712521 conn(0x7f94d40fa100 msgr2=0x7f94d40fc570 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:29:52.381 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:52.377+0000 7f94da676700 1 -- 172.21.15.98:0/2680712521 shutdown_connections 2023-07-14T19:29:52.381 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:52.377+0000 7f94da676700 1 -- 172.21.15.98:0/2680712521 wait complete. 2023-07-14T19:29:52.381 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:52.378+0000 7f94da676700 1 Processor -- start 2023-07-14T19:29:52.381 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:52.378+0000 7f94da676700 1 -- start start 2023-07-14T19:29:52.381 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:52.378+0000 7f94da676700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f94d40feb20 0x7f94d4193780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:29:52.381 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:52.378+0000 7f94da676700 1 --2- >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f94d40ffd10 0x7f94d4193cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:29:52.382 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:52.378+0000 7f94da676700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f94d41006a0 0x7f94d419ad30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:29:52.382 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:52.378+0000 7f94da676700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f94d419b300 con 0x7f94d40feb20 2023-07-14T19:29:52.382 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:52.378+0000 7f94da676700 1 -- --> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f94d419b440 con 0x7f94d40ffd10 2023-07-14T19:29:52.382 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:52.378+0000 7f94da676700 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f94d419b5d0 con 0x7f94d41006a0 2023-07-14T19:29:52.382 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:52.378+0000 7f94d3fff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f94d40feb20 0x7f94d4193780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:29:52.382 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:52.378+0000 7f94d3fff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f94d40feb20 0x7f94d4193780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:46380/0 (socket says 172.21.15.98:46380) 2023-07-14T19:29:52.382 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:52.378+0000 7f94d3fff700 1 -- 172.21.15.98:0/209642085 learned_addr learned my addr 172.21.15.98:0/209642085 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-07-14T19:29:52.383 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:52.378+0000 7f94d8c13700 1 --2- 172.21.15.98:0/209642085 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f94d41006a0 0x7f94d419ad30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:29:52.383 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:52.378+0000 7f94d37fe700 1 --2- 172.21.15.98:0/209642085 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f94d40ffd10 0x7f94d4193cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:29:52.383 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:52.378+0000 7f94d3fff700 1 -- 172.21.15.98:0/209642085 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f94d41006a0 msgr2=0x7f94d419ad30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:29:52.383 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:52.378+0000 7f94d3fff700 1 --2- 172.21.15.98:0/209642085 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f94d41006a0 0x7f94d419ad30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:29:52.383 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:52.378+0000 7f94d3fff700 1 -- 172.21.15.98:0/209642085 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f94d40ffd10 msgr2=0x7f94d4193cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:29:52.383 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:52.378+0000 7f94d3fff700 1 --2- 172.21.15.98:0/209642085 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f94d40ffd10 0x7f94d4193cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:29:52.383 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:52.378+0000 7f94d3fff700 1 -- 172.21.15.98:0/209642085 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f94c40095f0 con 0x7f94d40feb20 2023-07-14T19:29:52.384 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:52.378+0000 7f94d37fe700 1 --2- 172.21.15.98:0/209642085 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f94d40ffd10 0x7f94d4193cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-07-14T19:29:52.384 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:52.378+0000 7f94d3fff700 1 --2- 172.21.15.98:0/209642085 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f94d40feb20 0x7f94d4193780 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7f94c4006010 tx=0x7f94c4003dd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:29:52.384 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:52.379+0000 7f94d17fa700 1 -- 172.21.15.98:0/209642085 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f94c401c070 con 0x7f94d40feb20 2023-07-14T19:29:52.384 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:52.379+0000 7f94da676700 1 -- 172.21.15.98:0/209642085 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f94d419b850 con 0x7f94d40feb20 2023-07-14T19:29:52.384 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:52.379+0000 7f94d17fa700 1 -- 172.21.15.98:0/209642085 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f94c4004350 con 0x7f94d40feb20 2023-07-14T19:29:52.385 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:52.379+0000 7f94d17fa700 1 -- 172.21.15.98:0/209642085 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f94c4004c80 con 0x7f94d40feb20 2023-07-14T19:29:52.385 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:52.379+0000 7f94da676700 1 -- 172.21.15.98:0/209642085 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f94d419bc70 con 0x7f94d40feb20 2023-07-14T19:29:52.386 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:52.380+0000 7f94da676700 1 -- 172.21.15.98:0/209642085 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f94d4106100 con 0x7f94d40feb20 2023-07-14T19:29:52.386 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:52.381+0000 7f94d17fa700 1 -- 172.21.15.98:0/209642085 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 17) v1 ==== 135766+0+0 (secure 0 0 0) 0x7f94c400d020 con 0x7f94d40feb20 2023-07-14T19:29:52.386 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:52.381+0000 7f94d17fa700 1 --2- 172.21.15.98:0/209642085 >> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] conn(0x7f94bc0a1220 0x7f94bc0a36d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:29:52.387 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:52.381+0000 7f94d37fe700 1 --2- 172.21.15.98:0/209642085 >> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] conn(0x7f94bc0a1220 0x7f94bc0a36d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:29:52.387 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:52.381+0000 7f94d17fa700 1 -- 172.21.15.98:0/209642085 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 7979+0+0 (secure 0 0 0) 0x7f94c40cd920 con 0x7f94d40feb20 2023-07-14T19:29:52.387 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:52.382+0000 7f94d37fe700 1 --2- 172.21.15.98:0/209642085 >> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] conn(0x7f94bc0a1220 0x7f94bc0a36d0 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f94c8009cb0 tx=0x7f94c8009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:29:52.387 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:52.385+0000 7f94d17fa700 1 -- 172.21.15.98:0/209642085 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f94c4097150 con 0x7f94d40feb20 2023-07-14T19:29:52.579 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:52.579+0000 7f94da676700 1 -- 172.21.15.98:0/209642085 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f94d402c8e0 con 0x7f94d40feb20 2023-07-14T19:29:52.580 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:52.580+0000 7f94d17fa700 1 -- 172.21.15.98:0/209642085 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v83) v1 ==== 74+0+22117 (secure 0 0 0) 0x7f94c409a770 con 0x7f94d40feb20 2023-07-14T19:29:52.581 INFO:teuthology.orchestra.run.smithi098.stdout: 2023-07-14T19:29:52.581 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":83,"fsid":"9adea648-227b-11ee-9b33-001a4aab830c","created":"2023-07-14T19:22:07.178854+0000","modified":"2023-07-14T19:29:02.293099+0000","last_up_change":"2023-07-14T19:28:19.646827+0000","last_in_change":"2023-07-14T19:28:02.018325+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-07-14T19:25:09.845827+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2023-07-14T19:28:58.952921+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"82","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2023-07-14T19:29:00.228272+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"83","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"a3797e89-1754-41cf-9517-227e06ffdd72","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6802","nonce":120507228},{"type":"v1","addr":"172.21.15.98:6803","nonce":120507228}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6804","nonce":120507228},{"type":"v1","addr":"172.21.15.98:6805","nonce":120507228}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6808","nonce":120507228},{"type":"v1","addr":"172.21.15.98:6809","nonce":120507228}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6806","nonce":120507228},{"type":"v1","addr":"172.21.15.98:6807","nonce":120507228}]},"public_addr":"172.21.15.98:6803/120507228","cluster_addr":"172.21.15.98:6805/120507228","heartbeat_back_addr":"172.21.15.98:6809/120507228","heartbeat_front_addr":"172.21.15.98:6807/120507228","state":["exists","up"]},{"osd":1,"uuid":"2b001add-1aaa-4496-a403-7ef9725b80a0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.101:6800","nonce":2450584584},{"type":"v1","addr":"172.21.15.101:6801","nonce":2450584584}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.101:6802","nonce":2450584584},{"type":"v1","addr":"172.21.15.101:6803","nonce":2450584584}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.101:6806","nonce":2450584584},{"type":"v1","addr":"172.21.15.101:6807","nonce":2450584584}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.101:6804","nonce":2450584584},{"type":"v1","addr":"172.21.15.101:6805","nonce":2450584584}]},"public_addr":"172.21.15.101:6801/2450584584","cluster_addr":"172.21.15.101:6803/2450584584","heartbeat_back_addr":"172.21.15.101:6807/2450584584","heartbeat_front_addr":"172.21.15.101:6805/2450584584","state":["exists","up"]},{"osd":2,"uuid":"48e6d499-3ca9-424b-b0fb-3ba5f33b9fc9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":945038995},{"type":"v1","addr":"172.21.15.139:6801","nonce":945038995}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":945038995},{"type":"v1","addr":"172.21.15.139:6803","nonce":945038995}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":945038995},{"type":"v1","addr":"172.21.15.139:6807","nonce":945038995}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":945038995},{"type":"v1","addr":"172.21.15.139:6805","nonce":945038995}]},"public_addr":"172.21.15.139:6801/945038995","cluster_addr":"172.21.15.139:6803/945038995","heartbeat_back_addr":"172.21.15.139:6807/945038995","heartbeat_front_addr":"172.21.15.139:6805/945038995","state":["exists","up"]},{"osd":3,"uuid":"5530b265-31a5-447b-a1ea-17ebae9cfddf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6810","nonce":3095249665},{"type":"v1","addr":"172.21.15.98:6811","nonce":3095249665}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6812","nonce":3095249665},{"type":"v1","addr":"172.21.15.98:6813","nonce":3095249665}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6816","nonce":3095249665},{"type":"v1","addr":"172.21.15.98:6817","nonce":3095249665}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6814","nonce":3095249665},{"type":"v1","addr":"172.21.15.98:6815","nonce":3095249665}]},"public_addr":"172.21.15.98:6811/3095249665","cluster_addr":"172.21.15.98:6813/3095249665","heartbeat_back_addr":"172.21.15.98:6817/3095249665","heartbeat_front_addr":"172.21.15.98:6815/3095249665","state":["exists","up"]},{"osd":4,"uuid":"615d4642-a997-4f95-9e38-9fbdaa831400","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":32,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.101:6808","nonce":2844761406},{"type":"v1","addr":"172.21.15.101:6809","nonce":2844761406}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.101:6810","nonce":2844761406},{"type":"v1","addr":"172.21.15.101:6811","nonce":2844761406}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.101:6814","nonce":2844761406},{"type":"v1","addr":"172.21.15.101:6815","nonce":2844761406}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.101:6812","nonce":2844761406},{"type":"v1","addr":"172.21.15.101:6813","nonce":2844761406}]},"public_addr":"172.21.15.101:6809/2844761406","cluster_addr":"172.21.15.101:6811/2844761406","heartbeat_back_addr":"172.21.15.101:6815/2844761406","heartbeat_front_addr":"172.21.15.101:6813/2844761406","state":["exists","up"]},{"osd":5,"uuid":"415de5fa-607a-45a8-b732-28daf3be6dd3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":37,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":3532263042},{"type":"v1","addr":"172.21.15.139:6809","nonce":3532263042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6810","nonce":3532263042},{"type":"v1","addr":"172.21.15.139:6811","nonce":3532263042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6814","nonce":3532263042},{"type":"v1","addr":"172.21.15.139:6815","nonce":3532263042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":3532263042},{"type":"v1","addr":"172.21.15.139:6813","nonce":3532263042}]},"public_addr":"172.21.15.139:6809/3532263042","cluster_addr":"172.21.15.139:6811/3532263042","heartbeat_back_addr":"172.21.15.139:6815/3532263042","heartbeat_front_addr":"172.21.15.139:6813/3532263042","state":["exists","up"]},{"osd":6,"uuid":"d7f9d4ba-c309-4b75-a836-5a36384b528e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":45,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6818","nonce":658948743},{"type":"v1","addr":"172.21.15.98:6819","nonce":658948743}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6820","nonce":658948743},{"type":"v1","addr":"172.21.15.98:6821","nonce":658948743}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6824","nonce":658948743},{"type":"v1","addr":"172.21.15.98:6825","nonce":658948743}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6822","nonce":658948743},{"type":"v1","addr":"172.21.15.98:6823","nonce":658948743}]},"public_addr":"172.21.15.98:6819/658948743","cluster_addr":"172.21.15.98:6821/658948743","heartbeat_back_addr":"172.21.15.98:6825/658948743","heartbeat_front_addr":"172.21.15.98:6823/658948743","state":["exists","up"]},{"osd":7,"uuid":"eedb94d9-c5cd-4a93-8a8c-2648804b7195","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":50,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.101:6816","nonce":2881214066},{"type":"v1","addr":"172.21.15.101:6817","nonce":2881214066}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.101:6818","nonce":2881214066},{"type":"v1","addr":"172.21.15.101:6819","nonce":2881214066}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.101:6822","nonce":2881214066},{"type":"v1","addr":"172.21.15.101:6823","nonce":2881214066}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.101:6820","nonce":2881214066},{"type":"v1","addr":"172.21.15.101:6821","nonce":2881214066}]},"public_addr":"172.21.15.101:6817/2881214066","cluster_addr":"172.21.15.101:6819/2881214066","heartbeat_back_addr":"172.21.15.101:6823/2881214066","heartbeat_front_addr":"172.21.15.101:6821/2881214066","state":["exists","up"]},{"osd":8,"uuid":"6bfd77c2-5770-466e-aa6f-84b50c74ffb6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":57,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":3669514524},{"type":"v1","addr":"172.21.15.139:6817","nonce":3669514524}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":3669514524},{"type":"v1","addr":"172.21.15.139:6819","nonce":3669514524}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":3669514524},{"type":"v1","addr":"172.21.15.139:6823","nonce":3669514524}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6820","nonce":3669514524},{"type":"v1","addr":"172.21.15.139:6821","nonce":3669514524}]},"public_addr":"172.21.15.139:6817/3669514524","cluster_addr":"172.21.15.139:6819/3669514524","heartbeat_back_addr":"172.21.15.139:6823/3669514524","heartbeat_front_addr":"172.21.15.139:6821/3669514524","state":["exists","up"]},{"osd":9,"uuid":"19c5b54a-2242-4b23-b721-63be9559653d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":65,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6826","nonce":2893994080},{"type":"v1","addr":"172.21.15.98:6827","nonce":2893994080}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6828","nonce":2893994080},{"type":"v1","addr":"172.21.15.98:6829","nonce":2893994080}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6832","nonce":2893994080},{"type":"v1","addr":"172.21.15.98:6833","nonce":2893994080}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6830","nonce":2893994080},{"type":"v1","addr":"172.21.15.98:6831","nonce":2893994080}]},"public_addr":"172.21.15.98:6827/2893994080","cluster_addr":"172.21.15.98:6829/2893994080","heartbeat_back_addr":"172.21.15.98:6833/2893994080","heartbeat_front_addr":"172.21.15.98:6831/2893994080","state":["exists","up"]},{"osd":10,"uuid":"326f0209-ba9d-4b59-97cc-d61a758a8f29","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":70,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.101:6824","nonce":3545244225},{"type":"v1","addr":"172.21.15.101:6825","nonce":3545244225}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.101:6826","nonce":3545244225},{"type":"v1","addr":"172.21.15.101:6827","nonce":3545244225}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.101:6830","nonce":3545244225},{"type":"v1","addr":"172.21.15.101:6831","nonce":3545244225}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.101:6828","nonce":3545244225},{"type":"v1","addr":"172.21.15.101:6829","nonce":3545244225}]},"public_addr":"172.21.15.101:6825/3545244225","cluster_addr":"172.21.15.101:6827/3545244225","heartbeat_back_addr":"172.21.15.101:6831/3545244225","heartbeat_front_addr":"172.21.15.101:6829/3545244225","state":["exists","up"]},{"osd":11,"uuid":"c61c0320-1057-4f17-9bf1-7ad306ed7ed3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":78,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":2630023454},{"type":"v1","addr":"172.21.15.139:6825","nonce":2630023454}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6826","nonce":2630023454},{"type":"v1","addr":"172.21.15.139:6827","nonce":2630023454}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":2630023454},{"type":"v1","addr":"172.21.15.139:6831","nonce":2630023454}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":2630023454},{"type":"v1","addr":"172.21.15.139:6829","nonce":2630023454}]},"public_addr":"172.21.15.139:6825/2630023454","cluster_addr":"172.21.15.139:6827/2630023454","heartbeat_back_addr":"172.21.15.139:6831/2630023454","heartbeat_front_addr":"172.21.15.139:6829/2630023454","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T19:24:23.620028+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T19:24:45.779846+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T19:25:06.877949+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T19:25:27.994199+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T19:25:49.461139+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T19:26:10.409282+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T19:26:32.116390+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T19:26:53.536474+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T19:27:14.384839+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T19:27:35.635921+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T19:27:56.858744+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T19:28:18.342519+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.98:0/2164590253":"2023-07-15T19:23:01.520302+0000","172.21.15.98:0/2094734896":"2023-07-15T19:23:01.520302+0000","172.21.15.98:0/784745009":"2023-07-15T19:22:39.481502+0000","172.21.15.98:0/3657190818":"2023-07-15T19:22:39.481502+0000","172.21.15.98:6800/1732769087":"2023-07-15T19:23:01.520302+0000","172.21.15.98:0/3393542419":"2023-07-15T19:22:39.481502+0000","172.21.15.98:6800/749234665":"2023-07-15T19:22:39.481502+0000","172.21.15.98:6801/1732769087":"2023-07-15T19:23:01.520302+0000","172.21.15.98:6801/749234665":"2023-07-15T19:22:39.481502+0000","172.21.15.98:0/187822873":"2023-07-15T19:23:01.520302+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-07-14T19:29:52.583 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:52.582+0000 7f94da676700 1 -- 172.21.15.98:0/209642085 >> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] conn(0x7f94bc0a1220 msgr2=0x7f94bc0a36d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:29:52.583 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:52.582+0000 7f94da676700 1 --2- 172.21.15.98:0/209642085 >> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] conn(0x7f94bc0a1220 0x7f94bc0a36d0 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f94c8009cb0 tx=0x7f94c8009260 comp rx=0 tx=0).stop 2023-07-14T19:29:52.583 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:52.582+0000 7f94da676700 1 -- 172.21.15.98:0/209642085 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f94d40feb20 msgr2=0x7f94d4193780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:29:52.583 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:52.582+0000 7f94da676700 1 --2- 172.21.15.98:0/209642085 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f94d40feb20 0x7f94d4193780 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7f94c4006010 tx=0x7f94c4003dd0 comp rx=0 tx=0).stop 2023-07-14T19:29:52.583 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:52.582+0000 7f94da676700 1 -- 172.21.15.98:0/209642085 shutdown_connections 2023-07-14T19:29:52.584 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:52.582+0000 7f94da676700 1 --2- 172.21.15.98:0/209642085 >> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] conn(0x7f94bc0a1220 0x7f94bc0a36d0 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:29:52.584 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:52.582+0000 7f94da676700 1 --2- 172.21.15.98:0/209642085 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f94d41006a0 0x7f94d419ad30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:29:52.584 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:52.582+0000 7f94da676700 1 --2- 172.21.15.98:0/209642085 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f94d40feb20 0x7f94d4193780 unknown :-1 s=CLOSED pgs=306 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:29:52.584 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:52.582+0000 7f94da676700 1 --2- 172.21.15.98:0/209642085 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f94d40ffd10 0x7f94d4193cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:29:52.584 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:52.582+0000 7f94da676700 1 -- 172.21.15.98:0/209642085 >> 172.21.15.98:0/209642085 conn(0x7f94d40fa100 msgr2=0x7f94d4103240 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:29:52.584 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:52.582+0000 7f94da676700 1 -- 172.21.15.98:0/209642085 shutdown_connections 2023-07-14T19:29:52.584 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:52.582+0000 7f94da676700 1 -- 172.21.15.98:0/209642085 wait complete. 2023-07-14T19:29:52.595 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-07-14T19:25:09.845827+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2023-07-14T19:28:58.952921+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '82', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 2.25, 'score_stable': 2.25, 'optimal_score': 1, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2023-07-14T19:29:00.228272+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '83', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-07-14T19:29:52.596 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-07-14T19:29:52.723 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:52.721+0000 7f864f927700 1 -- 172.21.15.98:0/2675385612 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f86480ffd60 msgr2=0x7f86481001b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:29:52.723 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:52.721+0000 7f864f927700 1 --2- 172.21.15.98:0/2675385612 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f86480ffd60 0x7f86481001b0 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7f863c009990 tx=0x7f863c009ca0 comp rx=0 tx=0).stop 2023-07-14T19:29:52.723 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:52.721+0000 7f864f927700 1 -- 172.21.15.98:0/2675385612 shutdown_connections 2023-07-14T19:29:52.724 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:52.721+0000 7f864f927700 1 --2- 172.21.15.98:0/2675385612 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f86481006f0 0x7f8648106fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:29:52.724 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:52.721+0000 7f864f927700 1 --2- 172.21.15.98:0/2675385612 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f86480feb70 0x7f86480fef80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:29:52.724 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:52.721+0000 7f864f927700 1 --2- 172.21.15.98:0/2675385612 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f86480ffd60 0x7f86481001b0 unknown :-1 s=CLOSED pgs=307 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:29:52.724 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:52.721+0000 7f864f927700 1 -- 172.21.15.98:0/2675385612 >> 172.21.15.98:0/2675385612 conn(0x7f86480fa1d0 msgr2=0x7f86480fc5e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:29:52.724 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:52.721+0000 7f864f927700 1 -- 172.21.15.98:0/2675385612 shutdown_connections 2023-07-14T19:29:52.724 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:52.721+0000 7f864f927700 1 -- 172.21.15.98:0/2675385612 wait complete. 2023-07-14T19:29:52.725 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:52.722+0000 7f864f927700 1 Processor -- start 2023-07-14T19:29:52.725 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:52.722+0000 7f864f927700 1 -- start start 2023-07-14T19:29:52.725 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:52.722+0000 7f864f927700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f86480feb70 0x7f8648193730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:29:52.725 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:52.722+0000 7f864f927700 1 --2- >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f86480ffd60 0x7f8648193c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:29:52.725 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:52.722+0000 7f864f927700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f86481006f0 0x7f864819ace0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:29:52.726 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:52.722+0000 7f864f927700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f864819b2b0 con 0x7f86481006f0 2023-07-14T19:29:52.726 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:52.722+0000 7f864f927700 1 -- --> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f864819b3f0 con 0x7f86480ffd60 2023-07-14T19:29:52.726 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:52.722+0000 7f864f927700 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f864819b580 con 0x7f86480feb70 2023-07-14T19:29:52.726 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:52.722+0000 7f864dec4700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f86481006f0 0x7f864819ace0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:29:52.726 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:52.722+0000 7f864dec4700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f86481006f0 0x7f864819ace0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:46398/0 (socket says 172.21.15.98:46398) 2023-07-14T19:29:52.726 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:52.722+0000 7f864dec4700 1 -- 172.21.15.98:0/3951381758 learned_addr learned my addr 172.21.15.98:0/3951381758 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-07-14T19:29:52.726 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:52.722+0000 7f864dec4700 1 -- 172.21.15.98:0/3951381758 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f86480feb70 msgr2=0x7f8648193730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:29:52.727 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:52.722+0000 7f864cec2700 1 --2- 172.21.15.98:0/3951381758 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f86480ffd60 0x7f8648193c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:29:52.727 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:52.722+0000 7f864d6c3700 1 --2- 172.21.15.98:0/3951381758 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f86480feb70 0x7f8648193730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:29:52.727 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:52.722+0000 7f864dec4700 1 --2- 172.21.15.98:0/3951381758 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f86480feb70 0x7f8648193730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:29:52.727 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:52.722+0000 7f864dec4700 1 -- 172.21.15.98:0/3951381758 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f86480ffd60 msgr2=0x7f8648193c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:29:52.727 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:52.722+0000 7f864dec4700 1 --2- 172.21.15.98:0/3951381758 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f86480ffd60 0x7f8648193c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:29:52.727 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:52.722+0000 7f864dec4700 1 -- 172.21.15.98:0/3951381758 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f863c0095f0 con 0x7f86481006f0 2023-07-14T19:29:52.727 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:52.723+0000 7f864dec4700 1 --2- 172.21.15.98:0/3951381758 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f86481006f0 0x7f864819ace0 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7f864000ada0 tx=0x7f864000c330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:29:52.728 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:52.723+0000 7f863a7fc700 1 -- 172.21.15.98:0/3951381758 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f864000f070 con 0x7f86481006f0 2023-07-14T19:29:52.728 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:52.723+0000 7f864f927700 1 -- 172.21.15.98:0/3951381758 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f864819b860 con 0x7f86481006f0 2023-07-14T19:29:52.728 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:52.723+0000 7f863a7fc700 1 -- 172.21.15.98:0/3951381758 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f8640007b30 con 0x7f86481006f0 2023-07-14T19:29:52.728 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:52.723+0000 7f863a7fc700 1 -- 172.21.15.98:0/3951381758 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8640005440 con 0x7f86481006f0 2023-07-14T19:29:52.728 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:52.723+0000 7f864f927700 1 -- 172.21.15.98:0/3951381758 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f864819bd20 con 0x7f86481006f0 2023-07-14T19:29:52.729 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:52.724+0000 7f864f927700 1 -- 172.21.15.98:0/3951381758 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8648109220 con 0x7f86481006f0 2023-07-14T19:29:52.729 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:52.725+0000 7f863a7fc700 1 -- 172.21.15.98:0/3951381758 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 17) v1 ==== 135766+0+0 (secure 0 0 0) 0x7f8640004ad0 con 0x7f86481006f0 2023-07-14T19:29:52.729 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:52.725+0000 7f863a7fc700 1 --2- 172.21.15.98:0/3951381758 >> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] conn(0x7f86340a33f0 0x7f86340a58a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:29:52.729 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:52.725+0000 7f864d6c3700 1 --2- 172.21.15.98:0/3951381758 >> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] conn(0x7f86340a33f0 0x7f86340a58a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:29:52.730 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:52.726+0000 7f863a7fc700 1 -- 172.21.15.98:0/3951381758 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 7979+0+0 (secure 0 0 0) 0x7f86400cc590 con 0x7f86481006f0 2023-07-14T19:29:52.730 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:52.726+0000 7f864d6c3700 1 --2- 172.21.15.98:0/3951381758 >> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] conn(0x7f86340a33f0 0x7f86340a58a0 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f86480ffbc0 tx=0x7f8644005cf0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:29:52.730 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:52.729+0000 7f863a7fc700 1 -- 172.21.15.98:0/3951381758 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f8640095dc0 con 0x7f86481006f0 2023-07-14T19:29:52.916 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:52.916+0000 7f864f927700 1 -- 172.21.15.98:0/3951381758 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f8648105a50 con 0x7f86481006f0 2023-07-14T19:29:52.917 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:52.916+0000 7f863a7fc700 1 -- 172.21.15.98:0/3951381758 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v83) v1 ==== 93+0+10 (secure 0 0 0) 0x7f86400993e0 con 0x7f86481006f0 2023-07-14T19:29:52.917 INFO:teuthology.orchestra.run.smithi098.stdout:pg_num: 1 2023-07-14T19:29:52.919 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:52.918+0000 7f864f927700 1 -- 172.21.15.98:0/3951381758 >> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] conn(0x7f86340a33f0 msgr2=0x7f86340a58a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:29:52.919 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:52.918+0000 7f864f927700 1 --2- 172.21.15.98:0/3951381758 >> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] conn(0x7f86340a33f0 0x7f86340a58a0 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f86480ffbc0 tx=0x7f8644005cf0 comp rx=0 tx=0).stop 2023-07-14T19:29:52.919 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:52.918+0000 7f864f927700 1 -- 172.21.15.98:0/3951381758 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f86481006f0 msgr2=0x7f864819ace0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:29:52.919 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:52.918+0000 7f864f927700 1 --2- 172.21.15.98:0/3951381758 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f86481006f0 0x7f864819ace0 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7f864000ada0 tx=0x7f864000c330 comp rx=0 tx=0).stop 2023-07-14T19:29:52.919 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:52.918+0000 7f864f927700 1 -- 172.21.15.98:0/3951381758 shutdown_connections 2023-07-14T19:29:52.920 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:52.918+0000 7f864f927700 1 --2- 172.21.15.98:0/3951381758 >> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] conn(0x7f86340a33f0 0x7f86340a58a0 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:29:52.920 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:52.918+0000 7f864f927700 1 --2- 172.21.15.98:0/3951381758 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f86480feb70 0x7f8648193730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:29:52.920 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:52.918+0000 7f864f927700 1 --2- 172.21.15.98:0/3951381758 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f86481006f0 0x7f864819ace0 unknown :-1 s=CLOSED pgs=308 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:29:52.920 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:52.918+0000 7f864f927700 1 --2- 172.21.15.98:0/3951381758 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f86480ffd60 0x7f8648193c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:29:52.920 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:52.918+0000 7f864f927700 1 -- 172.21.15.98:0/3951381758 >> 172.21.15.98:0/3951381758 conn(0x7f86480fa1d0 msgr2=0x7f86480fd400 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:29:52.920 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:52.918+0000 7f864f927700 1 -- 172.21.15.98:0/3951381758 shutdown_connections 2023-07-14T19:29:52.920 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:52.918+0000 7f864f927700 1 -- 172.21.15.98:0/3951381758 wait complete. 2023-07-14T19:29:52.930 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:29:52 smithi098 ceph-mon[127486]: pgmap v278: 97 pgs: 97 active+clean; 634 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2023-07-14T19:29:52.930 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:29:52 smithi098 ceph-mon[127486]: from='client.? 172.21.15.98:0/1946735337' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2023-07-14T19:29:52.930 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:29:52 smithi098 ceph-mon[127486]: from='client.? 172.21.15.98:0/1153644383' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-07-14T19:29:52.930 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:29:52 smithi098 ceph-mon[127486]: from='client.? 172.21.15.98:0/209642085' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-07-14T19:29:52.931 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-07-14T19:29:52.941 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:29:52 smithi101 ceph-mon[128807]: pgmap v278: 97 pgs: 97 active+clean; 634 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2023-07-14T19:29:52.941 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:29:52 smithi101 ceph-mon[128807]: from='client.? 172.21.15.98:0/1946735337' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2023-07-14T19:29:52.941 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:29:52 smithi101 ceph-mon[128807]: from='client.? 172.21.15.98:0/1153644383' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-07-14T19:29:52.941 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:29:52 smithi101 ceph-mon[128807]: from='client.? 172.21.15.98:0/209642085' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-07-14T19:29:52.970 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:29:52 smithi139 ceph-mon[129293]: pgmap v278: 97 pgs: 97 active+clean; 634 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2023-07-14T19:29:52.970 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:29:52 smithi139 ceph-mon[129293]: from='client.? 172.21.15.98:0/1946735337' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2023-07-14T19:29:52.971 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:29:52 smithi139 ceph-mon[129293]: from='client.? 172.21.15.98:0/1153644383' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-07-14T19:29:52.971 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:29:52 smithi139 ceph-mon[129293]: from='client.? 172.21.15.98:0/209642085' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-07-14T19:29:53.057 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:53.055+0000 7f576f01e700 1 -- 172.21.15.98:0/2683331648 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f57680ff020 msgr2=0x7f57680ff430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:29:53.058 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:53.055+0000 7f576f01e700 1 --2- 172.21.15.98:0/2683331648 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f57680ff020 0x7f57680ff430 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7f575c009990 tx=0x7f575c009ca0 comp rx=0 tx=0).stop 2023-07-14T19:29:53.058 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:53.055+0000 7f576f01e700 1 -- 172.21.15.98:0/2683331648 shutdown_connections 2023-07-14T19:29:53.058 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:53.056+0000 7f576f01e700 1 --2- 172.21.15.98:0/2683331648 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f57681006e0 0x7f5768106f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:29:53.058 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:53.056+0000 7f576f01e700 1 --2- 172.21.15.98:0/2683331648 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f57680ff020 0x7f57680ff430 unknown :-1 s=CLOSED pgs=309 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:29:53.058 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:53.056+0000 7f576f01e700 1 --2- 172.21.15.98:0/2683331648 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f57680ffd30 0x7f57681001a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:29:53.058 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:53.056+0000 7f576f01e700 1 -- 172.21.15.98:0/2683331648 >> 172.21.15.98:0/2683331648 conn(0x7f57680f98a0 msgr2=0x7f57680fbcd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:29:53.059 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:53.056+0000 7f576f01e700 1 -- 172.21.15.98:0/2683331648 shutdown_connections 2023-07-14T19:29:53.059 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:53.056+0000 7f576f01e700 1 -- 172.21.15.98:0/2683331648 wait complete. 2023-07-14T19:29:53.059 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:53.056+0000 7f576f01e700 1 Processor -- start 2023-07-14T19:29:53.059 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:53.056+0000 7f576f01e700 1 -- start start 2023-07-14T19:29:53.059 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:53.056+0000 7f576f01e700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f57680ff020 0x7f57681948c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:29:53.059 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:53.056+0000 7f576f01e700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f57680ffd30 0x7f5768194e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:29:53.059 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:53.056+0000 7f576f01e700 1 --2- >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f57681006e0 0x7f576819be70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:29:53.059 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:53.056+0000 7f576f01e700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f576819c440 con 0x7f57680ff020 2023-07-14T19:29:53.060 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:53.056+0000 7f576f01e700 1 -- --> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f576819c580 con 0x7f57681006e0 2023-07-14T19:29:53.060 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:53.056+0000 7f576f01e700 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f576819c710 con 0x7f57680ffd30 2023-07-14T19:29:53.060 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:53.056+0000 7f576cdba700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f57680ff020 0x7f57681948c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:29:53.060 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:53.056+0000 7f576cdba700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f57680ff020 0x7f57681948c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:46408/0 (socket says 172.21.15.98:46408) 2023-07-14T19:29:53.060 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:53.056+0000 7f576cdba700 1 -- 172.21.15.98:0/1953217030 learned_addr learned my addr 172.21.15.98:0/1953217030 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-07-14T19:29:53.060 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:53.056+0000 7f5767fff700 1 --2- 172.21.15.98:0/1953217030 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f57680ffd30 0x7f5768194e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:29:53.060 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:53.056+0000 7f576d5bb700 1 --2- 172.21.15.98:0/1953217030 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f57681006e0 0x7f576819be70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:29:53.061 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:53.057+0000 7f576cdba700 1 -- 172.21.15.98:0/1953217030 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f57680ffd30 msgr2=0x7f5768194e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:29:53.061 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:53.057+0000 7f576cdba700 1 --2- 172.21.15.98:0/1953217030 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f57680ffd30 0x7f5768194e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:29:53.061 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:53.057+0000 7f576cdba700 1 -- 172.21.15.98:0/1953217030 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f57681006e0 msgr2=0x7f576819be70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:29:53.061 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:53.057+0000 7f576cdba700 1 --2- 172.21.15.98:0/1953217030 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f57681006e0 0x7f576819be70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:29:53.061 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:53.057+0000 7f576cdba700 1 -- 172.21.15.98:0/1953217030 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f575c0095f0 con 0x7f57680ff020 2023-07-14T19:29:53.061 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:53.057+0000 7f576cdba700 1 --2- 172.21.15.98:0/1953217030 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f57680ff020 0x7f57681948c0 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7f575c0095c0 tx=0x7f575c00f690 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:29:53.061 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:53.057+0000 7f5765ffb700 1 -- 172.21.15.98:0/1953217030 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f575c01d070 con 0x7f57680ff020 2023-07-14T19:29:53.062 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:53.057+0000 7f576f01e700 1 -- 172.21.15.98:0/1953217030 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f576819c990 con 0x7f57680ff020 2023-07-14T19:29:53.062 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:53.057+0000 7f5765ffb700 1 -- 172.21.15.98:0/1953217030 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f575c005030 con 0x7f57680ff020 2023-07-14T19:29:53.062 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:53.057+0000 7f5765ffb700 1 -- 172.21.15.98:0/1953217030 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f575c004af0 con 0x7f57680ff020 2023-07-14T19:29:53.062 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:53.057+0000 7f576f01e700 1 -- 172.21.15.98:0/1953217030 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f576819ce10 con 0x7f57680ff020 2023-07-14T19:29:53.063 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:53.058+0000 7f576f01e700 1 -- 172.21.15.98:0/1953217030 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f576810a330 con 0x7f57680ff020 2023-07-14T19:29:53.063 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:53.059+0000 7f5765ffb700 1 -- 172.21.15.98:0/1953217030 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 17) v1 ==== 135766+0+0 (secure 0 0 0) 0x7f575c004c90 con 0x7f57680ff020 2023-07-14T19:29:53.063 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:53.060+0000 7f5765ffb700 1 --2- 172.21.15.98:0/1953217030 >> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] conn(0x7f57580a11e0 0x7f57580a3690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:29:53.063 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:53.060+0000 7f5767fff700 1 --2- 172.21.15.98:0/1953217030 >> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] conn(0x7f57580a11e0 0x7f57580a3690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:29:53.063 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:53.060+0000 7f5765ffb700 1 -- 172.21.15.98:0/1953217030 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 7979+0+0 (secure 0 0 0) 0x7f575c0cdc90 con 0x7f57680ff020 2023-07-14T19:29:53.063 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:53.060+0000 7f5767fff700 1 --2- 172.21.15.98:0/1953217030 >> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] conn(0x7f57580a11e0 0x7f57580a3690 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f57680fed40 tx=0x7f575400b410 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:29:53.064 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:53.063+0000 7f5765ffb700 1 -- 172.21.15.98:0/1953217030 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f575c0974c0 con 0x7f57680ff020 2023-07-14T19:29:53.250 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:53.250+0000 7f576f01e700 1 -- 172.21.15.98:0/1953217030 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f5768105a90 con 0x7f57680ff020 2023-07-14T19:29:53.251 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:53.250+0000 7f5765ffb700 1 -- 172.21.15.98:0/1953217030 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v83) v1 ==== 104+0+11 (secure 0 0 0) 0x7f575c09aae0 con 0x7f57680ff020 2023-07-14T19:29:53.251 INFO:teuthology.orchestra.run.smithi098.stdout:pg_num: 32 2023-07-14T19:29:53.254 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:53.253+0000 7f576f01e700 1 -- 172.21.15.98:0/1953217030 >> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] conn(0x7f57580a11e0 msgr2=0x7f57580a3690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:29:53.254 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:53.253+0000 7f576f01e700 1 --2- 172.21.15.98:0/1953217030 >> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] conn(0x7f57580a11e0 0x7f57580a3690 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f57680fed40 tx=0x7f575400b410 comp rx=0 tx=0).stop 2023-07-14T19:29:53.254 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:53.253+0000 7f576f01e700 1 -- 172.21.15.98:0/1953217030 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f57680ff020 msgr2=0x7f57681948c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:29:53.254 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:53.253+0000 7f576f01e700 1 --2- 172.21.15.98:0/1953217030 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f57680ff020 0x7f57681948c0 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7f575c0095c0 tx=0x7f575c00f690 comp rx=0 tx=0).stop 2023-07-14T19:29:53.254 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:53.253+0000 7f576f01e700 1 -- 172.21.15.98:0/1953217030 shutdown_connections 2023-07-14T19:29:53.255 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:53.253+0000 7f576f01e700 1 --2- 172.21.15.98:0/1953217030 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f57681006e0 0x7f576819be70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:29:53.255 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:53.253+0000 7f576f01e700 1 --2- 172.21.15.98:0/1953217030 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f57680ff020 0x7f57681948c0 unknown :-1 s=CLOSED pgs=310 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:29:53.255 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:53.253+0000 7f576f01e700 1 --2- 172.21.15.98:0/1953217030 >> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] conn(0x7f57580a11e0 0x7f57580a3690 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:29:53.255 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:53.253+0000 7f576f01e700 1 --2- 172.21.15.98:0/1953217030 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f57680ffd30 0x7f5768194e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:29:53.255 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:53.253+0000 7f576f01e700 1 -- 172.21.15.98:0/1953217030 >> 172.21.15.98:0/1953217030 conn(0x7f57680f98a0 msgr2=0x7f57680fbb40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:29:53.255 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:53.253+0000 7f576f01e700 1 -- 172.21.15.98:0/1953217030 shutdown_connections 2023-07-14T19:29:53.255 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:53.253+0000 7f576f01e700 1 -- 172.21.15.98:0/1953217030 wait complete. 2023-07-14T19:29:53.266 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-07-14T19:29:53.392 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:53.389+0000 7fc73e33f700 1 -- 172.21.15.98:0/3065655503 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc738100110 msgr2=0x7fc738100520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:29:53.393 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:53.389+0000 7fc73e33f700 1 --2- 172.21.15.98:0/3065655503 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc738100110 0x7fc738100520 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7fc72c009990 tx=0x7fc72c009ca0 comp rx=0 tx=0).stop 2023-07-14T19:29:53.393 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:53.389+0000 7fc73e33f700 1 -- 172.21.15.98:0/3065655503 shutdown_connections 2023-07-14T19:29:53.393 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:53.389+0000 7fc73e33f700 1 --2- 172.21.15.98:0/3065655503 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fc7380ff710 0x7fc738106fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:29:53.393 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:53.389+0000 7fc73e33f700 1 --2- 172.21.15.98:0/3065655503 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc738100110 0x7fc738100520 unknown :-1 s=CLOSED pgs=311 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:29:53.394 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:53.389+0000 7fc73e33f700 1 --2- 172.21.15.98:0/3065655503 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7fc7380fed60 0x7fc7380ff1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:29:53.394 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:53.389+0000 7fc73e33f700 1 -- 172.21.15.98:0/3065655503 >> 172.21.15.98:0/3065655503 conn(0x7fc7380f98a0 msgr2=0x7fc7380fbcd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:29:53.394 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:53.389+0000 7fc73e33f700 1 -- 172.21.15.98:0/3065655503 shutdown_connections 2023-07-14T19:29:53.394 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:53.389+0000 7fc73e33f700 1 -- 172.21.15.98:0/3065655503 wait complete. 2023-07-14T19:29:53.395 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:53.389+0000 7fc73e33f700 1 Processor -- start 2023-07-14T19:29:53.395 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:53.389+0000 7fc73e33f700 1 -- start start 2023-07-14T19:29:53.396 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:53.390+0000 7fc73e33f700 1 --2- >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7fc7380fed60 0x7fc738102b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:29:53.396 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:53.390+0000 7fc73e33f700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fc7380ff710 0x7fc738101150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:29:53.396 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:53.390+0000 7fc73e33f700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc738100110 0x7fc738101690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:29:53.396 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:53.390+0000 7fc73e33f700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc738101e30 con 0x7fc738100110 2023-07-14T19:29:53.397 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:53.390+0000 7fc73e33f700 1 -- --> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc738101fa0 con 0x7fc7380fed60 2023-07-14T19:29:53.397 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:53.390+0000 7fc73e33f700 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc73819fa20 con 0x7fc7380ff710 2023-07-14T19:29:53.397 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:53.390+0000 7fc73c8dc700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc738100110 0x7fc738101690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:29:53.397 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:53.390+0000 7fc737fff700 1 --2- >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7fc7380fed60 0x7fc738102b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:29:53.398 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:53.390+0000 7fc73c8dc700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc738100110 0x7fc738101690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:46426/0 (socket says 172.21.15.98:46426) 2023-07-14T19:29:53.398 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:53.390+0000 7fc73c8dc700 1 -- 172.21.15.98:0/2856764090 learned_addr learned my addr 172.21.15.98:0/2856764090 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-07-14T19:29:53.398 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:53.390+0000 7fc7377fe700 1 --2- 172.21.15.98:0/2856764090 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fc7380ff710 0x7fc738101150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:29:53.399 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:53.390+0000 7fc73c8dc700 1 -- 172.21.15.98:0/2856764090 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fc7380ff710 msgr2=0x7fc738101150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:29:53.399 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:53.390+0000 7fc73c8dc700 1 --2- 172.21.15.98:0/2856764090 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fc7380ff710 0x7fc738101150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:29:53.400 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:53.390+0000 7fc73c8dc700 1 -- 172.21.15.98:0/2856764090 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7fc7380fed60 msgr2=0x7fc738102b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:29:53.400 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:53.390+0000 7fc73c8dc700 1 --2- 172.21.15.98:0/2856764090 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7fc7380fed60 0x7fc738102b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:29:53.400 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:53.390+0000 7fc73c8dc700 1 -- 172.21.15.98:0/2856764090 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc72c0095f0 con 0x7fc738100110 2023-07-14T19:29:53.401 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:53.390+0000 7fc73c8dc700 1 --2- 172.21.15.98:0/2856764090 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc738100110 0x7fc738101690 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7fc73000ef30 tx=0x7fc73000c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:29:53.401 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:53.390+0000 7fc7357fa700 1 -- 172.21.15.98:0/2856764090 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc730018070 con 0x7fc738100110 2023-07-14T19:29:53.401 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:53.390+0000 7fc73e33f700 1 -- 172.21.15.98:0/2856764090 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc73819fca0 con 0x7fc738100110 2023-07-14T19:29:53.401 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:53.390+0000 7fc7357fa700 1 -- 172.21.15.98:0/2856764090 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fc73000f040 con 0x7fc738100110 2023-07-14T19:29:53.402 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:53.390+0000 7fc7357fa700 1 -- 172.21.15.98:0/2856764090 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc73000cd00 con 0x7fc738100110 2023-07-14T19:29:53.402 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:53.390+0000 7fc73e33f700 1 -- 172.21.15.98:0/2856764090 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc7381a01b0 con 0x7fc738100110 2023-07-14T19:29:53.402 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:53.391+0000 7fc73e33f700 1 -- 172.21.15.98:0/2856764090 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc73810a320 con 0x7fc738100110 2023-07-14T19:29:53.403 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:53.392+0000 7fc7357fa700 1 -- 172.21.15.98:0/2856764090 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 17) v1 ==== 135766+0+0 (secure 0 0 0) 0x7fc730003a00 con 0x7fc738100110 2023-07-14T19:29:53.404 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:53.393+0000 7fc7357fa700 1 --2- 172.21.15.98:0/2856764090 >> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] conn(0x7fc7200a11e0 0x7fc7200a3690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:29:53.404 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:53.393+0000 7fc737fff700 1 --2- 172.21.15.98:0/2856764090 >> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] conn(0x7fc7200a11e0 0x7fc7200a3690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:29:53.404 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:53.393+0000 7fc7357fa700 1 -- 172.21.15.98:0/2856764090 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 7979+0+0 (secure 0 0 0) 0x7fc7300cc3f0 con 0x7fc738100110 2023-07-14T19:29:53.405 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:53.393+0000 7fc737fff700 1 --2- 172.21.15.98:0/2856764090 >> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] conn(0x7fc7200a11e0 0x7fc7200a3690 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7fc72c00b2e0 tx=0x7fc72c005fb0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:29:53.405 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:53.397+0000 7fc7357fa700 1 -- 172.21.15.98:0/2856764090 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7fc730095c20 con 0x7fc738100110 2023-07-14T19:29:53.586 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:53.586+0000 7fc73e33f700 1 -- 172.21.15.98:0/2856764090 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7fc7381a0760 con 0x7fc738100110 2023-07-14T19:29:53.587 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:53.586+0000 7fc7357fa700 1 -- 172.21.15.98:0/2856764090 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v83) v1 ==== 100+0+11 (secure 0 0 0) 0x7fc730099240 con 0x7fc738100110 2023-07-14T19:29:53.587 INFO:teuthology.orchestra.run.smithi098.stdout:pg_num: 64 2023-07-14T19:29:53.589 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:53.588+0000 7fc73e33f700 1 -- 172.21.15.98:0/2856764090 >> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] conn(0x7fc7200a11e0 msgr2=0x7fc7200a3690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:29:53.589 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:53.588+0000 7fc73e33f700 1 --2- 172.21.15.98:0/2856764090 >> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] conn(0x7fc7200a11e0 0x7fc7200a3690 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7fc72c00b2e0 tx=0x7fc72c005fb0 comp rx=0 tx=0).stop 2023-07-14T19:29:53.590 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:53.588+0000 7fc73e33f700 1 -- 172.21.15.98:0/2856764090 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc738100110 msgr2=0x7fc738101690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:29:53.590 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:53.588+0000 7fc73e33f700 1 --2- 172.21.15.98:0/2856764090 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc738100110 0x7fc738101690 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7fc73000ef30 tx=0x7fc73000c3a0 comp rx=0 tx=0).stop 2023-07-14T19:29:53.590 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:53.589+0000 7fc73e33f700 1 -- 172.21.15.98:0/2856764090 shutdown_connections 2023-07-14T19:29:53.590 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:53.589+0000 7fc73e33f700 1 --2- 172.21.15.98:0/2856764090 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc738100110 0x7fc738101690 unknown :-1 s=CLOSED pgs=312 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:29:53.591 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:53.589+0000 7fc73e33f700 1 --2- 172.21.15.98:0/2856764090 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7fc7380fed60 0x7fc738102b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:29:53.591 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:53.589+0000 7fc73e33f700 1 --2- 172.21.15.98:0/2856764090 >> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] conn(0x7fc7200a11e0 0x7fc7200a3690 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:29:53.591 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:53.589+0000 7fc73e33f700 1 --2- 172.21.15.98:0/2856764090 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fc7380ff710 0x7fc738101150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:29:53.591 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:53.589+0000 7fc73e33f700 1 -- 172.21.15.98:0/2856764090 >> 172.21.15.98:0/2856764090 conn(0x7fc7380f98a0 msgr2=0x7fc7380fdc50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:29:53.591 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:53.589+0000 7fc73e33f700 1 -- 172.21.15.98:0/2856764090 shutdown_connections 2023-07-14T19:29:53.591 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:53.589+0000 7fc73e33f700 1 -- 172.21.15.98:0/2856764090 wait complete. 2023-07-14T19:29:53.601 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-07-14T19:29:53.726 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:53.724+0000 7f0143a79700 1 -- 172.21.15.98:0/3485179055 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f013c0ffd10 msgr2=0x7f013c100160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:29:53.727 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:53.724+0000 7f0143a79700 1 --2- 172.21.15.98:0/3485179055 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f013c0ffd10 0x7f013c100160 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7f0138009990 tx=0x7f0138009ca0 comp rx=0 tx=0).stop 2023-07-14T19:29:53.727 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:53.724+0000 7f0143a79700 1 -- 172.21.15.98:0/3485179055 shutdown_connections 2023-07-14T19:29:53.727 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:53.724+0000 7f0143a79700 1 --2- 172.21.15.98:0/3485179055 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f013c1006a0 0x7f013c106f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:29:53.727 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:53.724+0000 7f0143a79700 1 --2- 172.21.15.98:0/3485179055 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f013c0feb20 0x7f013c0fef30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:29:53.727 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:53.724+0000 7f0143a79700 1 --2- 172.21.15.98:0/3485179055 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f013c0ffd10 0x7f013c100160 unknown :-1 s=CLOSED pgs=313 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:29:53.727 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:53.724+0000 7f0143a79700 1 -- 172.21.15.98:0/3485179055 >> 172.21.15.98:0/3485179055 conn(0x7f013c0fa100 msgr2=0x7f013c0fc570 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:29:53.727 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:53.724+0000 7f0143a79700 1 -- 172.21.15.98:0/3485179055 shutdown_connections 2023-07-14T19:29:53.728 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:53.724+0000 7f0143a79700 1 -- 172.21.15.98:0/3485179055 wait complete. 2023-07-14T19:29:53.728 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:53.725+0000 7f0143a79700 1 Processor -- start 2023-07-14T19:29:53.728 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:53.725+0000 7f0143a79700 1 -- start start 2023-07-14T19:29:53.728 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:53.725+0000 7f0143a79700 1 --2- >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f013c0feb20 0x7f013c195860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:29:53.728 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:53.725+0000 7f0143a79700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f013c0ffd10 0x7f013c195da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:29:53.728 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:53.725+0000 7f0143a79700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f013c1006a0 0x7f013c19ce10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:29:53.728 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:53.725+0000 7f0143a79700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f013c19d3e0 con 0x7f013c0ffd10 2023-07-14T19:29:53.729 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:53.725+0000 7f0143a79700 1 -- --> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f013c19d520 con 0x7f013c0feb20 2023-07-14T19:29:53.729 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:53.725+0000 7f0143a79700 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f013c19d6b0 con 0x7f013c1006a0 2023-07-14T19:29:53.729 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:53.725+0000 7f0141014700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f013c0ffd10 0x7f013c195da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:29:53.729 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:53.725+0000 7f0141014700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f013c0ffd10 0x7f013c195da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:46438/0 (socket says 172.21.15.98:46438) 2023-07-14T19:29:53.729 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:53.725+0000 7f0141014700 1 -- 172.21.15.98:0/3051150914 learned_addr learned my addr 172.21.15.98:0/3051150914 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-07-14T19:29:53.729 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:53.725+0000 7f0141815700 1 --2- 172.21.15.98:0/3051150914 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f013c0feb20 0x7f013c195860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:29:53.729 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:53.725+0000 7f0141014700 1 -- 172.21.15.98:0/3051150914 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f013c1006a0 msgr2=0x7f013c19ce10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:29:53.730 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:53.725+0000 7f0141014700 1 --2- 172.21.15.98:0/3051150914 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f013c1006a0 0x7f013c19ce10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:29:53.730 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:53.725+0000 7f0141014700 1 -- 172.21.15.98:0/3051150914 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f013c0feb20 msgr2=0x7f013c195860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:29:53.730 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:53.725+0000 7f0141014700 1 --2- 172.21.15.98:0/3051150914 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f013c0feb20 0x7f013c195860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:29:53.730 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:53.725+0000 7f0141014700 1 -- 172.21.15.98:0/3051150914 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f01380095f0 con 0x7f013c0ffd10 2023-07-14T19:29:53.730 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:53.725+0000 7f0141014700 1 --2- 172.21.15.98:0/3051150914 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f013c0ffd10 0x7f013c195da0 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7f0138004760 tx=0x7f0138004790 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:29:53.730 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:53.726+0000 7f012effd700 1 -- 172.21.15.98:0/3051150914 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f013801d070 con 0x7f013c0ffd10 2023-07-14T19:29:53.730 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:53.726+0000 7f012effd700 1 -- 172.21.15.98:0/3051150914 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f01380041d0 con 0x7f013c0ffd10 2023-07-14T19:29:53.730 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:53.726+0000 7f012effd700 1 -- 172.21.15.98:0/3051150914 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f01380053a0 con 0x7f013c0ffd10 2023-07-14T19:29:53.731 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:53.726+0000 7f0143a79700 1 -- 172.21.15.98:0/3051150914 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f013c19d930 con 0x7f013c0ffd10 2023-07-14T19:29:53.731 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:53.726+0000 7f0143a79700 1 -- 172.21.15.98:0/3051150914 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f013c19dd30 con 0x7f013c0ffd10 2023-07-14T19:29:53.732 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:53.726+0000 7f0143a79700 1 -- 172.21.15.98:0/3051150914 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f013c105960 con 0x7f013c0ffd10 2023-07-14T19:29:53.732 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:53.728+0000 7f012effd700 1 -- 172.21.15.98:0/3051150914 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 17) v1 ==== 135766+0+0 (secure 0 0 0) 0x7f0138005500 con 0x7f013c0ffd10 2023-07-14T19:29:53.732 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:53.728+0000 7f012effd700 1 --2- 172.21.15.98:0/3051150914 >> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] conn(0x7f01280a1220 0x7f01280a36d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:29:53.732 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:53.728+0000 7f0141815700 1 --2- 172.21.15.98:0/3051150914 >> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] conn(0x7f01280a1220 0x7f01280a36d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:29:53.732 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:53.728+0000 7f012effd700 1 -- 172.21.15.98:0/3051150914 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 7979+0+0 (secure 0 0 0) 0x7f01380ce070 con 0x7f013c0ffd10 2023-07-14T19:29:53.732 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:53.729+0000 7f0141815700 1 --2- 172.21.15.98:0/3051150914 >> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] conn(0x7f01280a1220 0x7f01280a36d0 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f013c0ffb70 tx=0x7f013000a3b0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:29:53.732 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:53.731+0000 7f012effd700 1 -- 172.21.15.98:0/3051150914 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f01380978a0 con 0x7f013c0ffd10 2023-07-14T19:29:53.941 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:29:53 smithi101 ceph-mon[128807]: from='client.? 172.21.15.98:0/3951381758' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2023-07-14T19:29:53.941 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:29:53 smithi101 ceph-mon[128807]: from='client.? 172.21.15.98:0/1953217030' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2023-07-14T19:29:53.942 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:29:53 smithi101 ceph-mon[128807]: from='client.? 172.21.15.98:0/2856764090' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2023-07-14T19:29:53.965 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:53.964+0000 7f0143a79700 1 -- 172.21.15.98:0/3051150914 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f013c02cd00 con 0x7f013c0ffd10 2023-07-14T19:29:53.966 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:53.965+0000 7f012effd700 1 -- 172.21.15.98:0/3051150914 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 22 v22) v1 ==== 94+0+10826 (secure 0 0 0) 0x7f0138097030 con 0x7f013c0ffd10 2023-07-14T19:29:53.966 INFO:teuthology.orchestra.run.smithi098.stdout: 2023-07-14T19:29:53.966 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":22,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14565,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6835/586280675","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6834","nonce":586280675},{"type":"v1","addr":"172.21.15.98:6835","nonce":586280675}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":4},{"gid":14577,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6837/487577257","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6836","nonce":487577257},{"type":"v1","addr":"172.21.15.98:6837","nonce":487577257}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":5},{"gid":14634,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6839/4085731209","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6838","nonce":4085731209},{"type":"v1","addr":"172.21.15.98:6839","nonce":4085731209}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":11},{"gid":14646,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6841/1679318259","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6840","nonce":1679318259},{"type":"v1","addr":"172.21.15.98:6841","nonce":1679318259}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":12},{"gid":24421,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.139:6833/3505477820","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6832","nonce":3505477820},{"type":"v1","addr":"172.21.15.139:6833","nonce":3505477820}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":24437,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.101:6833/3474460171","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.101:6832","nonce":3474460171},{"type":"v1","addr":"172.21.15.101:6833","nonce":3474460171}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":22,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2023-07-14T19:29:01.033156+0000","modified":"2023-07-14T19:29:06.291916+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24521,"mds_1":24493,"mds_2":24479},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24455":{"gid":24455,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.101:6835/1583264648","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.101:6834","nonce":1583264648},{"type":"v1","addr":"172.21.15.101:6835","nonce":1583264648}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24460":{"gid":24460,"name":"f","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.139:6835/2315281724","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6834","nonce":2315281724},{"type":"v1","addr":"172.21.15.139:6835","nonce":2315281724}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24469":{"gid":24469,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.139:6837/3597660552","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6836","nonce":3597660552},{"type":"v1","addr":"172.21.15.139:6837","nonce":3597660552}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24479":{"gid":24479,"name":"h","rank":2,"incarnation":21,"state":"up:active","state_seq":6,"addr":"172.21.15.101:6837/485741528","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.101:6836","nonce":485741528},{"type":"v1","addr":"172.21.15.101:6837","nonce":485741528}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24493":{"gid":24493,"name":"i","rank":1,"incarnation":19,"state":"up:active","state_seq":5,"addr":"172.21.15.139:6839/1258854143","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6838","nonce":1258854143},{"type":"v1","addr":"172.21.15.139:6839","nonce":1258854143}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24521":{"gid":24521,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.101:6839/48393381","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.101:6838","nonce":48393381},{"type":"v1","addr":"172.21.15.101:6839","nonce":48393381}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-07-14T19:29:53.968 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:53.967+0000 7f0143a79700 1 -- 172.21.15.98:0/3051150914 >> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] conn(0x7f01280a1220 msgr2=0x7f01280a36d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:29:53.968 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:53.967+0000 7f0143a79700 1 --2- 172.21.15.98:0/3051150914 >> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] conn(0x7f01280a1220 0x7f01280a36d0 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f013c0ffb70 tx=0x7f013000a3b0 comp rx=0 tx=0).stop 2023-07-14T19:29:53.968 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:53.967+0000 7f0143a79700 1 -- 172.21.15.98:0/3051150914 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f013c0ffd10 msgr2=0x7f013c195da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:29:53.968 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:53.967+0000 7f0143a79700 1 --2- 172.21.15.98:0/3051150914 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f013c0ffd10 0x7f013c195da0 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7f0138004760 tx=0x7f0138004790 comp rx=0 tx=0).stop 2023-07-14T19:29:53.968 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:53.967+0000 7f0143a79700 1 -- 172.21.15.98:0/3051150914 shutdown_connections 2023-07-14T19:29:53.968 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:53.967+0000 7f0143a79700 1 --2- 172.21.15.98:0/3051150914 >> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] conn(0x7f01280a1220 0x7f01280a36d0 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:29:53.969 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:53.967+0000 7f0143a79700 1 --2- 172.21.15.98:0/3051150914 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f013c1006a0 0x7f013c19ce10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:29:53.969 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:53.967+0000 7f0143a79700 1 --2- 172.21.15.98:0/3051150914 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f013c0feb20 0x7f013c195860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:29:53.969 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:53.967+0000 7f0143a79700 1 --2- 172.21.15.98:0/3051150914 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f013c0ffd10 0x7f013c195da0 unknown :-1 s=CLOSED pgs=314 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:29:53.969 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:53.967+0000 7f0143a79700 1 -- 172.21.15.98:0/3051150914 >> 172.21.15.98:0/3051150914 conn(0x7f013c0fa100 msgr2=0x7f013c0fc2f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:29:53.969 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:53.967+0000 7f0143a79700 1 -- 172.21.15.98:0/3051150914 shutdown_connections 2023-07-14T19:29:53.969 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:53.967+0000 7f0143a79700 1 -- 172.21.15.98:0/3051150914 wait complete. 2023-07-14T19:29:53.970 INFO:teuthology.orchestra.run.smithi098.stderr:dumped fsmap epoch 22 2023-07-14T19:29:53.970 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:29:53 smithi139 ceph-mon[129293]: from='client.? 172.21.15.98:0/3951381758' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2023-07-14T19:29:53.970 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:29:53 smithi139 ceph-mon[129293]: from='client.? 172.21.15.98:0/1953217030' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2023-07-14T19:29:53.970 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:29:53 smithi139 ceph-mon[129293]: from='client.? 172.21.15.98:0/2856764090' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2023-07-14T19:29:53.979 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:29:53 smithi098 ceph-mon[127486]: from='client.? 172.21.15.98:0/3951381758' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2023-07-14T19:29:53.979 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:29:53 smithi098 ceph-mon[127486]: from='client.? 172.21.15.98:0/1953217030' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2023-07-14T19:29:53.979 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:29:53 smithi098 ceph-mon[127486]: from='client.? 172.21.15.98:0/2856764090' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2023-07-14T19:29:53.980 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-07-14T19:29:54.105 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:54.100+0000 7f20ff0e9700 1 -- 172.21.15.98:0/1358073527 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f20f8103000 msgr2=0x7f20f81053e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:29:54.106 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:54.100+0000 7f20ff0e9700 1 --2- 172.21.15.98:0/1358073527 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f20f8103000 0x7f20f81053e0 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7f20e4009990 tx=0x7f20e4009ca0 comp rx=0 tx=0).stop 2023-07-14T19:29:54.106 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:54.100+0000 7f20ff0e9700 1 -- 172.21.15.98:0/1358073527 shutdown_connections 2023-07-14T19:29:54.106 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:54.100+0000 7f20ff0e9700 1 --2- 172.21.15.98:0/1358073527 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f20f81006e0 0x7f20f8102ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:29:54.106 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:54.100+0000 7f20ff0e9700 1 --2- 172.21.15.98:0/1358073527 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f20f8105920 0x7f20f8107d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:29:54.107 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:54.100+0000 7f20ff0e9700 1 --2- 172.21.15.98:0/1358073527 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f20f8103000 0x7f20f81053e0 unknown :-1 s=CLOSED pgs=315 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:29:54.107 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:54.100+0000 7f20ff0e9700 1 -- 172.21.15.98:0/1358073527 >> 172.21.15.98:0/1358073527 conn(0x7f20f80fa100 msgr2=0x7f20f80fc550 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:29:54.107 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:54.100+0000 7f20ff0e9700 1 -- 172.21.15.98:0/1358073527 shutdown_connections 2023-07-14T19:29:54.108 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:54.100+0000 7f20ff0e9700 1 -- 172.21.15.98:0/1358073527 wait complete. 2023-07-14T19:29:54.108 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:54.101+0000 7f20ff0e9700 1 Processor -- start 2023-07-14T19:29:54.108 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:54.101+0000 7f20ff0e9700 1 -- start start 2023-07-14T19:29:54.109 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:54.101+0000 7f20ff0e9700 1 --2- >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f20f81006e0 0x7f20f8100000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:29:54.109 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:54.101+0000 7f20ff0e9700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f20f8103000 0x7f20f80fe690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:29:54.109 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:54.101+0000 7f20ff0e9700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f20f8105920 0x7f20f80febd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:29:54.109 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:54.101+0000 7f20ff0e9700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f20f80ff370 con 0x7f20f8105920 2023-07-14T19:29:54.110 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:54.101+0000 7f20ff0e9700 1 -- --> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f20f80ff4e0 con 0x7f20f81006e0 2023-07-14T19:29:54.110 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:54.101+0000 7f20ff0e9700 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f20f819fa80 con 0x7f20f8103000 2023-07-14T19:29:54.110 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:54.101+0000 7f20fd686700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f20f8105920 0x7f20f80febd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:29:54.110 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:54.101+0000 7f20fd686700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f20f8105920 0x7f20f80febd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:46454/0 (socket says 172.21.15.98:46454) 2023-07-14T19:29:54.110 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:54.101+0000 7f20fd686700 1 -- 172.21.15.98:0/2909637721 learned_addr learned my addr 172.21.15.98:0/2909637721 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-07-14T19:29:54.111 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:54.101+0000 7f20f7fff700 1 --2- 172.21.15.98:0/2909637721 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f20f8103000 0x7f20f80fe690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:29:54.111 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:54.101+0000 7f20fce85700 1 --2- 172.21.15.98:0/2909637721 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f20f81006e0 0x7f20f8100000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:29:54.111 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:54.101+0000 7f20fd686700 1 -- 172.21.15.98:0/2909637721 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f20f8103000 msgr2=0x7f20f80fe690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:29:54.111 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:54.101+0000 7f20fd686700 1 --2- 172.21.15.98:0/2909637721 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f20f8103000 0x7f20f80fe690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:29:54.111 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:54.101+0000 7f20fd686700 1 -- 172.21.15.98:0/2909637721 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f20f81006e0 msgr2=0x7f20f8100000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:29:54.111 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:54.101+0000 7f20fd686700 1 --2- 172.21.15.98:0/2909637721 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f20f81006e0 0x7f20f8100000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:29:54.111 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:54.101+0000 7f20fd686700 1 -- 172.21.15.98:0/2909637721 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f20e40095f0 con 0x7f20f8105920 2023-07-14T19:29:54.112 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:54.101+0000 7f20fce85700 1 --2- 172.21.15.98:0/2909637721 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f20f81006e0 0x7f20f8100000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-07-14T19:29:54.112 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:54.102+0000 7f20fd686700 1 --2- 172.21.15.98:0/2909637721 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f20f8105920 0x7f20f80febd0 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7f20f000b970 tx=0x7f20f000bd30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:29:54.112 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:54.102+0000 7f20f5ffb700 1 -- 172.21.15.98:0/2909637721 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f20f000c950 con 0x7f20f8105920 2023-07-14T19:29:54.112 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:54.102+0000 7f20ff0e9700 1 -- 172.21.15.98:0/2909637721 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f20f819fd10 con 0x7f20f8105920 2023-07-14T19:29:54.112 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:54.102+0000 7f20f5ffb700 1 -- 172.21.15.98:0/2909637721 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f20f0014440 con 0x7f20f8105920 2023-07-14T19:29:54.112 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:54.102+0000 7f20f5ffb700 1 -- 172.21.15.98:0/2909637721 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f20f00075d0 con 0x7f20f8105920 2023-07-14T19:29:54.112 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:54.102+0000 7f20ff0e9700 1 -- 172.21.15.98:0/2909637721 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f20f81a01d0 con 0x7f20f8105920 2023-07-14T19:29:54.113 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:54.103+0000 7f20ff0e9700 1 -- 172.21.15.98:0/2909637721 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f20f80fbc70 con 0x7f20f8105920 2023-07-14T19:29:54.113 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:54.104+0000 7f20f5ffb700 1 -- 172.21.15.98:0/2909637721 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 17) v1 ==== 135766+0+0 (secure 0 0 0) 0x7f20f00145b0 con 0x7f20f8105920 2023-07-14T19:29:54.113 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:54.104+0000 7f20f5ffb700 1 --2- 172.21.15.98:0/2909637721 >> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] conn(0x7f20e80a1220 0x7f20e80a36d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:29:54.113 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:54.104+0000 7f20fce85700 1 --2- 172.21.15.98:0/2909637721 >> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] conn(0x7f20e80a1220 0x7f20e80a36d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:29:54.113 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:54.104+0000 7f20f5ffb700 1 -- 172.21.15.98:0/2909637721 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 7979+0+0 (secure 0 0 0) 0x7f20f0006e40 con 0x7f20f8105920 2023-07-14T19:29:54.113 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:54.105+0000 7f20fce85700 1 --2- 172.21.15.98:0/2909637721 >> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] conn(0x7f20e80a1220 0x7f20e80a36d0 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f20ec007900 tx=0x7f20ec008040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:29:54.114 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:54.108+0000 7f20f5ffb700 1 -- 172.21.15.98:0/2909637721 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f20f004e780 con 0x7f20f8105920 2023-07-14T19:29:54.299 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:54.299+0000 7f20ff0e9700 1 -- 172.21.15.98:0/2909637721 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f20f802ca50 con 0x7f20f8105920 2023-07-14T19:29:54.300 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:54.300+0000 7f20f5ffb700 1 -- 172.21.15.98:0/2909637721 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v83) v1 ==== 74+0+22117 (secure 0 0 0) 0x7f20f00957f0 con 0x7f20f8105920 2023-07-14T19:29:54.301 INFO:teuthology.orchestra.run.smithi098.stdout: 2023-07-14T19:29:54.301 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":83,"fsid":"9adea648-227b-11ee-9b33-001a4aab830c","created":"2023-07-14T19:22:07.178854+0000","modified":"2023-07-14T19:29:02.293099+0000","last_up_change":"2023-07-14T19:28:19.646827+0000","last_in_change":"2023-07-14T19:28:02.018325+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-07-14T19:25:09.845827+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2023-07-14T19:28:58.952921+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"82","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2023-07-14T19:29:00.228272+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"83","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"a3797e89-1754-41cf-9517-227e06ffdd72","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6802","nonce":120507228},{"type":"v1","addr":"172.21.15.98:6803","nonce":120507228}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6804","nonce":120507228},{"type":"v1","addr":"172.21.15.98:6805","nonce":120507228}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6808","nonce":120507228},{"type":"v1","addr":"172.21.15.98:6809","nonce":120507228}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6806","nonce":120507228},{"type":"v1","addr":"172.21.15.98:6807","nonce":120507228}]},"public_addr":"172.21.15.98:6803/120507228","cluster_addr":"172.21.15.98:6805/120507228","heartbeat_back_addr":"172.21.15.98:6809/120507228","heartbeat_front_addr":"172.21.15.98:6807/120507228","state":["exists","up"]},{"osd":1,"uuid":"2b001add-1aaa-4496-a403-7ef9725b80a0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.101:6800","nonce":2450584584},{"type":"v1","addr":"172.21.15.101:6801","nonce":2450584584}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.101:6802","nonce":2450584584},{"type":"v1","addr":"172.21.15.101:6803","nonce":2450584584}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.101:6806","nonce":2450584584},{"type":"v1","addr":"172.21.15.101:6807","nonce":2450584584}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.101:6804","nonce":2450584584},{"type":"v1","addr":"172.21.15.101:6805","nonce":2450584584}]},"public_addr":"172.21.15.101:6801/2450584584","cluster_addr":"172.21.15.101:6803/2450584584","heartbeat_back_addr":"172.21.15.101:6807/2450584584","heartbeat_front_addr":"172.21.15.101:6805/2450584584","state":["exists","up"]},{"osd":2,"uuid":"48e6d499-3ca9-424b-b0fb-3ba5f33b9fc9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":945038995},{"type":"v1","addr":"172.21.15.139:6801","nonce":945038995}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":945038995},{"type":"v1","addr":"172.21.15.139:6803","nonce":945038995}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":945038995},{"type":"v1","addr":"172.21.15.139:6807","nonce":945038995}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":945038995},{"type":"v1","addr":"172.21.15.139:6805","nonce":945038995}]},"public_addr":"172.21.15.139:6801/945038995","cluster_addr":"172.21.15.139:6803/945038995","heartbeat_back_addr":"172.21.15.139:6807/945038995","heartbeat_front_addr":"172.21.15.139:6805/945038995","state":["exists","up"]},{"osd":3,"uuid":"5530b265-31a5-447b-a1ea-17ebae9cfddf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6810","nonce":3095249665},{"type":"v1","addr":"172.21.15.98:6811","nonce":3095249665}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6812","nonce":3095249665},{"type":"v1","addr":"172.21.15.98:6813","nonce":3095249665}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6816","nonce":3095249665},{"type":"v1","addr":"172.21.15.98:6817","nonce":3095249665}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6814","nonce":3095249665},{"type":"v1","addr":"172.21.15.98:6815","nonce":3095249665}]},"public_addr":"172.21.15.98:6811/3095249665","cluster_addr":"172.21.15.98:6813/3095249665","heartbeat_back_addr":"172.21.15.98:6817/3095249665","heartbeat_front_addr":"172.21.15.98:6815/3095249665","state":["exists","up"]},{"osd":4,"uuid":"615d4642-a997-4f95-9e38-9fbdaa831400","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":32,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.101:6808","nonce":2844761406},{"type":"v1","addr":"172.21.15.101:6809","nonce":2844761406}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.101:6810","nonce":2844761406},{"type":"v1","addr":"172.21.15.101:6811","nonce":2844761406}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.101:6814","nonce":2844761406},{"type":"v1","addr":"172.21.15.101:6815","nonce":2844761406}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.101:6812","nonce":2844761406},{"type":"v1","addr":"172.21.15.101:6813","nonce":2844761406}]},"public_addr":"172.21.15.101:6809/2844761406","cluster_addr":"172.21.15.101:6811/2844761406","heartbeat_back_addr":"172.21.15.101:6815/2844761406","heartbeat_front_addr":"172.21.15.101:6813/2844761406","state":["exists","up"]},{"osd":5,"uuid":"415de5fa-607a-45a8-b732-28daf3be6dd3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":37,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":3532263042},{"type":"v1","addr":"172.21.15.139:6809","nonce":3532263042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6810","nonce":3532263042},{"type":"v1","addr":"172.21.15.139:6811","nonce":3532263042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6814","nonce":3532263042},{"type":"v1","addr":"172.21.15.139:6815","nonce":3532263042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":3532263042},{"type":"v1","addr":"172.21.15.139:6813","nonce":3532263042}]},"public_addr":"172.21.15.139:6809/3532263042","cluster_addr":"172.21.15.139:6811/3532263042","heartbeat_back_addr":"172.21.15.139:6815/3532263042","heartbeat_front_addr":"172.21.15.139:6813/3532263042","state":["exists","up"]},{"osd":6,"uuid":"d7f9d4ba-c309-4b75-a836-5a36384b528e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":45,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6818","nonce":658948743},{"type":"v1","addr":"172.21.15.98:6819","nonce":658948743}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6820","nonce":658948743},{"type":"v1","addr":"172.21.15.98:6821","nonce":658948743}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6824","nonce":658948743},{"type":"v1","addr":"172.21.15.98:6825","nonce":658948743}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6822","nonce":658948743},{"type":"v1","addr":"172.21.15.98:6823","nonce":658948743}]},"public_addr":"172.21.15.98:6819/658948743","cluster_addr":"172.21.15.98:6821/658948743","heartbeat_back_addr":"172.21.15.98:6825/658948743","heartbeat_front_addr":"172.21.15.98:6823/658948743","state":["exists","up"]},{"osd":7,"uuid":"eedb94d9-c5cd-4a93-8a8c-2648804b7195","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":50,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.101:6816","nonce":2881214066},{"type":"v1","addr":"172.21.15.101:6817","nonce":2881214066}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.101:6818","nonce":2881214066},{"type":"v1","addr":"172.21.15.101:6819","nonce":2881214066}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.101:6822","nonce":2881214066},{"type":"v1","addr":"172.21.15.101:6823","nonce":2881214066}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.101:6820","nonce":2881214066},{"type":"v1","addr":"172.21.15.101:6821","nonce":2881214066}]},"public_addr":"172.21.15.101:6817/2881214066","cluster_addr":"172.21.15.101:6819/2881214066","heartbeat_back_addr":"172.21.15.101:6823/2881214066","heartbeat_front_addr":"172.21.15.101:6821/2881214066","state":["exists","up"]},{"osd":8,"uuid":"6bfd77c2-5770-466e-aa6f-84b50c74ffb6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":57,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":3669514524},{"type":"v1","addr":"172.21.15.139:6817","nonce":3669514524}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":3669514524},{"type":"v1","addr":"172.21.15.139:6819","nonce":3669514524}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":3669514524},{"type":"v1","addr":"172.21.15.139:6823","nonce":3669514524}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6820","nonce":3669514524},{"type":"v1","addr":"172.21.15.139:6821","nonce":3669514524}]},"public_addr":"172.21.15.139:6817/3669514524","cluster_addr":"172.21.15.139:6819/3669514524","heartbeat_back_addr":"172.21.15.139:6823/3669514524","heartbeat_front_addr":"172.21.15.139:6821/3669514524","state":["exists","up"]},{"osd":9,"uuid":"19c5b54a-2242-4b23-b721-63be9559653d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":65,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6826","nonce":2893994080},{"type":"v1","addr":"172.21.15.98:6827","nonce":2893994080}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6828","nonce":2893994080},{"type":"v1","addr":"172.21.15.98:6829","nonce":2893994080}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6832","nonce":2893994080},{"type":"v1","addr":"172.21.15.98:6833","nonce":2893994080}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6830","nonce":2893994080},{"type":"v1","addr":"172.21.15.98:6831","nonce":2893994080}]},"public_addr":"172.21.15.98:6827/2893994080","cluster_addr":"172.21.15.98:6829/2893994080","heartbeat_back_addr":"172.21.15.98:6833/2893994080","heartbeat_front_addr":"172.21.15.98:6831/2893994080","state":["exists","up"]},{"osd":10,"uuid":"326f0209-ba9d-4b59-97cc-d61a758a8f29","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":70,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.101:6824","nonce":3545244225},{"type":"v1","addr":"172.21.15.101:6825","nonce":3545244225}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.101:6826","nonce":3545244225},{"type":"v1","addr":"172.21.15.101:6827","nonce":3545244225}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.101:6830","nonce":3545244225},{"type":"v1","addr":"172.21.15.101:6831","nonce":3545244225}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.101:6828","nonce":3545244225},{"type":"v1","addr":"172.21.15.101:6829","nonce":3545244225}]},"public_addr":"172.21.15.101:6825/3545244225","cluster_addr":"172.21.15.101:6827/3545244225","heartbeat_back_addr":"172.21.15.101:6831/3545244225","heartbeat_front_addr":"172.21.15.101:6829/3545244225","state":["exists","up"]},{"osd":11,"uuid":"c61c0320-1057-4f17-9bf1-7ad306ed7ed3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":78,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":2630023454},{"type":"v1","addr":"172.21.15.139:6825","nonce":2630023454}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6826","nonce":2630023454},{"type":"v1","addr":"172.21.15.139:6827","nonce":2630023454}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":2630023454},{"type":"v1","addr":"172.21.15.139:6831","nonce":2630023454}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":2630023454},{"type":"v1","addr":"172.21.15.139:6829","nonce":2630023454}]},"public_addr":"172.21.15.139:6825/2630023454","cluster_addr":"172.21.15.139:6827/2630023454","heartbeat_back_addr":"172.21.15.139:6831/2630023454","heartbeat_front_addr":"172.21.15.139:6829/2630023454","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T19:24:23.620028+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T19:24:45.779846+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T19:25:06.877949+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T19:25:27.994199+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T19:25:49.461139+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T19:26:10.409282+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T19:26:32.116390+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T19:26:53.536474+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T19:27:14.384839+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T19:27:35.635921+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T19:27:56.858744+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T19:28:18.342519+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.98:0/2164590253":"2023-07-15T19:23:01.520302+0000","172.21.15.98:0/2094734896":"2023-07-15T19:23:01.520302+0000","172.21.15.98:0/784745009":"2023-07-15T19:22:39.481502+0000","172.21.15.98:0/3657190818":"2023-07-15T19:22:39.481502+0000","172.21.15.98:6800/1732769087":"2023-07-15T19:23:01.520302+0000","172.21.15.98:0/3393542419":"2023-07-15T19:22:39.481502+0000","172.21.15.98:6800/749234665":"2023-07-15T19:22:39.481502+0000","172.21.15.98:6801/1732769087":"2023-07-15T19:23:01.520302+0000","172.21.15.98:6801/749234665":"2023-07-15T19:22:39.481502+0000","172.21.15.98:0/187822873":"2023-07-15T19:23:01.520302+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-07-14T19:29:54.303 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:54.302+0000 7f20ff0e9700 1 -- 172.21.15.98:0/2909637721 >> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] conn(0x7f20e80a1220 msgr2=0x7f20e80a36d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:29:54.303 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:54.302+0000 7f20ff0e9700 1 --2- 172.21.15.98:0/2909637721 >> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] conn(0x7f20e80a1220 0x7f20e80a36d0 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f20ec007900 tx=0x7f20ec008040 comp rx=0 tx=0).stop 2023-07-14T19:29:54.303 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:54.302+0000 7f20ff0e9700 1 -- 172.21.15.98:0/2909637721 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f20f8105920 msgr2=0x7f20f80febd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:29:54.303 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:54.302+0000 7f20ff0e9700 1 --2- 172.21.15.98:0/2909637721 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f20f8105920 0x7f20f80febd0 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7f20f000b970 tx=0x7f20f000bd30 comp rx=0 tx=0).stop 2023-07-14T19:29:54.303 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:54.302+0000 7f20ff0e9700 1 -- 172.21.15.98:0/2909637721 shutdown_connections 2023-07-14T19:29:54.303 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:54.302+0000 7f20ff0e9700 1 --2- 172.21.15.98:0/2909637721 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f20f8105920 0x7f20f80febd0 unknown :-1 s=CLOSED pgs=316 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:29:54.304 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:54.302+0000 7f20ff0e9700 1 --2- 172.21.15.98:0/2909637721 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f20f81006e0 0x7f20f8100000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:29:54.304 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:54.302+0000 7f20ff0e9700 1 --2- 172.21.15.98:0/2909637721 >> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] conn(0x7f20e80a1220 0x7f20e80a36d0 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:29:54.304 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:54.302+0000 7f20ff0e9700 1 --2- 172.21.15.98:0/2909637721 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f20f8103000 0x7f20f80fe690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:29:54.304 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:54.302+0000 7f20ff0e9700 1 -- 172.21.15.98:0/2909637721 >> 172.21.15.98:0/2909637721 conn(0x7f20f80fa100 msgr2=0x7f20f8101360 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:29:54.304 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:54.302+0000 7f20ff0e9700 1 -- 172.21.15.98:0/2909637721 shutdown_connections 2023-07-14T19:29:54.304 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:54.302+0000 7f20ff0e9700 1 -- 172.21.15.98:0/2909637721 wait complete. 2023-07-14T19:29:54.315 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-07-14T19:29:54.436 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:54.436+0000 7f5f0aa90700 1 -- 172.21.15.98:0/2439345813 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5f041000b0 msgr2=0x7f5f041004c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:29:54.437 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:54.436+0000 7f5f0aa90700 1 --2- 172.21.15.98:0/2439345813 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5f041000b0 0x7f5f041004c0 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7f5ef8009990 tx=0x7f5ef8009ca0 comp rx=0 tx=0).stop 2023-07-14T19:29:54.437 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:54.436+0000 7f5f0aa90700 1 -- 172.21.15.98:0/2439345813 shutdown_connections 2023-07-14T19:29:54.437 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:54.436+0000 7f5f0aa90700 1 --2- 172.21.15.98:0/2439345813 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f5f040ff6b0 0x7f5f04106f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:29:54.437 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:54.436+0000 7f5f0aa90700 1 --2- 172.21.15.98:0/2439345813 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5f041000b0 0x7f5f041004c0 unknown :-1 s=CLOSED pgs=317 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:29:54.437 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:54.436+0000 7f5f0aa90700 1 --2- 172.21.15.98:0/2439345813 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f5f040fed00 0x7f5f040ff170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:29:54.437 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:54.436+0000 7f5f0aa90700 1 -- 172.21.15.98:0/2439345813 >> 172.21.15.98:0/2439345813 conn(0x7f5f040f98a0 msgr2=0x7f5f040fbcd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:29:54.437 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:54.436+0000 7f5f0aa90700 1 -- 172.21.15.98:0/2439345813 shutdown_connections 2023-07-14T19:29:54.438 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:54.436+0000 7f5f0aa90700 1 -- 172.21.15.98:0/2439345813 wait complete. 2023-07-14T19:29:54.439 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:54.436+0000 7f5f0aa90700 1 Processor -- start 2023-07-14T19:29:54.440 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:54.437+0000 7f5f0aa90700 1 -- start start 2023-07-14T19:29:54.440 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:54.437+0000 7f5f0aa90700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f5f040fed00 0x7f5f04194860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:29:54.440 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:54.437+0000 7f5f0aa90700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5f040ff6b0 0x7f5f04194da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:29:54.440 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:54.437+0000 7f5f0aa90700 1 --2- >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f5f041000b0 0x7f5f0419be10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:29:54.440 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:54.437+0000 7f5f0aa90700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5f0419c3e0 con 0x7f5f040ff6b0 2023-07-14T19:29:54.440 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:54.437+0000 7f5f0aa90700 1 -- --> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5f0419c520 con 0x7f5f041000b0 2023-07-14T19:29:54.441 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:54.437+0000 7f5f0aa90700 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5f0419c6b0 con 0x7f5f040fed00 2023-07-14T19:29:54.441 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:54.437+0000 7f5f03fff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5f040ff6b0 0x7f5f04194da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:29:54.441 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:54.437+0000 7f5f03fff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5f040ff6b0 0x7f5f04194da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:46466/0 (socket says 172.21.15.98:46466) 2023-07-14T19:29:54.441 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:54.437+0000 7f5f03fff700 1 -- 172.21.15.98:0/3625700843 learned_addr learned my addr 172.21.15.98:0/3625700843 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-07-14T19:29:54.441 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:54.437+0000 7f5f0882c700 1 --2- 172.21.15.98:0/3625700843 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f5f040fed00 0x7f5f04194860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:29:54.441 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:54.437+0000 7f5f0902d700 1 --2- 172.21.15.98:0/3625700843 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f5f041000b0 0x7f5f0419be10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:29:54.441 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:54.437+0000 7f5f03fff700 1 -- 172.21.15.98:0/3625700843 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f5f040fed00 msgr2=0x7f5f04194860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:29:54.442 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:54.437+0000 7f5f03fff700 1 --2- 172.21.15.98:0/3625700843 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f5f040fed00 0x7f5f04194860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:29:54.442 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:54.437+0000 7f5f03fff700 1 -- 172.21.15.98:0/3625700843 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f5f041000b0 msgr2=0x7f5f0419be10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:29:54.442 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:54.437+0000 7f5f03fff700 1 --2- 172.21.15.98:0/3625700843 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f5f041000b0 0x7f5f0419be10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:29:54.442 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:54.437+0000 7f5f03fff700 1 -- 172.21.15.98:0/3625700843 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5ef80095f0 con 0x7f5f040ff6b0 2023-07-14T19:29:54.442 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:54.437+0000 7f5f0882c700 1 --2- 172.21.15.98:0/3625700843 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f5f040fed00 0x7f5f04194860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-07-14T19:29:54.442 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:54.437+0000 7f5f03fff700 1 --2- 172.21.15.98:0/3625700843 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5f040ff6b0 0x7f5f04194da0 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7f5ef000d980 tx=0x7f5ef000dd40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:29:54.442 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:54.437+0000 7f5f01ffb700 1 -- 172.21.15.98:0/3625700843 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ef0009b40 con 0x7f5f040ff6b0 2023-07-14T19:29:54.442 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:54.437+0000 7f5f01ffb700 1 -- 172.21.15.98:0/3625700843 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f5ef000de60 con 0x7f5f040ff6b0 2023-07-14T19:29:54.443 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:54.437+0000 7f5f0aa90700 1 -- 172.21.15.98:0/3625700843 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5f0419c990 con 0x7f5f040ff6b0 2023-07-14T19:29:54.443 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:54.438+0000 7f5f01ffb700 1 -- 172.21.15.98:0/3625700843 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ef0005580 con 0x7f5f040ff6b0 2023-07-14T19:29:54.443 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:54.438+0000 7f5f0aa90700 1 -- 172.21.15.98:0/3625700843 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5f0419ce50 con 0x7f5f040ff6b0 2023-07-14T19:29:54.443 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:54.439+0000 7f5f0aa90700 1 -- 172.21.15.98:0/3625700843 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5f04105aa0 con 0x7f5f040ff6b0 2023-07-14T19:29:54.443 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:54.439+0000 7f5f01ffb700 1 -- 172.21.15.98:0/3625700843 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 17) v1 ==== 135766+0+0 (secure 0 0 0) 0x7f5ef0004750 con 0x7f5f040ff6b0 2023-07-14T19:29:54.444 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:54.440+0000 7f5f01ffb700 1 --2- 172.21.15.98:0/3625700843 >> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] conn(0x7f5ef40a12a0 0x7f5ef40a3750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:29:54.444 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:54.440+0000 7f5f0882c700 1 --2- 172.21.15.98:0/3625700843 >> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] conn(0x7f5ef40a12a0 0x7f5ef40a3750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:29:54.444 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:54.440+0000 7f5f01ffb700 1 -- 172.21.15.98:0/3625700843 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 7979+0+0 (secure 0 0 0) 0x7f5ef00cc9b0 con 0x7f5f040ff6b0 2023-07-14T19:29:54.444 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:54.440+0000 7f5f0882c700 1 --2- 172.21.15.98:0/3625700843 >> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] conn(0x7f5ef40a12a0 0x7f5ef40a3750 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f5ef800b5c0 tx=0x7f5ef8005fb0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:29:54.445 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:54.444+0000 7f5f01ffb700 1 -- 172.21.15.98:0/3625700843 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f5ef0096050 con 0x7f5f040ff6b0 2023-07-14T19:29:54.678 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:54.677+0000 7f5f0aa90700 1 -- 172.21.15.98:0/3625700843 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f5f0402c8e0 con 0x7f5f040ff6b0 2023-07-14T19:29:54.678 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:54.678+0000 7f5f01ffb700 1 -- 172.21.15.98:0/3625700843 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 22 v22) v1 ==== 94+0+10826 (secure 0 0 0) 0x7f5ef0017020 con 0x7f5f040ff6b0 2023-07-14T19:29:54.679 INFO:teuthology.orchestra.run.smithi098.stdout: 2023-07-14T19:29:54.679 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":22,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14565,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6835/586280675","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6834","nonce":586280675},{"type":"v1","addr":"172.21.15.98:6835","nonce":586280675}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":4},{"gid":14577,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6837/487577257","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6836","nonce":487577257},{"type":"v1","addr":"172.21.15.98:6837","nonce":487577257}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":5},{"gid":14634,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6839/4085731209","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6838","nonce":4085731209},{"type":"v1","addr":"172.21.15.98:6839","nonce":4085731209}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":11},{"gid":14646,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6841/1679318259","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6840","nonce":1679318259},{"type":"v1","addr":"172.21.15.98:6841","nonce":1679318259}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":12},{"gid":24421,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.139:6833/3505477820","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6832","nonce":3505477820},{"type":"v1","addr":"172.21.15.139:6833","nonce":3505477820}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":24437,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.101:6833/3474460171","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.101:6832","nonce":3474460171},{"type":"v1","addr":"172.21.15.101:6833","nonce":3474460171}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":22,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2023-07-14T19:29:01.033156+0000","modified":"2023-07-14T19:29:06.291916+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24521,"mds_1":24493,"mds_2":24479},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24455":{"gid":24455,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.101:6835/1583264648","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.101:6834","nonce":1583264648},{"type":"v1","addr":"172.21.15.101:6835","nonce":1583264648}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24460":{"gid":24460,"name":"f","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.139:6835/2315281724","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6834","nonce":2315281724},{"type":"v1","addr":"172.21.15.139:6835","nonce":2315281724}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24469":{"gid":24469,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.139:6837/3597660552","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6836","nonce":3597660552},{"type":"v1","addr":"172.21.15.139:6837","nonce":3597660552}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24479":{"gid":24479,"name":"h","rank":2,"incarnation":21,"state":"up:active","state_seq":6,"addr":"172.21.15.101:6837/485741528","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.101:6836","nonce":485741528},{"type":"v1","addr":"172.21.15.101:6837","nonce":485741528}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24493":{"gid":24493,"name":"i","rank":1,"incarnation":19,"state":"up:active","state_seq":5,"addr":"172.21.15.139:6839/1258854143","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6838","nonce":1258854143},{"type":"v1","addr":"172.21.15.139:6839","nonce":1258854143}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24521":{"gid":24521,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.101:6839/48393381","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.101:6838","nonce":48393381},{"type":"v1","addr":"172.21.15.101:6839","nonce":48393381}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-07-14T19:29:54.681 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:54.680+0000 7f5f0aa90700 1 -- 172.21.15.98:0/3625700843 >> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] conn(0x7f5ef40a12a0 msgr2=0x7f5ef40a3750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:29:54.681 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:54.680+0000 7f5f0aa90700 1 --2- 172.21.15.98:0/3625700843 >> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] conn(0x7f5ef40a12a0 0x7f5ef40a3750 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f5ef800b5c0 tx=0x7f5ef8005fb0 comp rx=0 tx=0).stop 2023-07-14T19:29:54.681 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:54.680+0000 7f5f0aa90700 1 -- 172.21.15.98:0/3625700843 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5f040ff6b0 msgr2=0x7f5f04194da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:29:54.681 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:54.680+0000 7f5f0aa90700 1 --2- 172.21.15.98:0/3625700843 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5f040ff6b0 0x7f5f04194da0 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7f5ef000d980 tx=0x7f5ef000dd40 comp rx=0 tx=0).stop 2023-07-14T19:29:54.681 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:54.680+0000 7f5f0aa90700 1 -- 172.21.15.98:0/3625700843 shutdown_connections 2023-07-14T19:29:54.681 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:54.680+0000 7f5f0aa90700 1 --2- 172.21.15.98:0/3625700843 >> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] conn(0x7f5ef40a12a0 0x7f5ef40a3750 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:29:54.682 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:54.680+0000 7f5f0aa90700 1 --2- 172.21.15.98:0/3625700843 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f5f040fed00 0x7f5f04194860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:29:54.682 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:54.680+0000 7f5f0aa90700 1 --2- 172.21.15.98:0/3625700843 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f5f041000b0 0x7f5f0419be10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:29:54.682 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:54.680+0000 7f5f0aa90700 1 --2- 172.21.15.98:0/3625700843 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5f040ff6b0 0x7f5f04194da0 unknown :-1 s=CLOSED pgs=318 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:29:54.682 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:54.680+0000 7f5f0aa90700 1 -- 172.21.15.98:0/3625700843 >> 172.21.15.98:0/3625700843 conn(0x7f5f040f98a0 msgr2=0x7f5f040fbb60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:29:54.682 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:54.680+0000 7f5f0aa90700 1 -- 172.21.15.98:0/3625700843 shutdown_connections 2023-07-14T19:29:54.682 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:54.681+0000 7f5f0aa90700 1 -- 172.21.15.98:0/3625700843 wait complete. 2023-07-14T19:29:54.683 INFO:teuthology.orchestra.run.smithi098.stderr:dumped fsmap epoch 22 2023-07-14T19:29:54.693 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 22, 'flags': 50, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': True, 'refuse_client_session': False, 'balance_automate': False}, 'ever_allowed_features': 32, 'explicitly_allowed_features': 32, 'created': '2023-07-14T19:29:01.033156+0000', 'modified': '2023-07-14T19:29:06.291916+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 3, 'in': [0, 1, 2], 'up': {'mds_0': 24521, 'mds_1': 24493, 'mds_2': 24479}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_24455': {'gid': 24455, 'name': 'e', 'rank': 2, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.101:6835/1583264648', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.101:6834', 'nonce': 1583264648}, {'type': 'v1', 'addr': '172.21.15.101:6835', 'nonce': 1583264648}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138322906710015, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}}, 'gid_24460': {'gid': 24460, 'name': 'f', 'rank': 1, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.139:6835/2315281724', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.139:6834', 'nonce': 2315281724}, {'type': 'v1', 'addr': '172.21.15.139:6835', 'nonce': 2315281724}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138322906710015, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}}, 'gid_24469': {'gid': 24469, 'name': 'c', 'rank': 0, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.139:6837/3597660552', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.139:6836', 'nonce': 3597660552}, {'type': 'v1', 'addr': '172.21.15.139:6837', 'nonce': 3597660552}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138322906710015, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}}, 'gid_24479': {'gid': 24479, 'name': 'h', 'rank': 2, 'incarnation': 21, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.101:6837/485741528', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.101:6836', 'nonce': 485741528}, {'type': 'v1', 'addr': '172.21.15.101:6837', 'nonce': 485741528}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138322906710015, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}}, 'gid_24493': {'gid': 24493, 'name': 'i', 'rank': 1, 'incarnation': 19, 'state': 'up:active', 'state_seq': 5, 'addr': '172.21.15.139:6839/1258854143', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.139:6838', 'nonce': 1258854143}, {'type': 'v1', 'addr': '172.21.15.139:6839', 'nonce': 1258854143}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138322906710015, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}}, 'gid_24521': {'gid': 24521, 'name': 'b', 'rank': 0, 'incarnation': 14, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.101:6839/48393381', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.101:6838', 'nonce': 48393381}, {'type': 'v1', 'addr': '172.21.15.101:6839', 'nonce': 48393381}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138322906710015, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2023-07-14T19:29:54.694 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 3/3 2023-07-14T19:29:54.694 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.h status 2023-07-14T19:29:54.817 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:54.816+0000 7f137e7a0700 1 -- 172.21.15.98:0/2273706963 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f13780feb40 msgr2=0x7f13780fef50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:29:54.817 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:54.816+0000 7f137e7a0700 1 --2- 172.21.15.98:0/2273706963 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f13780feb40 0x7f13780fef50 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7f1368009990 tx=0x7f1368009ca0 comp rx=0 tx=0).stop 2023-07-14T19:29:54.817 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:54.816+0000 7f137e7a0700 1 -- 172.21.15.98:0/2273706963 shutdown_connections 2023-07-14T19:29:54.817 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:54.816+0000 7f137e7a0700 1 --2- 172.21.15.98:0/2273706963 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f13781006c0 0x7f1378106f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:29:54.817 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:54.816+0000 7f137e7a0700 1 --2- 172.21.15.98:0/2273706963 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f13780feb40 0x7f13780fef50 unknown :-1 s=CLOSED pgs=319 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:29:54.818 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:54.816+0000 7f137e7a0700 1 --2- 172.21.15.98:0/2273706963 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f13780ffd30 0x7f1378100180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:29:54.818 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:54.816+0000 7f137e7a0700 1 -- 172.21.15.98:0/2273706963 >> 172.21.15.98:0/2273706963 conn(0x7f13780fa160 msgr2=0x7f13780fc590 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:29:54.818 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:54.816+0000 7f137e7a0700 1 -- 172.21.15.98:0/2273706963 shutdown_connections 2023-07-14T19:29:54.818 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:54.816+0000 7f137e7a0700 1 -- 172.21.15.98:0/2273706963 wait complete. 2023-07-14T19:29:54.818 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:54.816+0000 7f137e7a0700 1 Processor -- start 2023-07-14T19:29:54.818 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:54.816+0000 7f137e7a0700 1 -- start start 2023-07-14T19:29:54.820 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:54.817+0000 7f137e7a0700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f13780feb40 0x7f13780692e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:29:54.820 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:54.817+0000 7f137e7a0700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f13780ffd30 0x7f137819a610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:29:54.820 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:54.817+0000 7f137e7a0700 1 --2- >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f13781006c0 0x7f137819cc50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:29:54.821 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:54.817+0000 7f137e7a0700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f137819d3f0 con 0x7f13780feb40 2023-07-14T19:29:54.821 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:54.817+0000 7f137e7a0700 1 -- --> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f137819d560 con 0x7f13781006c0 2023-07-14T19:29:54.821 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:54.817+0000 7f137e7a0700 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f137819d720 con 0x7f13780ffd30 2023-07-14T19:29:54.821 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:54.817+0000 7f1377fff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f13780feb40 0x7f13780692e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:29:54.821 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:54.817+0000 7f1377fff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f13780feb40 0x7f13780692e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:46478/0 (socket says 172.21.15.98:46478) 2023-07-14T19:29:54.821 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:54.817+0000 7f1377fff700 1 -- 172.21.15.98:0/92442632 learned_addr learned my addr 172.21.15.98:0/92442632 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-07-14T19:29:54.821 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:54.817+0000 7f13777fe700 1 --2- 172.21.15.98:0/92442632 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f13780ffd30 0x7f137819a610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:29:54.822 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:54.817+0000 7f137cd3d700 1 --2- 172.21.15.98:0/92442632 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f13781006c0 0x7f137819cc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:29:54.822 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:54.817+0000 7f1377fff700 1 -- 172.21.15.98:0/92442632 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f13780ffd30 msgr2=0x7f137819a610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:29:54.822 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:54.817+0000 7f1377fff700 1 --2- 172.21.15.98:0/92442632 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f13780ffd30 0x7f137819a610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:29:54.822 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:54.817+0000 7f1377fff700 1 -- 172.21.15.98:0/92442632 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f13781006c0 msgr2=0x7f137819cc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:29:54.822 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:54.817+0000 7f1377fff700 1 --2- 172.21.15.98:0/92442632 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f13781006c0 0x7f137819cc50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:29:54.822 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:54.817+0000 7f1377fff700 1 -- 172.21.15.98:0/92442632 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f13680095f0 con 0x7f13780feb40 2023-07-14T19:29:54.822 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:54.817+0000 7f137cd3d700 1 --2- 172.21.15.98:0/92442632 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f13781006c0 0x7f137819cc50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-07-14T19:29:54.822 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:54.817+0000 7f1377fff700 1 --2- 172.21.15.98:0/92442632 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f13780feb40 0x7f13780692e0 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7f1368000c00 tx=0x7f136800f510 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:29:54.823 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:54.817+0000 7f13757fa700 1 -- 172.21.15.98:0/92442632 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f136801d070 con 0x7f13780feb40 2023-07-14T19:29:54.823 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:54.817+0000 7f13757fa700 1 -- 172.21.15.98:0/92442632 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f136800fc60 con 0x7f13780feb40 2023-07-14T19:29:54.823 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:54.817+0000 7f137e7a0700 1 -- 172.21.15.98:0/92442632 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f137819d9a0 con 0x7f13780feb40 2023-07-14T19:29:54.823 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:54.818+0000 7f13757fa700 1 -- 172.21.15.98:0/92442632 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1368017650 con 0x7f13780feb40 2023-07-14T19:29:54.823 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:54.818+0000 7f137e7a0700 1 -- 172.21.15.98:0/92442632 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f137819ddc0 con 0x7f13780feb40 2023-07-14T19:29:54.823 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:54.820+0000 7f13757fa700 1 -- 172.21.15.98:0/92442632 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 17) v1 ==== 135766+0+0 (secure 0 0 0) 0x7f13680177b0 con 0x7f13780feb40 2023-07-14T19:29:54.828 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:54.820+0000 7f13757fa700 1 --2- 172.21.15.98:0/92442632 >> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] conn(0x7f13600a11e0 0x7f13600a3690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:29:54.828 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:54.820+0000 7f13777fe700 1 --2- 172.21.15.98:0/92442632 >> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] conn(0x7f13600a11e0 0x7f13600a3690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:29:54.829 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:54.820+0000 7f13757fa700 1 -- 172.21.15.98:0/92442632 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 7979+0+0 (secure 0 0 0) 0x7f13680cdd40 con 0x7f13780feb40 2023-07-14T19:29:54.829 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:54.821+0000 7f13777fe700 1 --2- 172.21.15.98:0/92442632 >> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] conn(0x7f13600a11e0 0x7f13600a3690 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f136c005ea0 tx=0x7f136c005e30 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:29:54.829 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:54.821+0000 7f138a938b80 1 Processor -- start 2023-07-14T19:29:54.829 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:54.821+0000 7f138a938b80 1 -- start start 2023-07-14T19:29:54.829 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:54.821+0000 7f138a938b80 1 --2- >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x5647e516d2d0 0x5647e516d6e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:29:54.830 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:54.821+0000 7f138a938b80 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5647e516dc20 0x5647e516e2c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:29:54.830 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:54.821+0000 7f138a938b80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5647e516e850 0x5647e516ecc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:29:54.830 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:54.821+0000 7f138a938b80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x5647e506dc60 con 0x5647e516e850 2023-07-14T19:29:54.830 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:54.821+0000 7f138a938b80 1 -- --> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] -- mon_getmap magic: 0 v1 -- 0x5647e50c09f0 con 0x5647e516d2d0 2023-07-14T19:29:54.830 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:54.821+0000 7f138a938b80 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 v1 -- 0x5647e50b9e80 con 0x5647e516dc20 2023-07-14T19:29:54.830 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:54.821+0000 7f137cd3d700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5647e516e850 0x5647e516ecc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:29:54.830 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:54.821+0000 7f137cd3d700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5647e516e850 0x5647e516ecc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:46490/0 (socket says 172.21.15.98:46490) 2023-07-14T19:29:54.831 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:54.821+0000 7f137cd3d700 1 -- 172.21.15.98:0/4239633349 learned_addr learned my addr 172.21.15.98:0/4239633349 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-07-14T19:29:54.831 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:54.821+0000 7f1377fff700 1 --2- 172.21.15.98:0/4239633349 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x5647e516d2d0 0x5647e516d6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:29:54.831 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:54.821+0000 7f13777fe700 1 --2- 172.21.15.98:0/4239633349 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5647e516dc20 0x5647e516e2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:29:54.831 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:54.821+0000 7f137cd3d700 1 -- 172.21.15.98:0/4239633349 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5647e516dc20 msgr2=0x5647e516e2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-14T19:29:54.831 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:54.821+0000 7f137cd3d700 1 --2- 172.21.15.98:0/4239633349 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5647e516dc20 0x5647e516e2c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:29:54.831 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:54.822+0000 7f137cd3d700 1 -- 172.21.15.98:0/4239633349 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x5647e516d2d0 msgr2=0x5647e516d6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-14T19:29:54.831 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:54.822+0000 7f137cd3d700 1 --2- 172.21.15.98:0/4239633349 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x5647e516d2d0 0x5647e516d6e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:29:54.832 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:54.822+0000 7f137cd3d700 1 -- 172.21.15.98:0/4239633349 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5647e4ee7d90 con 0x5647e516e850 2023-07-14T19:29:54.832 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:54.822+0000 7f1377fff700 1 --2- 172.21.15.98:0/4239633349 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x5647e516d2d0 0x5647e516d6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-07-14T19:29:54.832 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:54.822+0000 7f137cd3d700 1 --2- 172.21.15.98:0/4239633349 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5647e516e850 0x5647e516ecc0 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7f1370009400 tx=0x7f1370009920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f4bd76fd32abeaf5 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:29:54.832 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:54.822+0000 7f135e7fc700 1 -- 172.21.15.98:0/4239633349 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1370014610 con 0x5647e516e850 2023-07-14T19:29:54.832 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:54.822+0000 7f135e7fc700 1 -- 172.21.15.98:0/4239633349 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f137000f050 con 0x5647e516e850 2023-07-14T19:29:54.832 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:54.822+0000 7f135e7fc700 1 -- 172.21.15.98:0/4239633349 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1370013a70 con 0x5647e516e850 2023-07-14T19:29:54.832 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:54.822+0000 7f138a938b80 1 -- 172.21.15.98:0/4239633349 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5647e516e850 msgr2=0x5647e516ecc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:29:54.832 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:54.822+0000 7f138a938b80 1 --2- 172.21.15.98:0/4239633349 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5647e516e850 0x5647e516ecc0 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7f1370009400 tx=0x7f1370009920 comp rx=0 tx=0).stop 2023-07-14T19:29:54.833 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:54.823+0000 7f138a938b80 1 -- 172.21.15.98:0/4239633349 shutdown_connections 2023-07-14T19:29:54.833 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:54.823+0000 7f138a938b80 1 --2- 172.21.15.98:0/4239633349 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5647e516e850 0x5647e516ecc0 unknown :-1 s=CLOSED pgs=321 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:29:54.833 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:54.823+0000 7f138a938b80 1 --2- 172.21.15.98:0/4239633349 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x5647e516d2d0 0x5647e516d6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:29:54.833 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:54.823+0000 7f138a938b80 1 --2- 172.21.15.98:0/4239633349 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5647e516dc20 0x5647e516e2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:29:54.833 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:54.823+0000 7f138a938b80 1 -- 172.21.15.98:0/4239633349 >> 172.21.15.98:0/4239633349 conn(0x5647e515e630 msgr2=0x5647e5163980 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:29:54.833 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:54.823+0000 7f138a938b80 1 -- 172.21.15.98:0/4239633349 shutdown_connections 2023-07-14T19:29:54.833 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:54.823+0000 7f138a938b80 1 -- 172.21.15.98:0/4239633349 wait complete. 2023-07-14T19:29:54.834 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:54.823+0000 7f138a938b80 1 -- start start 2023-07-14T19:29:54.834 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:54.823+0000 7f138a938b80 1 Processor -- start 2023-07-14T19:29:54.834 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:54.823+0000 7f138a938b80 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5647e516d2d0 0x5647e5176cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:29:54.834 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:54.823+0000 7f138a938b80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5647e516dc20 0x5647e5177200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:29:54.834 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:54.823+0000 7f138a938b80 1 --2- >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x5647e516e850 0x5647e517e240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:29:54.834 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:54.823+0000 7f138a938b80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x5647e5059a70 con 0x5647e516dc20 2023-07-14T19:29:54.834 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:54.823+0000 7f138a938b80 1 -- --> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] -- mon_getmap magic: 0 v1 -- 0x5647e5059770 con 0x5647e516e850 2023-07-14T19:29:54.834 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:54.823+0000 7f138a938b80 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 v1 -- 0x5647e4f3d8d0 con 0x5647e516d2d0 2023-07-14T19:29:54.835 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:54.823+0000 7f138a938b80 10 client.0 _ll_register_callbacks cb 0x5647e5057710 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-07-14T19:29:54.835 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:54.823+0000 7f13777fe700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5647e516dc20 0x5647e5177200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:29:54.835 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:54.823+0000 7f13777fe700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5647e516dc20 0x5647e5177200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:46494/0 (socket says 172.21.15.98:46494) 2023-07-14T19:29:54.835 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:54.823+0000 7f13777fe700 1 -- 172.21.15.98:0/3118738620 learned_addr learned my addr 172.21.15.98:0/3118738620 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-07-14T19:29:54.835 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:54.824+0000 7f1377fff700 1 --2- 172.21.15.98:0/3118738620 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5647e516d2d0 0x5647e5176cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:29:54.835 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:54.824+0000 7f13777fe700 1 -- 172.21.15.98:0/3118738620 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5647e516d2d0 msgr2=0x5647e5176cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-14T19:29:54.835 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:54.824+0000 7f13777fe700 1 --2- 172.21.15.98:0/3118738620 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5647e516d2d0 0x5647e5176cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:29:54.836 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:54.824+0000 7f13777fe700 1 -- 172.21.15.98:0/3118738620 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x5647e516e850 msgr2=0x5647e517e240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-14T19:29:54.836 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:54.824+0000 7f13777fe700 1 --2- 172.21.15.98:0/3118738620 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x5647e516e850 0x5647e517e240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:29:54.836 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:54.824+0000 7f13777fe700 1 -- 172.21.15.98:0/3118738620 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f13700090b0 con 0x5647e516dc20 2023-07-14T19:29:54.836 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:54.824+0000 7f13777fe700 1 --2- 172.21.15.98:0/3118738620 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5647e516dc20 0x5647e5177200 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7f136c00a520 tx=0x7f136c0047c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7860605c9d2cf3f5 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:29:54.836 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:54.824+0000 7f13577fe700 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2023-07-14T19:29:54.836 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:54.824+0000 7f13577fe700 1 -- 172.21.15.98:0/3118738620 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f136c00fa50 con 0x5647e516dc20 2023-07-14T19:29:54.836 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:54.824+0000 7f13577fe700 1 -- 172.21.15.98:0/3118738620 <== mon.0 v2:172.21.15.98:3300/0 2 ==== osd_map(83..83 src has 1..83) v4 ==== 7979+0+0 (secure 0 0 0) 0x7f136c00ee80 con 0x5647e516dc20 2023-07-14T19:29:54.836 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:54.824+0000 7f138a938b80 1 -- 172.21.15.98:0/3118738620 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x5647e50134a0 con 0x5647e516dc20 2023-07-14T19:29:54.837 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:54.824+0000 7f13577fe700 1 -- 172.21.15.98:0/3118738620 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=22) v2 ==== 24+0+0 (secure 0 0 0) 0x7f136c00fbb0 con 0x5647e516dc20 2023-07-14T19:29:54.837 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:54.824+0000 7f138a938b80 10 client.15060 fetch_fsmap learned FSMap version 22 2023-07-14T19:29:54.837 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:54.824+0000 7f138a938b80 1 -- 172.21.15.98:0/3118738620 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=22,monmap=4+}) v3 -- 0x5647e4ed0c10 con 0x5647e516dc20 2023-07-14T19:29:54.837 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:54.824+0000 7f13577fe700 1 -- 172.21.15.98:0/3118738620 <== mon.0 v2:172.21.15.98:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x5647e50134a0 con 0x5647e516dc20 2023-07-14T19:29:54.837 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:54.824+0000 7f13577fe700 1 -- 172.21.15.98:0/3118738620 <== mon.0 v2:172.21.15.98:3300/0 5 ==== fsmap(e 22) v1 ==== 7462+0+0 (secure 0 0 0) 0x7f136c023020 con 0x5647e516dc20 2023-07-14T19:29:54.837 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:54.825+0000 7f138a938b80 10 client.15060 fetch_fsmap finished waiting for FSMap version 22 2023-07-14T19:29:54.837 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:54.825+0000 7f138a938b80 10 client.15060 resolve_mds: resolved name 'h' to daemon mds.h 2023-07-14T19:29:54.837 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:54.825+0000 7f138a938b80 20 client.15060 populate_metadata read hostname 'smithi098' 2023-07-14T19:29:54.838 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:54.825+0000 7f138a938b80 1 --2- 172.21.15.98:0/3118738620 >> [v2:172.21.15.101:6836/485741528,v1:172.21.15.101:6837/485741528] conn(0x5647e5181fc0 0x5647e5181360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:29:54.838 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:54.825+0000 7f138a938b80 4 client.15060 mds_command: new command op to 24479 tid=0[{"prefix": "get_command_descriptions"}] 2023-07-14T19:29:54.838 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:54.825+0000 7f138a938b80 1 -- 172.21.15.98:0/3118738620 --> [v2:172.21.15.101:6836/485741528,v1:172.21.15.101:6837/485741528] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5647e5137d50 con 0x5647e5181fc0 2023-07-14T19:29:54.838 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:54.825+0000 7f1377fff700 1 --2- 172.21.15.98:0/3118738620 >> [v2:172.21.15.101:6836/485741528,v1:172.21.15.101:6837/485741528] conn(0x5647e5181fc0 0x5647e5181360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:29:54.838 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:54.825+0000 7f13577fe700 10 client.15060 ms_handle_connect on v2:172.21.15.101:6836/485741528 2023-07-14T19:29:54.838 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:54.825+0000 7f1377fff700 1 --2- 172.21.15.98:0/3118738620 >> [v2:172.21.15.101:6836/485741528,v1:172.21.15.101:6837/485741528] conn(0x5647e5181fc0 0x5647e5181360 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=f714a9e4feed1ac6 server_cookie=9bb93b9acd0df715 in_seq=0 out_seq=0 2023-07-14T19:29:54.838 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:54.826+0000 7f13577fe700 1 -- 172.21.15.98:0/3118738620 <== mds.2 v2:172.21.15.101:6836/485741528 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18017 (crc 0 0 0) 0x5647e5137d50 con 0x5647e5181fc0 2023-07-14T19:29:54.839 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:54.826+0000 7f13577fe700 10 client.15060 handle_command_reply: tid=0 2023-07-14T19:29:54.839 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:54.826+0000 7f138a938b80 1 client.15060 shutdown 2023-07-14T19:29:54.839 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:54.826+0000 7f138a938b80 1 -- 172.21.15.98:0/3118738620 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5647e516dc20 msgr2=0x5647e5177200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:29:54.839 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:54.826+0000 7f138a938b80 1 --2- 172.21.15.98:0/3118738620 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5647e516dc20 0x5647e5177200 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7f136c00a520 tx=0x7f136c0047c0 comp rx=0 tx=0).stop 2023-07-14T19:29:54.839 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:54.826+0000 7f138a938b80 1 -- 172.21.15.98:0/3118738620 shutdown_connections 2023-07-14T19:29:54.839 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:54.826+0000 7f138a938b80 1 --2- 172.21.15.98:0/3118738620 >> [v2:172.21.15.101:6836/485741528,v1:172.21.15.101:6837/485741528] conn(0x5647e5181fc0 0x5647e5181360 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:29:54.839 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:54.826+0000 7f138a938b80 1 --2- 172.21.15.98:0/3118738620 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5647e516d2d0 0x5647e5176cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:29:54.839 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:54.826+0000 7f13577fe700 0 client.15060 ms_handle_reset on v2:172.21.15.101:6836/485741528 2023-07-14T19:29:54.840 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:54.826+0000 7f138a938b80 1 --2- 172.21.15.98:0/3118738620 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x5647e516e850 0x5647e517e240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:29:54.840 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:54.826+0000 7f138a938b80 1 --2- 172.21.15.98:0/3118738620 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5647e516dc20 0x5647e5177200 unknown :-1 s=CLOSED pgs=322 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:29:54.840 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:54.826+0000 7f138a938b80 1 -- 172.21.15.98:0/3118738620 >> 172.21.15.98:0/3118738620 conn(0x5647e515e630 msgr2=0x5647e5162600 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:29:54.840 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:54.826+0000 7f138a938b80 1 -- 172.21.15.98:0/3118738620 shutdown_connections 2023-07-14T19:29:54.840 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:54.826+0000 7f138a938b80 1 -- 172.21.15.98:0/3118738620 wait complete. 2023-07-14T19:29:54.840 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:54.827+0000 7f138a938b80 20 client.15060 trim_cache size 0 max 16384 2023-07-14T19:29:54.845 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:54.844+0000 7f138a938b80 1 Processor -- start 2023-07-14T19:29:54.845 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:54.844+0000 7f138a938b80 1 -- start start 2023-07-14T19:29:54.846 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:54.844+0000 7f138a938b80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5647e516d2d0 0x5647e5181360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:29:54.846 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:54.844+0000 7f138a938b80 1 --2- >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x5647e516dc20 0x5647e51818a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:29:54.846 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:54.844+0000 7f138a938b80 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5647e516e850 0x5647e517e200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:29:54.846 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:54.844+0000 7f138a938b80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x5647e51832e0 con 0x5647e516d2d0 2023-07-14T19:29:54.846 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:54.844+0000 7f138a938b80 1 -- --> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] -- mon_getmap magic: 0 v1 -- 0x5647e5137d50 con 0x5647e516dc20 2023-07-14T19:29:54.846 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:54.844+0000 7f138a938b80 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 v1 -- 0x5647e5088790 con 0x5647e516e850 2023-07-14T19:29:54.846 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:54.844+0000 7f1377fff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5647e516d2d0 0x5647e5181360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:29:54.847 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:54.844+0000 7f1377fff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5647e516d2d0 0x5647e5181360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:46500/0 (socket says 172.21.15.98:46500) 2023-07-14T19:29:54.847 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:54.844+0000 7f1377fff700 1 -- 172.21.15.98:0/3211636443 learned_addr learned my addr 172.21.15.98:0/3211636443 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-07-14T19:29:54.847 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:54.844+0000 7f137cd3d700 1 --2- 172.21.15.98:0/3211636443 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5647e516e850 0x5647e517e200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:29:54.847 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:54.844+0000 7f1377fff700 1 -- 172.21.15.98:0/3211636443 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5647e516e850 msgr2=0x5647e517e200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-14T19:29:54.847 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:54.844+0000 7f1377fff700 1 --2- 172.21.15.98:0/3211636443 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5647e516e850 0x5647e517e200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:29:54.848 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:54.844+0000 7f1377fff700 1 -- 172.21.15.98:0/3211636443 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x5647e516dc20 msgr2=0x5647e51818a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-14T19:29:54.848 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:54.844+0000 7f13777fe700 1 --2- 172.21.15.98:0/3211636443 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x5647e516dc20 0x5647e51818a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:29:54.848 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:54.844+0000 7f1377fff700 1 --2- 172.21.15.98:0/3211636443 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x5647e516dc20 0x5647e51818a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:29:54.848 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:54.844+0000 7f1377fff700 1 -- 172.21.15.98:0/3211636443 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5647e4f0e200 con 0x5647e516d2d0 2023-07-14T19:29:54.848 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:54.845+0000 7f1377fff700 1 --2- 172.21.15.98:0/3211636443 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5647e516d2d0 0x5647e5181360 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7f1368097ca0 tx=0x7f1368009ca0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=786f30a5360a841f server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:29:54.848 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:54.845+0000 7f135effd700 1 -- 172.21.15.98:0/3211636443 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f13680aeea0 con 0x5647e516d2d0 2023-07-14T19:29:54.848 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:54.845+0000 7f135effd700 1 -- 172.21.15.98:0/3211636443 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f136809f020 con 0x5647e516d2d0 2023-07-14T19:29:54.849 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:54.845+0000 7f135effd700 1 -- 172.21.15.98:0/3211636443 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f13680a3160 con 0x5647e516d2d0 2023-07-14T19:29:54.852 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:54.845+0000 7f138a938b80 1 -- 172.21.15.98:0/3211636443 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5647e516d2d0 msgr2=0x5647e5181360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:29:54.852 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:54.845+0000 7f138a938b80 1 --2- 172.21.15.98:0/3211636443 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5647e516d2d0 0x5647e5181360 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7f1368097ca0 tx=0x7f1368009ca0 comp rx=0 tx=0).stop 2023-07-14T19:29:54.852 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:54.845+0000 7f138a938b80 1 -- 172.21.15.98:0/3211636443 shutdown_connections 2023-07-14T19:29:54.852 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:54.845+0000 7f138a938b80 1 --2- 172.21.15.98:0/3211636443 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5647e516e850 0x5647e517e200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:29:54.852 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:54.845+0000 7f138a938b80 1 --2- 172.21.15.98:0/3211636443 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5647e516d2d0 0x5647e5181360 unknown :-1 s=CLOSED pgs=323 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:29:54.852 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:54.845+0000 7f138a938b80 1 --2- 172.21.15.98:0/3211636443 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x5647e516dc20 0x5647e51818a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:29:54.853 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:54.845+0000 7f138a938b80 1 -- 172.21.15.98:0/3211636443 >> 172.21.15.98:0/3211636443 conn(0x5647e515e630 msgr2=0x5647e5174bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:29:54.853 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:54.845+0000 7f138a938b80 1 -- 172.21.15.98:0/3211636443 shutdown_connections 2023-07-14T19:29:54.853 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:54.845+0000 7f138a938b80 1 -- 172.21.15.98:0/3211636443 wait complete. 2023-07-14T19:29:54.853 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:54.846+0000 7f138a938b80 1 -- start start 2023-07-14T19:29:54.853 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:54.846+0000 7f138a938b80 1 Processor -- start 2023-07-14T19:29:54.853 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:54.846+0000 7f138a938b80 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5647e516d2d0 0x5647e51798e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:29:54.853 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:54.846+0000 7f138a938b80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5647e516dc20 0x5647e5179e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:29:54.853 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:54.846+0000 7f138a938b80 1 --2- >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x5647e516e850 0x5647e517cc60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:29:54.854 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:54.846+0000 7f138a938b80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x5647e517d1a0 con 0x5647e516dc20 2023-07-14T19:29:54.854 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:54.846+0000 7f138a938b80 1 -- --> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] -- mon_getmap magic: 0 v1 -- 0x5647e517d310 con 0x5647e516e850 2023-07-14T19:29:54.854 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:54.846+0000 7f138a938b80 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 v1 -- 0x5647e517d4d0 con 0x5647e516d2d0 2023-07-14T19:29:54.854 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:54.846+0000 7f138a938b80 10 client.0 _ll_register_callbacks cb 0x7f136c00edd0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-07-14T19:29:54.854 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:54.846+0000 7f13777fe700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5647e516dc20 0x5647e5179e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:29:54.854 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:54.846+0000 7f13777fe700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5647e516dc20 0x5647e5179e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:46508/0 (socket says 172.21.15.98:46508) 2023-07-14T19:29:54.854 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:54.846+0000 7f13777fe700 1 -- 172.21.15.98:0/2569765513 learned_addr learned my addr 172.21.15.98:0/2569765513 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-07-14T19:29:54.855 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:54.846+0000 7f1377fff700 1 --2- 172.21.15.98:0/2569765513 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5647e516d2d0 0x5647e51798e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:29:54.855 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:54.846+0000 7f137cd3d700 1 --2- 172.21.15.98:0/2569765513 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x5647e516e850 0x5647e517cc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:29:54.855 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:54.846+0000 7f13777fe700 1 -- 172.21.15.98:0/2569765513 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5647e516d2d0 msgr2=0x5647e51798e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-14T19:29:54.855 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:54.846+0000 7f13777fe700 1 --2- 172.21.15.98:0/2569765513 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5647e516d2d0 0x5647e51798e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:29:54.855 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:54.846+0000 7f13777fe700 1 -- 172.21.15.98:0/2569765513 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x5647e516e850 msgr2=0x5647e517cc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-14T19:29:54.855 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:54.846+0000 7f13777fe700 1 --2- 172.21.15.98:0/2569765513 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x5647e516e850 0x5647e517cc60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:29:54.855 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:54.846+0000 7f13777fe700 1 -- 172.21.15.98:0/2569765513 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f137819ddc0 con 0x5647e516dc20 2023-07-14T19:29:54.855 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:54.847+0000 7f13777fe700 1 --2- 172.21.15.98:0/2569765513 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5647e516dc20 0x5647e5179e20 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7f136c00aac0 tx=0x7f136c01a5c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c3b61bce42e1ccc9 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:29:54.856 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:54.847+0000 7f13577fe700 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2023-07-14T19:29:54.856 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:54.847+0000 7f13577fe700 1 -- 172.21.15.98:0/2569765513 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f136c00e6e0 con 0x5647e516dc20 2023-07-14T19:29:54.856 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:54.847+0000 7f13577fe700 1 -- 172.21.15.98:0/2569765513 <== mon.0 v2:172.21.15.98:3300/0 2 ==== osd_map(83..83 src has 1..83) v4 ==== 7979+0+0 (secure 0 0 0) 0x7f136c00ee80 con 0x5647e516dc20 2023-07-14T19:29:54.856 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:54.847+0000 7f138a938b80 1 -- 172.21.15.98:0/2569765513 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f137000f050 con 0x5647e516dc20 2023-07-14T19:29:54.856 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:54.847+0000 7f13577fe700 1 -- 172.21.15.98:0/2569765513 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=22) v2 ==== 24+0+0 (secure 0 0 0) 0x7f136c01c5c0 con 0x5647e516dc20 2023-07-14T19:29:54.856 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:54.847+0000 7f138a938b80 10 client.15066 fetch_fsmap learned FSMap version 22 2023-07-14T19:29:54.856 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:54.847+0000 7f138a938b80 1 -- 172.21.15.98:0/2569765513 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=22,monmap=4+}) v3 -- 0x5647e4ee6260 con 0x5647e516dc20 2023-07-14T19:29:54.857 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:54.847+0000 7f13577fe700 1 -- 172.21.15.98:0/2569765513 <== mon.0 v2:172.21.15.98:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f137000f050 con 0x5647e516dc20 2023-07-14T19:29:54.857 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:54.847+0000 7f13577fe700 1 -- 172.21.15.98:0/2569765513 <== mon.0 v2:172.21.15.98:3300/0 5 ==== fsmap(e 22) v1 ==== 7462+0+0 (secure 0 0 0) 0x7f136c010a60 con 0x5647e516dc20 2023-07-14T19:29:54.857 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:54.847+0000 7f138a938b80 10 client.15066 fetch_fsmap finished waiting for FSMap version 22 2023-07-14T19:29:54.857 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:54.847+0000 7f138a938b80 10 client.15066 resolve_mds: resolved name 'h' to daemon mds.h 2023-07-14T19:29:54.857 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:54.847+0000 7f138a938b80 20 client.15066 populate_metadata read hostname 'smithi098' 2023-07-14T19:29:54.857 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:54.847+0000 7f138a938b80 1 --2- 172.21.15.98:0/2569765513 >> [v2:172.21.15.101:6836/485741528,v1:172.21.15.101:6837/485741528] conn(0x5647e5181fc0 0x5647e51d8030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:29:54.857 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:54.847+0000 7f138a938b80 4 client.15066 mds_command: new command op to 24479 tid=0[{"prefix": "status"}] 2023-07-14T19:29:54.858 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:54.847+0000 7f138a938b80 1 -- 172.21.15.98:0/2569765513 --> [v2:172.21.15.101:6836/485741528,v1:172.21.15.101:6837/485741528] -- command(tid 0: {"prefix": "status"}) v1 -- 0x5647e50b2e60 con 0x5647e5181fc0 2023-07-14T19:29:54.858 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:54.848+0000 7f1377fff700 1 --2- 172.21.15.98:0/2569765513 >> [v2:172.21.15.101:6836/485741528,v1:172.21.15.101:6837/485741528] conn(0x5647e5181fc0 0x5647e51d8030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:29:54.858 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:54.848+0000 7f13577fe700 10 client.15066 ms_handle_connect on v2:172.21.15.101:6836/485741528 2023-07-14T19:29:54.858 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:54.848+0000 7f1377fff700 1 --2- 172.21.15.98:0/2569765513 >> [v2:172.21.15.101:6836/485741528,v1:172.21.15.101:6837/485741528] conn(0x5647e5181fc0 0x5647e51d8030 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=649bd8f1d511f9c0 server_cookie=b809e928c5302ad2 in_seq=0 out_seq=0 2023-07-14T19:29:54.858 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:54.848+0000 7f13577fe700 1 -- 172.21.15.98:0/2569765513 <== mds.2 v2:172.21.15.101:6836/485741528 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+332 (crc 0 0 0) 0x5647e50b2e60 con 0x5647e5181fc0 2023-07-14T19:29:54.858 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:54.848+0000 7f13577fe700 10 client.15066 handle_command_reply: tid=0 2023-07-14T19:29:54.858 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:54.848+0000 7f138a938b80 1 client.15066 shutdown 2023-07-14T19:29:54.858 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:54.849+0000 7f138a938b80 1 -- 172.21.15.98:0/2569765513 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5647e516dc20 msgr2=0x5647e5179e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:29:54.859 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:54.849+0000 7f138a938b80 1 --2- 172.21.15.98:0/2569765513 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5647e516dc20 0x5647e5179e20 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7f136c00aac0 tx=0x7f136c01a5c0 comp rx=0 tx=0).stop 2023-07-14T19:29:54.859 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:54.849+0000 7f138a938b80 1 -- 172.21.15.98:0/2569765513 shutdown_connections 2023-07-14T19:29:54.859 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:54.849+0000 7f138a938b80 1 --2- 172.21.15.98:0/2569765513 >> [v2:172.21.15.101:6836/485741528,v1:172.21.15.101:6837/485741528] conn(0x5647e5181fc0 0x5647e51d8030 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:29:54.859 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:54.849+0000 7f138a938b80 1 --2- 172.21.15.98:0/2569765513 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x5647e516d2d0 0x5647e51798e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:29:54.859 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:54.849+0000 7f138a938b80 1 --2- 172.21.15.98:0/2569765513 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x5647e516e850 0x5647e517cc60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:29:54.859 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:54.849+0000 7f138a938b80 1 --2- 172.21.15.98:0/2569765513 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5647e516dc20 0x5647e5179e20 unknown :-1 s=CLOSED pgs=324 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:29:54.859 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:54.849+0000 7f13577fe700 0 client.15066 ms_handle_reset on v2:172.21.15.101:6836/485741528 2023-07-14T19:29:54.859 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:54.849+0000 7f138a938b80 1 -- 172.21.15.98:0/2569765513 >> 172.21.15.98:0/2569765513 conn(0x5647e515e630 msgr2=0x5647e5181360 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:29:54.860 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:54.849+0000 7f138a938b80 1 -- 172.21.15.98:0/2569765513 shutdown_connections 2023-07-14T19:29:54.860 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:54.849+0000 7f138a938b80 1 -- 172.21.15.98:0/2569765513 wait complete. 2023-07-14T19:29:54.860 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:54.849+0000 7f138a938b80 20 client.15066 trim_cache size 0 max 16384 2023-07-14T19:29:54.860 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2023-07-14T19:29:54.860 INFO:teuthology.orchestra.run.smithi098.stdout: "cluster_fsid": "9adea648-227b-11ee-9b33-001a4aab830c", 2023-07-14T19:29:54.861 INFO:teuthology.orchestra.run.smithi098.stdout: "whoami": 2, 2023-07-14T19:29:54.861 INFO:teuthology.orchestra.run.smithi098.stdout: "id": 24479, 2023-07-14T19:29:54.861 INFO:teuthology.orchestra.run.smithi098.stdout: "want_state": "up:active", 2023-07-14T19:29:54.861 INFO:teuthology.orchestra.run.smithi098.stdout: "state": "up:active", 2023-07-14T19:29:54.861 INFO:teuthology.orchestra.run.smithi098.stdout: "fs_name": "cephfs", 2023-07-14T19:29:54.861 INFO:teuthology.orchestra.run.smithi098.stdout: "rank_uptime": 49.178637631000001, 2023-07-14T19:29:54.861 INFO:teuthology.orchestra.run.smithi098.stdout: "mdsmap_epoch": 22, 2023-07-14T19:29:54.862 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap_epoch": 83, 2023-07-14T19:29:54.862 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap_epoch_barrier": 83, 2023-07-14T19:29:54.862 INFO:teuthology.orchestra.run.smithi098.stdout: "uptime": 68.159811345999998 2023-07-14T19:29:54.862 INFO:teuthology.orchestra.run.smithi098.stdout:} 2023-07-14T19:29:54.863 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:54.849+0000 7f137e7a0700 1 -- 172.21.15.98:0/92442632 >> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] conn(0x7f13600a11e0 msgr2=0x7f13600a3690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:29:54.863 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:54.849+0000 7f137e7a0700 1 --2- 172.21.15.98:0/92442632 >> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] conn(0x7f13600a11e0 0x7f13600a3690 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f136c005ea0 tx=0x7f136c005e30 comp rx=0 tx=0).stop 2023-07-14T19:29:54.863 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:54.849+0000 7f137e7a0700 1 -- 172.21.15.98:0/92442632 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f13780feb40 msgr2=0x7f13780692e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:29:54.863 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:54.849+0000 7f137e7a0700 1 --2- 172.21.15.98:0/92442632 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f13780feb40 0x7f13780692e0 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7f1368000c00 tx=0x7f136800f510 comp rx=0 tx=0).stop 2023-07-14T19:29:54.864 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:54.849+0000 7f137e7a0700 1 -- 172.21.15.98:0/92442632 shutdown_connections 2023-07-14T19:29:54.864 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:54.849+0000 7f137e7a0700 1 --2- 172.21.15.98:0/92442632 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f13781006c0 0x7f137819cc50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:29:54.864 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:54.849+0000 7f137e7a0700 1 --2- 172.21.15.98:0/92442632 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f13780feb40 0x7f13780692e0 unknown :-1 s=CLOSED pgs=320 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:29:54.864 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:54.849+0000 7f137e7a0700 1 --2- 172.21.15.98:0/92442632 >> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] conn(0x7f13600a11e0 0x7f13600a3690 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:29:54.864 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:54.849+0000 7f137e7a0700 1 --2- 172.21.15.98:0/92442632 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f13780ffd30 0x7f137819a610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:29:54.864 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:54.849+0000 7f137e7a0700 1 -- 172.21.15.98:0/92442632 >> 172.21.15.98:0/92442632 conn(0x7f13780fa160 msgr2=0x7f13780fc370 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:29:54.864 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:54.850+0000 7f137e7a0700 1 -- 172.21.15.98:0/92442632 shutdown_connections 2023-07-14T19:29:54.865 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:54.850+0000 7f137e7a0700 1 -- 172.21.15.98:0/92442632 wait complete. 2023-07-14T19:29:54.865 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.i status 2023-07-14T19:29:54.941 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:29:54 smithi101 ceph-mon[128807]: pgmap v279: 97 pgs: 97 active+clean; 634 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2023-07-14T19:29:54.941 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:29:54 smithi101 ceph-mon[128807]: from='client.? 172.21.15.98:0/3051150914' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-07-14T19:29:54.942 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:29:54 smithi101 ceph-mon[128807]: from='client.? 172.21.15.98:0/2909637721' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-07-14T19:29:54.970 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:29:54 smithi139 ceph-mon[129293]: pgmap v279: 97 pgs: 97 active+clean; 634 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2023-07-14T19:29:54.970 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:29:54 smithi139 ceph-mon[129293]: from='client.? 172.21.15.98:0/3051150914' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-07-14T19:29:54.970 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:29:54 smithi139 ceph-mon[129293]: from='client.? 172.21.15.98:0/2909637721' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-07-14T19:29:54.987 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:54.986+0000 7f24322aa700 1 -- 172.21.15.98:0/543238341 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f242c102fb0 msgr2=0x7f242c105390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:29:54.987 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:54.986+0000 7f24322aa700 1 --2- 172.21.15.98:0/543238341 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f242c102fb0 0x7f242c105390 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7f241c009990 tx=0x7f241c009ca0 comp rx=0 tx=0).stop 2023-07-14T19:29:54.987 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:54.986+0000 7f24322aa700 1 -- 172.21.15.98:0/543238341 shutdown_connections 2023-07-14T19:29:54.987 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:54.986+0000 7f24322aa700 1 --2- 172.21.15.98:0/543238341 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f242c1058d0 0x7f242c107d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:29:54.987 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:54.986+0000 7f24322aa700 1 --2- 172.21.15.98:0/543238341 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f242c100690 0x7f242c102a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:29:54.988 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:54.986+0000 7f24322aa700 1 --2- 172.21.15.98:0/543238341 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f242c102fb0 0x7f242c105390 unknown :-1 s=CLOSED pgs=325 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:29:54.988 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:54.986+0000 7f24322aa700 1 -- 172.21.15.98:0/543238341 >> 172.21.15.98:0/543238341 conn(0x7f242c0fa130 msgr2=0x7f242c0fc560 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:29:54.988 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:54.986+0000 7f24322aa700 1 -- 172.21.15.98:0/543238341 shutdown_connections 2023-07-14T19:29:54.988 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:54.986+0000 7f24322aa700 1 -- 172.21.15.98:0/543238341 wait complete. 2023-07-14T19:29:54.988 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:54.987+0000 7f24322aa700 1 Processor -- start 2023-07-14T19:29:54.988 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:54.987+0000 7f24322aa700 1 -- start start 2023-07-14T19:29:54.990 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:54.987+0000 7f24322aa700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f242c100690 0x7f242c197ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:29:54.991 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:54.987+0000 7f24322aa700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f242c102fb0 0x7f242c198020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:29:54.991 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:54.987+0000 7f24322aa700 1 --2- >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f242c1058d0 0x7f242c19f090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:29:54.992 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:54.987+0000 7f24322aa700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f242c19f660 con 0x7f242c100690 2023-07-14T19:29:54.992 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:54.987+0000 7f24322aa700 1 -- --> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f242c19f7a0 con 0x7f242c1058d0 2023-07-14T19:29:54.992 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:54.987+0000 7f24322aa700 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f242c19f930 con 0x7f242c102fb0 2023-07-14T19:29:54.992 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:54.987+0000 7f242bfff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f242c100690 0x7f242c197ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:29:54.992 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:54.987+0000 7f242bfff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f242c100690 0x7f242c197ae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:46520/0 (socket says 172.21.15.98:46520) 2023-07-14T19:29:54.992 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:54.987+0000 7f242bfff700 1 -- 172.21.15.98:0/4258613513 learned_addr learned my addr 172.21.15.98:0/4258613513 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-07-14T19:29:54.993 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:54.987+0000 7f242b7fe700 1 --2- 172.21.15.98:0/4258613513 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f242c102fb0 0x7f242c198020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:29:54.993 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:54.987+0000 7f2430847700 1 --2- 172.21.15.98:0/4258613513 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f242c1058d0 0x7f242c19f090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:29:54.993 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:54.987+0000 7f242bfff700 1 -- 172.21.15.98:0/4258613513 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f242c102fb0 msgr2=0x7f242c198020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:29:54.993 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:54.987+0000 7f242bfff700 1 --2- 172.21.15.98:0/4258613513 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f242c102fb0 0x7f242c198020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:29:54.993 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:54.987+0000 7f242bfff700 1 -- 172.21.15.98:0/4258613513 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f242c1058d0 msgr2=0x7f242c19f090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:29:54.993 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:54.987+0000 7f242bfff700 1 --2- 172.21.15.98:0/4258613513 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f242c1058d0 0x7f242c19f090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:29:54.993 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:54.987+0000 7f242bfff700 1 -- 172.21.15.98:0/4258613513 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f241c0095f0 con 0x7f242c100690 2023-07-14T19:29:54.993 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:54.987+0000 7f2430847700 1 --2- 172.21.15.98:0/4258613513 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f242c1058d0 0x7f242c19f090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-07-14T19:29:54.994 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:54.987+0000 7f242bfff700 1 --2- 172.21.15.98:0/4258613513 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f242c100690 0x7f242c197ae0 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7f241400d950 tx=0x7f241400dc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:29:54.994 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:54.988+0000 7f24297fa700 1 -- 172.21.15.98:0/4258613513 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2414009940 con 0x7f242c100690 2023-07-14T19:29:54.994 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:54.988+0000 7f24297fa700 1 -- 172.21.15.98:0/4258613513 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f2414010460 con 0x7f242c100690 2023-07-14T19:29:54.994 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:54.988+0000 7f24322aa700 1 -- 172.21.15.98:0/4258613513 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f242c19fc10 con 0x7f242c100690 2023-07-14T19:29:54.994 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:54.988+0000 7f24297fa700 1 -- 172.21.15.98:0/4258613513 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f241400f620 con 0x7f242c100690 2023-07-14T19:29:54.994 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:54.988+0000 7f24322aa700 1 -- 172.21.15.98:0/4258613513 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f242c1a0170 con 0x7f242c100690 2023-07-14T19:29:54.994 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:54.990+0000 7f24297fa700 1 -- 172.21.15.98:0/4258613513 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 17) v1 ==== 135766+0+0 (secure 0 0 0) 0x7f24140108d0 con 0x7f242c100690 2023-07-14T19:29:54.995 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:54.990+0000 7f24297fa700 1 --2- 172.21.15.98:0/4258613513 >> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] conn(0x7f24180a1260 0x7f24180a3710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:29:54.995 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:54.990+0000 7f242b7fe700 1 --2- 172.21.15.98:0/4258613513 >> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] conn(0x7f24180a1260 0x7f24180a3710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:29:54.995 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:54.990+0000 7f24297fa700 1 -- 172.21.15.98:0/4258613513 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 7979+0+0 (secure 0 0 0) 0x7f24140cc6c0 con 0x7f242c100690 2023-07-14T19:29:54.999 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:54.991+0000 7f242b7fe700 1 --2- 172.21.15.98:0/4258613513 >> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] conn(0x7f24180a1260 0x7f24180a3710 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f241c00b2e0 tx=0x7f241c005fb0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:29:54.999 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:54.991+0000 7f243e442b80 1 Processor -- start 2023-07-14T19:29:55.000 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:54.991+0000 7f243e442b80 1 -- start start 2023-07-14T19:29:55.000 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:54.991+0000 7f243e442b80 1 --2- >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x55d12cf8bf10 0x55d12cf8c320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:29:55.000 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:54.991+0000 7f243e442b80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d12cf8c860 0x55d12cf8ce80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:29:55.000 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:54.991+0000 7f243e442b80 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55d12cf8d460 0x55d12cf8d8d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:29:55.000 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:54.991+0000 7f243e442b80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d12ce90c60 con 0x55d12cf8c860 2023-07-14T19:29:55.000 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:54.991+0000 7f243e442b80 1 -- --> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d12cee39f0 con 0x55d12cf8bf10 2023-07-14T19:29:55.001 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:54.991+0000 7f243e442b80 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d12cedce80 con 0x55d12cf8d460 2023-07-14T19:29:55.001 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:54.991+0000 7f242b7fe700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d12cf8c860 0x55d12cf8ce80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:29:55.001 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:54.991+0000 7f242b7fe700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d12cf8c860 0x55d12cf8ce80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:46530/0 (socket says 172.21.15.98:46530) 2023-07-14T19:29:55.001 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:54.991+0000 7f242b7fe700 1 -- 172.21.15.98:0/2419547003 learned_addr learned my addr 172.21.15.98:0/2419547003 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-07-14T19:29:55.001 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:54.992+0000 7f242bfff700 1 --2- 172.21.15.98:0/2419547003 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x55d12cf8bf10 0x55d12cf8c320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:29:55.001 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:54.992+0000 7f242b7fe700 1 -- 172.21.15.98:0/2419547003 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55d12cf8d460 msgr2=0x55d12cf8d8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-14T19:29:55.002 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:54.992+0000 7f2430847700 1 --2- 172.21.15.98:0/2419547003 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55d12cf8d460 0x55d12cf8d8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:29:55.002 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:54.992+0000 7f242b7fe700 1 --2- 172.21.15.98:0/2419547003 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55d12cf8d460 0x55d12cf8d8d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:29:55.002 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:54.992+0000 7f242b7fe700 1 -- 172.21.15.98:0/2419547003 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x55d12cf8bf10 msgr2=0x55d12cf8c320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-14T19:29:55.002 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:54.992+0000 7f242b7fe700 1 --2- 172.21.15.98:0/2419547003 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x55d12cf8bf10 0x55d12cf8c320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:29:55.003 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:54.992+0000 7f242b7fe700 1 -- 172.21.15.98:0/2419547003 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55d12cd0ad90 con 0x55d12cf8c860 2023-07-14T19:29:55.003 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:54.992+0000 7f242b7fe700 1 --2- 172.21.15.98:0/2419547003 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d12cf8c860 0x55d12cf8ce80 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7f241c00bfc0 tx=0x7f241c020000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5d7da6fe52c7e790 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:29:55.003 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:54.992+0000 7f24227fc700 1 -- 172.21.15.98:0/2419547003 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f241c02b070 con 0x55d12cf8c860 2023-07-14T19:29:55.003 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:54.992+0000 7f24227fc700 1 -- 172.21.15.98:0/2419547003 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f241c01e040 con 0x55d12cf8c860 2023-07-14T19:29:55.003 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:54.992+0000 7f24227fc700 1 -- 172.21.15.98:0/2419547003 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f241c00f9a0 con 0x55d12cf8c860 2023-07-14T19:29:55.003 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:54.993+0000 7f243e442b80 1 -- 172.21.15.98:0/2419547003 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d12cf8c860 msgr2=0x55d12cf8ce80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:29:55.003 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:54.993+0000 7f243e442b80 1 --2- 172.21.15.98:0/2419547003 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d12cf8c860 0x55d12cf8ce80 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7f241c00bfc0 tx=0x7f241c020000 comp rx=0 tx=0).stop 2023-07-14T19:29:55.003 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:54.993+0000 7f243e442b80 1 -- 172.21.15.98:0/2419547003 shutdown_connections 2023-07-14T19:29:55.004 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:54.993+0000 7f243e442b80 1 --2- 172.21.15.98:0/2419547003 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55d12cf8d460 0x55d12cf8d8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:29:55.004 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:54.993+0000 7f243e442b80 1 --2- 172.21.15.98:0/2419547003 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x55d12cf8bf10 0x55d12cf8c320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:29:55.004 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:54.993+0000 7f243e442b80 1 --2- 172.21.15.98:0/2419547003 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d12cf8c860 0x55d12cf8ce80 unknown :-1 s=CLOSED pgs=327 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:29:55.004 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:54.993+0000 7f243e442b80 1 -- 172.21.15.98:0/2419547003 >> 172.21.15.98:0/2419547003 conn(0x55d12cf81630 msgr2=0x55d12cf86980 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:29:55.004 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:54.993+0000 7f243e442b80 1 -- 172.21.15.98:0/2419547003 shutdown_connections 2023-07-14T19:29:55.004 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:54.993+0000 7f243e442b80 1 -- 172.21.15.98:0/2419547003 wait complete. 2023-07-14T19:29:55.004 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:54.993+0000 7f243e442b80 1 -- start start 2023-07-14T19:29:55.005 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:54.993+0000 7f243e442b80 1 Processor -- start 2023-07-14T19:29:55.005 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:54.993+0000 7f243e442b80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d12cf8bf10 0x55d12cf95860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:29:55.005 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:54.993+0000 7f243e442b80 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55d12cf8c860 0x55d12cf95da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:29:55.005 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:54.993+0000 7f243e442b80 1 --2- >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x55d12cf8d460 0x55d12cf9cde0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:29:55.005 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:54.993+0000 7f243e442b80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d12ce7ca70 con 0x55d12cf8bf10 2023-07-14T19:29:55.005 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:54.993+0000 7f243e442b80 1 -- --> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d12ce7c770 con 0x55d12cf8d460 2023-07-14T19:29:55.005 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:54.993+0000 7f243e442b80 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d12cd608d0 con 0x55d12cf8c860 2023-07-14T19:29:55.005 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:54.994+0000 7f243e442b80 10 client.0 _ll_register_callbacks cb 0x55d12ce7a710 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-07-14T19:29:55.006 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:54.994+0000 7f242bfff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d12cf8bf10 0x55d12cf95860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:29:55.006 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:54.994+0000 7f242bfff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d12cf8bf10 0x55d12cf95860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:46534/0 (socket says 172.21.15.98:46534) 2023-07-14T19:29:55.006 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:54.994+0000 7f242bfff700 1 -- 172.21.15.98:0/406663652 learned_addr learned my addr 172.21.15.98:0/406663652 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-07-14T19:29:55.006 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:54.994+0000 7f242b7fe700 1 --2- 172.21.15.98:0/406663652 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55d12cf8c860 0x55d12cf95da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:29:55.006 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:54.994+0000 7f2430847700 1 --2- 172.21.15.98:0/406663652 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x55d12cf8d460 0x55d12cf9cde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:29:55.006 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:54.994+0000 7f242bfff700 1 -- 172.21.15.98:0/406663652 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55d12cf8c860 msgr2=0x55d12cf95da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-14T19:29:55.006 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:54.994+0000 7f242bfff700 1 --2- 172.21.15.98:0/406663652 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55d12cf8c860 0x55d12cf95da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:29:55.007 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:54.994+0000 7f242bfff700 1 -- 172.21.15.98:0/406663652 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x55d12cf8d460 msgr2=0x55d12cf9cde0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-14T19:29:55.007 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:54.994+0000 7f242bfff700 1 --2- 172.21.15.98:0/406663652 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x55d12cf8d460 0x55d12cf9cde0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:29:55.007 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:54.994+0000 7f242bfff700 1 -- 172.21.15.98:0/406663652 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f242c108920 con 0x55d12cf8bf10 2023-07-14T19:29:55.007 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:54.994+0000 7f242bfff700 1 --2- 172.21.15.98:0/406663652 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d12cf8bf10 0x55d12cf95860 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7f2414096020 tx=0x7f241409db50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fccdcfee64e8fbe4 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:29:55.007 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:54.994+0000 7f240b7fe700 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2023-07-14T19:29:55.007 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:54.994+0000 7f240b7fe700 1 -- 172.21.15.98:0/406663652 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f241409a050 con 0x55d12cf8bf10 2023-07-14T19:29:55.007 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:54.994+0000 7f240b7fe700 1 -- 172.21.15.98:0/406663652 <== mon.0 v2:172.21.15.98:3300/0 2 ==== osd_map(83..83 src has 1..83) v4 ==== 7979+0+0 (secure 0 0 0) 0x7f2414099210 con 0x55d12cf8bf10 2023-07-14T19:29:55.007 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:54.994+0000 7f243e442b80 1 -- 172.21.15.98:0/406663652 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55d12ce364a0 con 0x55d12cf8bf10 2023-07-14T19:29:55.008 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:54.994+0000 7f240b7fe700 1 -- 172.21.15.98:0/406663652 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=22) v2 ==== 24+0+0 (secure 0 0 0) 0x7f24140b2200 con 0x55d12cf8bf10 2023-07-14T19:29:55.008 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:54.994+0000 7f243e442b80 10 client.15078 fetch_fsmap learned FSMap version 22 2023-07-14T19:29:55.008 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:54.994+0000 7f243e442b80 1 -- 172.21.15.98:0/406663652 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=22,monmap=4+}) v3 -- 0x55d12ccf3c10 con 0x55d12cf8bf10 2023-07-14T19:29:55.008 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:54.994+0000 7f240b7fe700 1 -- 172.21.15.98:0/406663652 <== mon.0 v2:172.21.15.98:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x55d12ce364a0 con 0x55d12cf8bf10 2023-07-14T19:29:55.008 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:54.995+0000 7f240b7fe700 1 -- 172.21.15.98:0/406663652 <== mon.0 v2:172.21.15.98:3300/0 5 ==== fsmap(e 22) v1 ==== 7462+0+0 (secure 0 0 0) 0x7f2414009aa0 con 0x55d12cf8bf10 2023-07-14T19:29:55.008 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:54.995+0000 7f243e442b80 10 client.15078 fetch_fsmap finished waiting for FSMap version 22 2023-07-14T19:29:55.008 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:54.995+0000 7f243e442b80 10 client.15078 resolve_mds: resolved name 'i' to daemon mds.i 2023-07-14T19:29:55.009 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:54.995+0000 7f243e442b80 20 client.15078 populate_metadata read hostname 'smithi098' 2023-07-14T19:29:55.009 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:54.995+0000 7f243e442b80 1 --2- 172.21.15.98:0/406663652 >> [v2:172.21.15.139:6838/1258854143,v1:172.21.15.139:6839/1258854143] conn(0x55d12cfa0960 0x55d12cf9fdd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:29:55.009 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:54.995+0000 7f243e442b80 4 client.15078 mds_command: new command op to 24493 tid=0[{"prefix": "get_command_descriptions"}] 2023-07-14T19:29:55.009 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:54.995+0000 7f243e442b80 1 -- 172.21.15.98:0/406663652 --> [v2:172.21.15.139:6838/1258854143,v1:172.21.15.139:6839/1258854143] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55d12cf5ad50 con 0x55d12cfa0960 2023-07-14T19:29:55.009 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:54.995+0000 7f242b7fe700 1 --2- 172.21.15.98:0/406663652 >> [v2:172.21.15.139:6838/1258854143,v1:172.21.15.139:6839/1258854143] conn(0x55d12cfa0960 0x55d12cf9fdd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:29:55.009 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:54.995+0000 7f240b7fe700 10 client.15078 ms_handle_connect on v2:172.21.15.139:6838/1258854143 2023-07-14T19:29:55.009 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:54.995+0000 7f242b7fe700 1 --2- 172.21.15.98:0/406663652 >> [v2:172.21.15.139:6838/1258854143,v1:172.21.15.139:6839/1258854143] conn(0x55d12cfa0960 0x55d12cf9fdd0 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=56dc639962adb691 server_cookie=916c031d4ed35fbd in_seq=0 out_seq=0 2023-07-14T19:29:55.009 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:54.996+0000 7f240b7fe700 1 -- 172.21.15.98:0/406663652 <== mds.1 v2:172.21.15.139:6838/1258854143 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18017 (crc 0 0 0) 0x55d12cf5ad50 con 0x55d12cfa0960 2023-07-14T19:29:55.010 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:54.996+0000 7f240b7fe700 10 client.15078 handle_command_reply: tid=0 2023-07-14T19:29:55.010 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:54.996+0000 7f243e442b80 1 client.15078 shutdown 2023-07-14T19:29:55.010 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:54.996+0000 7f243e442b80 1 -- 172.21.15.98:0/406663652 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d12cf8bf10 msgr2=0x55d12cf95860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:29:55.010 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:54.996+0000 7f243e442b80 1 --2- 172.21.15.98:0/406663652 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d12cf8bf10 0x55d12cf95860 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7f2414096020 tx=0x7f241409db50 comp rx=0 tx=0).stop 2023-07-14T19:29:55.010 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:54.996+0000 7f243e442b80 1 -- 172.21.15.98:0/406663652 shutdown_connections 2023-07-14T19:29:55.010 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:54.996+0000 7f243e442b80 1 --2- 172.21.15.98:0/406663652 >> [v2:172.21.15.139:6838/1258854143,v1:172.21.15.139:6839/1258854143] conn(0x55d12cfa0960 0x55d12cf9fdd0 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:29:55.010 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:54.996+0000 7f243e442b80 1 --2- 172.21.15.98:0/406663652 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x55d12cf8d460 0x55d12cf9cde0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:29:55.010 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:54.996+0000 7f243e442b80 1 --2- 172.21.15.98:0/406663652 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d12cf8bf10 0x55d12cf95860 unknown :-1 s=CLOSED pgs=328 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:29:55.011 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:54.996+0000 7f240b7fe700 0 client.15078 ms_handle_reset on v2:172.21.15.139:6838/1258854143 2023-07-14T19:29:55.011 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:54.996+0000 7f243e442b80 1 --2- 172.21.15.98:0/406663652 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55d12cf8c860 0x55d12cf95da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:29:55.011 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:54.996+0000 7f243e442b80 1 -- 172.21.15.98:0/406663652 >> 172.21.15.98:0/406663652 conn(0x55d12cf81630 msgr2=0x55d12cf85600 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:29:55.011 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:54.996+0000 7f243e442b80 1 -- 172.21.15.98:0/406663652 shutdown_connections 2023-07-14T19:29:55.011 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:54.996+0000 7f243e442b80 1 -- 172.21.15.98:0/406663652 wait complete. 2023-07-14T19:29:55.011 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:54.996+0000 7f243e442b80 20 client.15078 trim_cache size 0 max 16384 2023-07-14T19:29:55.015 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:55.014+0000 7f243e442b80 1 Processor -- start 2023-07-14T19:29:55.015 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:55.014+0000 7f243e442b80 1 -- start start 2023-07-14T19:29:55.016 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:55.014+0000 7f243e442b80 1 --2- >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x55d12cf8bf10 0x55d12cf94f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:29:55.016 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:55.014+0000 7f243e442b80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d12cf8c860 0x55d12cf9cda0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:29:55.016 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:55.014+0000 7f243e442b80 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55d12cf8d460 0x55d12cf9d5b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:29:55.016 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:55.014+0000 7f243e442b80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d12ceab7c0 con 0x55d12cf8c860 2023-07-14T19:29:55.016 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:55.014+0000 7f243e442b80 1 -- --> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d12cd6ce10 con 0x55d12cf8bf10 2023-07-14T19:29:55.016 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:55.014+0000 7f243e442b80 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d12ce78d50 con 0x55d12cf8d460 2023-07-14T19:29:55.016 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:55.014+0000 7f242b7fe700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d12cf8c860 0x55d12cf9cda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:29:55.017 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:55.014+0000 7f242b7fe700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d12cf8c860 0x55d12cf9cda0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:46544/0 (socket says 172.21.15.98:46544) 2023-07-14T19:29:55.017 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:55.014+0000 7f242b7fe700 1 -- 172.21.15.98:0/2134643785 learned_addr learned my addr 172.21.15.98:0/2134643785 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-07-14T19:29:55.017 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:55.015+0000 7f2430847700 1 --2- 172.21.15.98:0/2134643785 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55d12cf8d460 0x55d12cf9d5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:29:55.017 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:55.015+0000 7f242bfff700 1 --2- 172.21.15.98:0/2134643785 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x55d12cf8bf10 0x55d12cf94f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:29:55.017 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:55.015+0000 7f242b7fe700 1 -- 172.21.15.98:0/2134643785 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55d12cf8d460 msgr2=0x55d12cf9d5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-14T19:29:55.017 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:55.015+0000 7f242b7fe700 1 --2- 172.21.15.98:0/2134643785 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55d12cf8d460 0x55d12cf9d5b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:29:55.017 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:55.015+0000 7f242b7fe700 1 -- 172.21.15.98:0/2134643785 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x55d12cf8bf10 msgr2=0x55d12cf94f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-14T19:29:55.018 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:55.015+0000 7f242b7fe700 1 --2- 172.21.15.98:0/2134643785 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x55d12cf8bf10 0x55d12cf94f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:29:55.018 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:55.015+0000 7f242b7fe700 1 -- 172.21.15.98:0/2134643785 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55d12cf96a30 con 0x55d12cf8c860 2023-07-14T19:29:55.018 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:55.015+0000 7f242b7fe700 1 --2- 172.21.15.98:0/2134643785 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d12cf8c860 0x55d12cf9cda0 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7f241c009990 tx=0x7f241c020000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=803c4bc3ec354017 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:29:55.018 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:55.015+0000 7f2422ffd700 1 -- 172.21.15.98:0/2134643785 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f241c02b040 con 0x55d12cf8c860 2023-07-14T19:29:55.018 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:55.015+0000 7f2422ffd700 1 -- 172.21.15.98:0/2134643785 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f241c03c500 con 0x55d12cf8c860 2023-07-14T19:29:55.018 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:55.015+0000 7f2422ffd700 1 -- 172.21.15.98:0/2134643785 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f241c03cc20 con 0x55d12cf8c860 2023-07-14T19:29:55.021 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:55.016+0000 7f243e442b80 1 -- 172.21.15.98:0/2134643785 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d12cf8c860 msgr2=0x55d12cf9cda0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:29:55.022 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:55.016+0000 7f243e442b80 1 --2- 172.21.15.98:0/2134643785 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d12cf8c860 0x55d12cf9cda0 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7f241c009990 tx=0x7f241c020000 comp rx=0 tx=0).stop 2023-07-14T19:29:55.022 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:55.016+0000 7f243e442b80 1 -- 172.21.15.98:0/2134643785 shutdown_connections 2023-07-14T19:29:55.022 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:55.016+0000 7f243e442b80 1 --2- 172.21.15.98:0/2134643785 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55d12cf8d460 0x55d12cf9d5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:29:55.022 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:55.016+0000 7f243e442b80 1 --2- 172.21.15.98:0/2134643785 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x55d12cf8bf10 0x55d12cf94f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:29:55.023 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:55.016+0000 7f243e442b80 1 --2- 172.21.15.98:0/2134643785 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d12cf8c860 0x55d12cf9cda0 unknown :-1 s=CLOSED pgs=329 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:29:55.023 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:55.016+0000 7f243e442b80 1 -- 172.21.15.98:0/2134643785 >> 172.21.15.98:0/2134643785 conn(0x55d12cf81630 msgr2=0x55d12cf94590 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:29:55.023 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:55.016+0000 7f243e442b80 1 -- 172.21.15.98:0/2134643785 shutdown_connections 2023-07-14T19:29:55.023 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:55.016+0000 7f243e442b80 1 -- 172.21.15.98:0/2134643785 wait complete. 2023-07-14T19:29:55.023 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:55.016+0000 7f243e442b80 1 -- start start 2023-07-14T19:29:55.023 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:55.016+0000 7f243e442b80 1 Processor -- start 2023-07-14T19:29:55.023 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:55.016+0000 7f243e442b80 1 --2- >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x55d12cf8bf10 0x55d12cf99ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:29:55.024 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:55.016+0000 7f243e442b80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d12cf8c860 0x55d12cf9c540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:29:55.024 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:55.016+0000 7f243e442b80 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55d12cf8d460 0x55d12cf9cbe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:29:55.024 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:55.016+0000 7f243e442b80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d12cff45e0 con 0x55d12cf8c860 2023-07-14T19:29:55.024 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:55.016+0000 7f243e442b80 1 -- --> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d12cff4750 con 0x55d12cf8bf10 2023-07-14T19:29:55.024 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:55.016+0000 7f243e442b80 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d12cff4910 con 0x55d12cf8d460 2023-07-14T19:29:55.024 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:55.017+0000 7f243e442b80 10 client.0 _ll_register_callbacks cb 0x7f2414099480 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-07-14T19:29:55.024 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:55.017+0000 7f242b7fe700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d12cf8c860 0x55d12cf9c540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:29:55.025 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:55.017+0000 7f242b7fe700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d12cf8c860 0x55d12cf9c540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:46550/0 (socket says 172.21.15.98:46550) 2023-07-14T19:29:55.025 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:55.017+0000 7f242b7fe700 1 -- 172.21.15.98:0/817111129 learned_addr learned my addr 172.21.15.98:0/817111129 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-07-14T19:29:55.025 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:55.017+0000 7f242bfff700 1 --2- 172.21.15.98:0/817111129 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x55d12cf8bf10 0x55d12cf99ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:29:55.025 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:55.017+0000 7f2430847700 1 --2- 172.21.15.98:0/817111129 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55d12cf8d460 0x55d12cf9cbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:29:55.025 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:55.017+0000 7f242b7fe700 1 -- 172.21.15.98:0/817111129 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55d12cf8d460 msgr2=0x55d12cf9cbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-14T19:29:55.025 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:55.017+0000 7f242b7fe700 1 --2- 172.21.15.98:0/817111129 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55d12cf8d460 0x55d12cf9cbe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:29:55.025 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:55.017+0000 7f242b7fe700 1 -- 172.21.15.98:0/817111129 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x55d12cf8bf10 msgr2=0x55d12cf99ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-14T19:29:55.026 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:55.017+0000 7f242b7fe700 1 --2- 172.21.15.98:0/817111129 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x55d12cf8bf10 0x55d12cf99ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:29:55.026 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:55.017+0000 7f242b7fe700 1 -- 172.21.15.98:0/817111129 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55d12cd0ad90 con 0x55d12cf8c860 2023-07-14T19:29:55.026 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:55.017+0000 7f242bfff700 1 --2- 172.21.15.98:0/817111129 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x55d12cf8bf10 0x55d12cf99ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-07-14T19:29:55.026 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:55.017+0000 7f242b7fe700 1 --2- 172.21.15.98:0/817111129 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d12cf8c860 0x55d12cf9c540 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7f241c000c00 tx=0x7f241c0057e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=548e273baf2c68ad server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:29:55.026 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:55.017+0000 7f240b7fe700 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2023-07-14T19:29:55.026 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:55.017+0000 7f240b7fe700 1 -- 172.21.15.98:0/817111129 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f241c02b070 con 0x55d12cf8c860 2023-07-14T19:29:55.026 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:55.017+0000 7f240b7fe700 1 -- 172.21.15.98:0/817111129 <== mon.0 v2:172.21.15.98:3300/0 2 ==== osd_map(83..83 src has 1..83) v4 ==== 7979+0+0 (secure 0 0 0) 0x7f241c03c030 con 0x55d12cf8c860 2023-07-14T19:29:55.026 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:55.017+0000 7f243e442b80 1 -- 172.21.15.98:0/817111129 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f241c01e040 con 0x55d12cf8c860 2023-07-14T19:29:55.027 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:55.017+0000 7f240b7fe700 1 -- 172.21.15.98:0/817111129 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=22) v2 ==== 24+0+0 (secure 0 0 0) 0x7f241c03c1f0 con 0x55d12cf8c860 2023-07-14T19:29:55.027 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:55.017+0000 7f243e442b80 10 client.15084 fetch_fsmap learned FSMap version 22 2023-07-14T19:29:55.027 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:55.017+0000 7f243e442b80 1 -- 172.21.15.98:0/817111129 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=22,monmap=4+}) v3 -- 0x55d12cd31200 con 0x55d12cf8c860 2023-07-14T19:29:55.027 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:55.017+0000 7f240b7fe700 1 -- 172.21.15.98:0/817111129 <== mon.0 v2:172.21.15.98:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f241c01e040 con 0x55d12cf8c860 2023-07-14T19:29:55.027 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:55.018+0000 7f240b7fe700 1 -- 172.21.15.98:0/817111129 <== mon.0 v2:172.21.15.98:3300/0 5 ==== fsmap(e 22) v1 ==== 7462+0+0 (secure 0 0 0) 0x7f241c045030 con 0x55d12cf8c860 2023-07-14T19:29:55.027 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:55.018+0000 7f243e442b80 10 client.15084 fetch_fsmap finished waiting for FSMap version 22 2023-07-14T19:29:55.027 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:55.018+0000 7f243e442b80 10 client.15084 resolve_mds: resolved name 'i' to daemon mds.i 2023-07-14T19:29:55.027 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:55.018+0000 7f243e442b80 20 client.15084 populate_metadata read hostname 'smithi098' 2023-07-14T19:29:55.028 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:55.018+0000 7f243e442b80 1 --2- 172.21.15.98:0/817111129 >> [v2:172.21.15.139:6838/1258854143,v1:172.21.15.139:6839/1258854143] conn(0x55d12cff9200 0x55d12cff5b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:29:55.028 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:55.018+0000 7f243e442b80 4 client.15084 mds_command: new command op to 24493 tid=0[{"prefix": "status"}] 2023-07-14T19:29:55.028 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:55.018+0000 7f243e442b80 1 -- 172.21.15.98:0/817111129 --> [v2:172.21.15.139:6838/1258854143,v1:172.21.15.139:6839/1258854143] -- command(tid 0: {"prefix": "status"}) v1 -- 0x55d12ced5e60 con 0x55d12cff9200 2023-07-14T19:29:55.028 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:55.018+0000 7f242bfff700 1 --2- 172.21.15.98:0/817111129 >> [v2:172.21.15.139:6838/1258854143,v1:172.21.15.139:6839/1258854143] conn(0x55d12cff9200 0x55d12cff5b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:29:55.028 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:55.018+0000 7f240b7fe700 10 client.15084 ms_handle_connect on v2:172.21.15.139:6838/1258854143 2023-07-14T19:29:55.028 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:55.018+0000 7f242bfff700 1 --2- 172.21.15.98:0/817111129 >> [v2:172.21.15.139:6838/1258854143,v1:172.21.15.139:6839/1258854143] conn(0x55d12cff9200 0x55d12cff5b90 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=4197c33a1ec666a5 server_cookie=a1218219b9455a57 in_seq=0 out_seq=0 2023-07-14T19:29:55.028 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:55.019+0000 7f240b7fe700 1 -- 172.21.15.98:0/817111129 <== mds.1 v2:172.21.15.139:6838/1258854143 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+332 (crc 0 0 0) 0x55d12ced5e60 con 0x55d12cff9200 2023-07-14T19:29:55.029 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:55.019+0000 7f240b7fe700 10 client.15084 handle_command_reply: tid=0 2023-07-14T19:29:55.029 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:55.019+0000 7f243e442b80 1 client.15084 shutdown 2023-07-14T19:29:55.029 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:55.019+0000 7f243e442b80 1 -- 172.21.15.98:0/817111129 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d12cf8c860 msgr2=0x55d12cf9c540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:29:55.029 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:55.019+0000 7f243e442b80 1 --2- 172.21.15.98:0/817111129 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d12cf8c860 0x55d12cf9c540 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7f241c000c00 tx=0x7f241c0057e0 comp rx=0 tx=0).stop 2023-07-14T19:29:55.029 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:55.019+0000 7f243e442b80 1 -- 172.21.15.98:0/817111129 shutdown_connections 2023-07-14T19:29:55.029 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:55.019+0000 7f243e442b80 1 --2- 172.21.15.98:0/817111129 >> [v2:172.21.15.139:6838/1258854143,v1:172.21.15.139:6839/1258854143] conn(0x55d12cff9200 0x55d12cff5b90 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:29:55.029 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:55.019+0000 7f243e442b80 1 --2- 172.21.15.98:0/817111129 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55d12cf8d460 0x55d12cf9cbe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:29:55.029 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:55.019+0000 7f240b7fe700 0 client.15084 ms_handle_reset on v2:172.21.15.139:6838/1258854143 2023-07-14T19:29:55.030 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:55.019+0000 7f243e442b80 1 --2- 172.21.15.98:0/817111129 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x55d12cf8bf10 0x55d12cf99ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:29:55.030 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:55.019+0000 7f243e442b80 1 --2- 172.21.15.98:0/817111129 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d12cf8c860 0x55d12cf9c540 unknown :-1 s=CLOSED pgs=330 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:29:55.030 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:55.019+0000 7f243e442b80 1 -- 172.21.15.98:0/817111129 >> 172.21.15.98:0/817111129 conn(0x55d12cf81630 msgr2=0x55d12cf94d10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:29:55.030 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:55.019+0000 7f243e442b80 1 -- 172.21.15.98:0/817111129 shutdown_connections 2023-07-14T19:29:55.030 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:55.019+0000 7f243e442b80 1 -- 172.21.15.98:0/817111129 wait complete. 2023-07-14T19:29:55.030 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:55.019+0000 7f243e442b80 20 client.15084 trim_cache size 0 max 16384 2023-07-14T19:29:55.031 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2023-07-14T19:29:55.031 INFO:teuthology.orchestra.run.smithi098.stdout: "cluster_fsid": "9adea648-227b-11ee-9b33-001a4aab830c", 2023-07-14T19:29:55.031 INFO:teuthology.orchestra.run.smithi098.stdout: "whoami": 1, 2023-07-14T19:29:55.031 INFO:teuthology.orchestra.run.smithi098.stdout: "id": 24493, 2023-07-14T19:29:55.031 INFO:teuthology.orchestra.run.smithi098.stdout: "want_state": "up:active", 2023-07-14T19:29:55.031 INFO:teuthology.orchestra.run.smithi098.stdout: "state": "up:active", 2023-07-14T19:29:55.032 INFO:teuthology.orchestra.run.smithi098.stdout: "fs_name": "cephfs", 2023-07-14T19:29:55.032 INFO:teuthology.orchestra.run.smithi098.stdout: "rank_uptime": 50.359476280000003, 2023-07-14T19:29:55.032 INFO:teuthology.orchestra.run.smithi098.stdout: "mdsmap_epoch": 22, 2023-07-14T19:29:55.032 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap_epoch": 83, 2023-07-14T19:29:55.032 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap_epoch_barrier": 83, 2023-07-14T19:29:55.032 INFO:teuthology.orchestra.run.smithi098.stdout: "uptime": 66.284086501000004 2023-07-14T19:29:55.032 INFO:teuthology.orchestra.run.smithi098.stdout:} 2023-07-14T19:29:55.033 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:55.019+0000 7f24322aa700 1 -- 172.21.15.98:0/4258613513 >> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] conn(0x7f24180a1260 msgr2=0x7f24180a3710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:29:55.034 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:55.019+0000 7f24322aa700 1 --2- 172.21.15.98:0/4258613513 >> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] conn(0x7f24180a1260 0x7f24180a3710 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f241c00b2e0 tx=0x7f241c005fb0 comp rx=0 tx=0).stop 2023-07-14T19:29:55.034 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:55.019+0000 7f24322aa700 1 -- 172.21.15.98:0/4258613513 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f242c100690 msgr2=0x7f242c197ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:29:55.034 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:55.019+0000 7f24322aa700 1 --2- 172.21.15.98:0/4258613513 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f242c100690 0x7f242c197ae0 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7f241400d950 tx=0x7f241400dc60 comp rx=0 tx=0).stop 2023-07-14T19:29:55.034 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:55.020+0000 7f24322aa700 1 -- 172.21.15.98:0/4258613513 shutdown_connections 2023-07-14T19:29:55.034 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:55.020+0000 7f24322aa700 1 --2- 172.21.15.98:0/4258613513 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f242c1058d0 0x7f242c19f090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:29:55.034 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:55.020+0000 7f24322aa700 1 --2- 172.21.15.98:0/4258613513 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f242c100690 0x7f242c197ae0 unknown :-1 s=CLOSED pgs=326 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:29:55.035 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:55.020+0000 7f24322aa700 1 --2- 172.21.15.98:0/4258613513 >> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] conn(0x7f24180a1260 0x7f24180a3710 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:29:55.035 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:55.020+0000 7f24322aa700 1 --2- 172.21.15.98:0/4258613513 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f242c102fb0 0x7f242c198020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:29:55.035 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:55.020+0000 7f24322aa700 1 -- 172.21.15.98:0/4258613513 >> 172.21.15.98:0/4258613513 conn(0x7f242c0fa130 msgr2=0x7f242c0fed10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:29:55.035 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:55.020+0000 7f24322aa700 1 -- 172.21.15.98:0/4258613513 shutdown_connections 2023-07-14T19:29:55.035 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:55.020+0000 7f24322aa700 1 -- 172.21.15.98:0/4258613513 wait complete. 2023-07-14T19:29:55.035 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:29:54 smithi098 ceph-mon[127486]: pgmap v279: 97 pgs: 97 active+clean; 634 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2023-07-14T19:29:55.036 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:29:54 smithi098 ceph-mon[127486]: from='client.? 172.21.15.98:0/3051150914' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-07-14T19:29:55.036 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:29:54 smithi098 ceph-mon[127486]: from='client.? 172.21.15.98:0/2909637721' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-07-14T19:29:55.037 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.b status 2023-07-14T19:29:55.158 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:55.157+0000 7f0762379700 1 -- 172.21.15.98:0/294572437 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f075c100690 msgr2=0x7f075c106f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:29:55.158 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:55.157+0000 7f0762379700 1 --2- 172.21.15.98:0/294572437 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f075c100690 0x7f075c106f50 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7f075400b0a0 tx=0x7f075400b3b0 comp rx=0 tx=0).stop 2023-07-14T19:29:55.158 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:55.157+0000 7f0762379700 1 -- 172.21.15.98:0/294572437 shutdown_connections 2023-07-14T19:29:55.159 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:55.157+0000 7f0762379700 1 --2- 172.21.15.98:0/294572437 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f075c0feb10 0x7f075c0fef20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:29:55.159 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:55.157+0000 7f0762379700 1 --2- 172.21.15.98:0/294572437 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f075c100690 0x7f075c106f50 unknown :-1 s=CLOSED pgs=331 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:29:55.159 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:55.157+0000 7f0762379700 1 --2- 172.21.15.98:0/294572437 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f075c0ffd00 0x7f075c100150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:29:55.159 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:55.157+0000 7f0762379700 1 -- 172.21.15.98:0/294572437 >> 172.21.15.98:0/294572437 conn(0x7f075c0fa130 msgr2=0x7f075c0fc560 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:29:55.159 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:55.157+0000 7f0762379700 1 -- 172.21.15.98:0/294572437 shutdown_connections 2023-07-14T19:29:55.159 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:55.157+0000 7f0762379700 1 -- 172.21.15.98:0/294572437 wait complete. 2023-07-14T19:29:55.160 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:55.158+0000 7f0762379700 1 Processor -- start 2023-07-14T19:29:55.160 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:55.158+0000 7f0762379700 1 -- start start 2023-07-14T19:29:55.163 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:55.158+0000 7f0762379700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f075c0feb10 0x7f075c194890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:29:55.163 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:55.158+0000 7f0762379700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f075c0ffd00 0x7f075c194dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:29:55.164 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:55.158+0000 7f0762379700 1 --2- >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f075c100690 0x7f075c19be40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:29:55.164 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:55.158+0000 7f0762379700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f075c19c410 con 0x7f075c0feb10 2023-07-14T19:29:55.164 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:55.158+0000 7f0762379700 1 -- --> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f075c19c550 con 0x7f075c100690 2023-07-14T19:29:55.164 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:55.158+0000 7f0762379700 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f075c19c6e0 con 0x7f075c0ffd00 2023-07-14T19:29:55.164 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:55.158+0000 7f075bfff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f075c0feb10 0x7f075c194890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:29:55.164 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:55.158+0000 7f075bfff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f075c0feb10 0x7f075c194890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:46562/0 (socket says 172.21.15.98:46562) 2023-07-14T19:29:55.165 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:55.158+0000 7f075bfff700 1 -- 172.21.15.98:0/2939289279 learned_addr learned my addr 172.21.15.98:0/2939289279 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-07-14T19:29:55.165 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:55.158+0000 7f075b7fe700 1 --2- 172.21.15.98:0/2939289279 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f075c0ffd00 0x7f075c194dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:29:55.165 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:55.158+0000 7f0760916700 1 --2- 172.21.15.98:0/2939289279 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f075c100690 0x7f075c19be40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:29:55.165 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:55.158+0000 7f075bfff700 1 -- 172.21.15.98:0/2939289279 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f075c0ffd00 msgr2=0x7f075c194dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:29:55.165 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:55.158+0000 7f075bfff700 1 --2- 172.21.15.98:0/2939289279 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f075c0ffd00 0x7f075c194dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:29:55.165 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:55.158+0000 7f075bfff700 1 -- 172.21.15.98:0/2939289279 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f075c100690 msgr2=0x7f075c19be40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:29:55.165 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:55.158+0000 7f075bfff700 1 --2- 172.21.15.98:0/2939289279 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f075c100690 0x7f075c19be40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:29:55.165 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:55.158+0000 7f075bfff700 1 -- 172.21.15.98:0/2939289279 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0754009d00 con 0x7f075c0feb10 2023-07-14T19:29:55.166 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:55.158+0000 7f0760916700 1 --2- 172.21.15.98:0/2939289279 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f075c100690 0x7f075c19be40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-07-14T19:29:55.166 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:55.159+0000 7f075bfff700 1 --2- 172.21.15.98:0/2939289279 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f075c0feb10 0x7f075c194890 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7f074c00e970 tx=0x7f074c00ec80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:29:55.166 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:55.159+0000 7f07597fa700 1 -- 172.21.15.98:0/2939289279 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f074c009790 con 0x7f075c0feb10 2023-07-14T19:29:55.166 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:55.159+0000 7f07597fa700 1 -- 172.21.15.98:0/2939289279 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f074c00eec0 con 0x7f075c0feb10 2023-07-14T19:29:55.166 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:55.159+0000 7f0762379700 1 -- 172.21.15.98:0/2939289279 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f075c19c9c0 con 0x7f075c0feb10 2023-07-14T19:29:55.166 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:55.159+0000 7f07597fa700 1 -- 172.21.15.98:0/2939289279 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f074c003cf0 con 0x7f075c0feb10 2023-07-14T19:29:55.166 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:55.159+0000 7f0762379700 1 -- 172.21.15.98:0/2939289279 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f075c19ced0 con 0x7f075c0feb10 2023-07-14T19:29:55.167 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:55.161+0000 7f07597fa700 1 -- 172.21.15.98:0/2939289279 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 17) v1 ==== 135766+0+0 (secure 0 0 0) 0x7f074c00ccd0 con 0x7f075c0feb10 2023-07-14T19:29:55.167 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:55.161+0000 7f07597fa700 1 --2- 172.21.15.98:0/2939289279 >> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] conn(0x7f07440a11e0 0x7f07440a3690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:29:55.167 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:55.162+0000 7f075b7fe700 1 --2- 172.21.15.98:0/2939289279 >> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] conn(0x7f07440a11e0 0x7f07440a3690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:29:55.167 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:55.162+0000 7f07597fa700 1 -- 172.21.15.98:0/2939289279 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 7979+0+0 (secure 0 0 0) 0x7f074c014070 con 0x7f075c0feb10 2023-07-14T19:29:55.167 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:55.162+0000 7f075b7fe700 1 --2- 172.21.15.98:0/2939289279 >> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] conn(0x7f07440a11e0 0x7f07440a3690 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f075000ac50 tx=0x7f075000a380 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:29:55.168 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:55.162+0000 7f076e511b80 1 Processor -- start 2023-07-14T19:29:55.168 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:55.162+0000 7f076e511b80 1 -- start start 2023-07-14T19:29:55.168 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:55.162+0000 7f076e511b80 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x560bc70f2360 0x560bc70f2770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:29:55.169 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:55.162+0000 7f076e511b80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560bc70f0c40 0x560bc70f10b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:29:55.169 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:55.162+0000 7f076e511b80 1 --2- >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x560bc70f15f0 0x560bc70fd290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:29:55.169 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:55.162+0000 7f076e511b80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x560bc6ff5c60 con 0x560bc70f0c40 2023-07-14T19:29:55.169 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:55.162+0000 7f076e511b80 1 -- --> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] -- mon_getmap magic: 0 v1 -- 0x560bc70489f0 con 0x560bc70f15f0 2023-07-14T19:29:55.169 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:55.162+0000 7f076e511b80 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 v1 -- 0x560bc7041e80 con 0x560bc70f2360 2023-07-14T19:29:55.169 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:55.162+0000 7f075b7fe700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560bc70f0c40 0x560bc70f10b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:29:55.170 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:55.162+0000 7f075b7fe700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560bc70f0c40 0x560bc70f10b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:46572/0 (socket says 172.21.15.98:46572) 2023-07-14T19:29:55.170 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:55.162+0000 7f075b7fe700 1 -- 172.21.15.98:0/4061967842 learned_addr learned my addr 172.21.15.98:0/4061967842 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-07-14T19:29:55.170 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:55.163+0000 7f0760916700 1 --2- 172.21.15.98:0/4061967842 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x560bc70f15f0 0x560bc70fd290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:29:55.170 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:55.163+0000 7f075bfff700 1 --2- 172.21.15.98:0/4061967842 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x560bc70f2360 0x560bc70f2770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:29:55.170 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:55.163+0000 7f075b7fe700 1 -- 172.21.15.98:0/4061967842 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x560bc70f2360 msgr2=0x560bc70f2770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-14T19:29:55.171 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:55.163+0000 7f075b7fe700 1 --2- 172.21.15.98:0/4061967842 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x560bc70f2360 0x560bc70f2770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:29:55.171 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:55.163+0000 7f075b7fe700 1 -- 172.21.15.98:0/4061967842 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x560bc70f15f0 msgr2=0x560bc70fd290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-14T19:29:55.171 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:55.163+0000 7f075b7fe700 1 --2- 172.21.15.98:0/4061967842 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x560bc70f15f0 0x560bc70fd290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:29:55.171 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:55.163+0000 7f075b7fe700 1 -- 172.21.15.98:0/4061967842 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x560bc6e6fd90 con 0x560bc70f0c40 2023-07-14T19:29:55.171 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:55.163+0000 7f075b7fe700 1 --2- 172.21.15.98:0/4061967842 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560bc70f0c40 0x560bc70f10b0 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7f075000ff90 tx=0x7f0750012000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ab6d432f76f97743 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:29:55.171 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:55.163+0000 7f07427fc700 1 -- 172.21.15.98:0/4061967842 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0750003a90 con 0x560bc70f0c40 2023-07-14T19:29:55.171 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:55.163+0000 7f07427fc700 1 -- 172.21.15.98:0/4061967842 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f0750003bf0 con 0x560bc70f0c40 2023-07-14T19:29:55.172 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:55.163+0000 7f07427fc700 1 -- 172.21.15.98:0/4061967842 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f075001d660 con 0x560bc70f0c40 2023-07-14T19:29:55.175 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:55.164+0000 7f076e511b80 1 -- 172.21.15.98:0/4061967842 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560bc70f0c40 msgr2=0x560bc70f10b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:29:55.175 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:55.164+0000 7f076e511b80 1 --2- 172.21.15.98:0/4061967842 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560bc70f0c40 0x560bc70f10b0 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7f075000ff90 tx=0x7f0750012000 comp rx=0 tx=0).stop 2023-07-14T19:29:55.175 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:55.164+0000 7f076e511b80 1 -- 172.21.15.98:0/4061967842 shutdown_connections 2023-07-14T19:29:55.175 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:55.164+0000 7f076e511b80 1 --2- 172.21.15.98:0/4061967842 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x560bc70f2360 0x560bc70f2770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:29:55.175 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:55.164+0000 7f076e511b80 1 --2- 172.21.15.98:0/4061967842 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x560bc70f15f0 0x560bc70fd290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:29:55.176 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:55.164+0000 7f076e511b80 1 --2- 172.21.15.98:0/4061967842 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560bc70f0c40 0x560bc70f10b0 unknown :-1 s=CLOSED pgs=333 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:29:55.176 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:55.164+0000 7f076e511b80 1 -- 172.21.15.98:0/4061967842 >> 172.21.15.98:0/4061967842 conn(0x560bc70e6630 msgr2=0x560bc70eb980 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:29:55.176 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:55.164+0000 7f076e511b80 1 -- 172.21.15.98:0/4061967842 shutdown_connections 2023-07-14T19:29:55.176 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:55.164+0000 7f076e511b80 1 -- 172.21.15.98:0/4061967842 wait complete. 2023-07-14T19:29:55.176 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:55.164+0000 7f076e511b80 1 -- start start 2023-07-14T19:29:55.176 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:55.164+0000 7f076e511b80 1 Processor -- start 2023-07-14T19:29:55.177 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:55.164+0000 7f076e511b80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560bc70f0c40 0x560bc70fecf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:29:55.177 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:55.164+0000 7f076e511b80 1 --2- >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x560bc70f15f0 0x560bc70ff230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:29:55.177 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:55.165+0000 7f076e511b80 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x560bc70f2360 0x560bc7106270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:29:55.177 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:55.165+0000 7f076e511b80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x560bc6fe1a70 con 0x560bc70f0c40 2023-07-14T19:29:55.177 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:55.165+0000 7f076e511b80 1 -- --> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] -- mon_getmap magic: 0 v1 -- 0x560bc6fe1770 con 0x560bc70f15f0 2023-07-14T19:29:55.177 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:55.165+0000 7f076e511b80 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 v1 -- 0x560bc6ec58d0 con 0x560bc70f2360 2023-07-14T19:29:55.177 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:55.165+0000 7f076e511b80 10 client.0 _ll_register_callbacks cb 0x560bc6fdf710 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-07-14T19:29:55.177 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:55.165+0000 7f075bfff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560bc70f0c40 0x560bc70fecf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:29:55.178 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:55.165+0000 7f075bfff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560bc70f0c40 0x560bc70fecf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:46576/0 (socket says 172.21.15.98:46576) 2023-07-14T19:29:55.178 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:55.165+0000 7f075bfff700 1 -- 172.21.15.98:0/4102957919 learned_addr learned my addr 172.21.15.98:0/4102957919 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-07-14T19:29:55.178 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:55.165+0000 7f0760916700 1 --2- 172.21.15.98:0/4102957919 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x560bc70f2360 0x560bc7106270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:29:55.178 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:55.165+0000 7f075bfff700 1 -- 172.21.15.98:0/4102957919 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x560bc70f2360 msgr2=0x560bc7106270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-14T19:29:55.178 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:55.165+0000 7f075bfff700 1 --2- 172.21.15.98:0/4102957919 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x560bc70f2360 0x560bc7106270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:29:55.178 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:55.165+0000 7f075b7fe700 1 --2- 172.21.15.98:0/4102957919 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x560bc70f15f0 0x560bc70ff230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:29:55.178 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:55.165+0000 7f075bfff700 1 -- 172.21.15.98:0/4102957919 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x560bc70f15f0 msgr2=0x560bc70ff230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-14T19:29:55.179 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:55.165+0000 7f075bfff700 1 --2- 172.21.15.98:0/4102957919 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x560bc70f15f0 0x560bc70ff230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:29:55.179 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:55.165+0000 7f075bfff700 1 -- 172.21.15.98:0/4102957919 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f075000fc40 con 0x560bc70f0c40 2023-07-14T19:29:55.179 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:55.165+0000 7f075bfff700 1 --2- 172.21.15.98:0/4102957919 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560bc70f0c40 0x560bc70fecf0 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7f074c096a10 tx=0x7f074c09a050 comp rx=0 tx=0).ready entity=mon.0 client_cookie=36e8aeb270a892b8 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:29:55.179 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:55.165+0000 7f073b7fe700 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2023-07-14T19:29:55.179 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:55.165+0000 7f073b7fe700 1 -- 172.21.15.98:0/4102957919 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f074c0a23f0 con 0x560bc70f0c40 2023-07-14T19:29:55.179 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:55.165+0000 7f073b7fe700 1 -- 172.21.15.98:0/4102957919 <== mon.0 v2:172.21.15.98:3300/0 2 ==== osd_map(83..83 src has 1..83) v4 ==== 7979+0+0 (secure 0 0 0) 0x7f074c0a1f60 con 0x560bc70f0c40 2023-07-14T19:29:55.179 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:55.165+0000 7f076e511b80 1 -- 172.21.15.98:0/4102957919 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x560bc6f9b4a0 con 0x560bc70f0c40 2023-07-14T19:29:55.179 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:55.165+0000 7f073b7fe700 1 -- 172.21.15.98:0/4102957919 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=22) v2 ==== 24+0+0 (secure 0 0 0) 0x7f074c0a2120 con 0x560bc70f0c40 2023-07-14T19:29:55.180 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:55.165+0000 7f076e511b80 10 client.15096 fetch_fsmap learned FSMap version 22 2023-07-14T19:29:55.180 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:55.165+0000 7f076e511b80 1 -- 172.21.15.98:0/4102957919 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=22,monmap=4+}) v3 -- 0x560bc6e58c10 con 0x560bc70f0c40 2023-07-14T19:29:55.180 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:55.165+0000 7f073b7fe700 1 -- 172.21.15.98:0/4102957919 <== mon.0 v2:172.21.15.98:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x560bc6f9b4a0 con 0x560bc70f0c40 2023-07-14T19:29:55.180 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:55.166+0000 7f073b7fe700 1 -- 172.21.15.98:0/4102957919 <== mon.0 v2:172.21.15.98:3300/0 5 ==== fsmap(e 22) v1 ==== 7462+0+0 (secure 0 0 0) 0x7f074c098f00 con 0x560bc70f0c40 2023-07-14T19:29:55.180 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:55.166+0000 7f076e511b80 10 client.15096 fetch_fsmap finished waiting for FSMap version 22 2023-07-14T19:29:55.180 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:55.166+0000 7f076e511b80 10 client.15096 resolve_mds: resolved name 'b' to daemon mds.b 2023-07-14T19:29:55.181 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:55.166+0000 7f076e511b80 20 client.15096 populate_metadata read hostname 'smithi098' 2023-07-14T19:29:55.181 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:55.166+0000 7f076e511b80 1 --2- 172.21.15.98:0/4102957919 >> [v2:172.21.15.101:6838/48393381,v1:172.21.15.101:6839/48393381] conn(0x560bc710a060 0x560bc71093d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:29:55.181 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:55.166+0000 7f076e511b80 4 client.15096 mds_command: new command op to 24521 tid=0[{"prefix": "get_command_descriptions"}] 2023-07-14T19:29:55.181 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:55.166+0000 7f076e511b80 1 -- 172.21.15.98:0/4102957919 --> [v2:172.21.15.101:6838/48393381,v1:172.21.15.101:6839/48393381] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x560bc70bfd50 con 0x560bc710a060 2023-07-14T19:29:55.182 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:55.166+0000 7f075b7fe700 1 --2- 172.21.15.98:0/4102957919 >> [v2:172.21.15.101:6838/48393381,v1:172.21.15.101:6839/48393381] conn(0x560bc710a060 0x560bc71093d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:29:55.182 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:55.166+0000 7f073b7fe700 10 client.15096 ms_handle_connect on v2:172.21.15.101:6838/48393381 2023-07-14T19:29:55.182 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:55.166+0000 7f075b7fe700 1 --2- 172.21.15.98:0/4102957919 >> [v2:172.21.15.101:6838/48393381,v1:172.21.15.101:6839/48393381] conn(0x560bc710a060 0x560bc71093d0 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=941d00be27c5d464 server_cookie=2829195323d7777c in_seq=0 out_seq=0 2023-07-14T19:29:55.182 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:55.167+0000 7f073b7fe700 1 -- 172.21.15.98:0/4102957919 <== mds.0 v2:172.21.15.101:6838/48393381 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18017 (crc 0 0 0) 0x560bc70bfd50 con 0x560bc710a060 2023-07-14T19:29:55.182 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:55.167+0000 7f073b7fe700 10 client.15096 handle_command_reply: tid=0 2023-07-14T19:29:55.183 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:55.167+0000 7f076e511b80 1 client.15096 shutdown 2023-07-14T19:29:55.183 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:55.167+0000 7f076e511b80 1 -- 172.21.15.98:0/4102957919 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560bc70f0c40 msgr2=0x560bc70fecf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:29:55.183 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:55.167+0000 7f076e511b80 1 --2- 172.21.15.98:0/4102957919 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560bc70f0c40 0x560bc70fecf0 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7f074c096a10 tx=0x7f074c09a050 comp rx=0 tx=0).stop 2023-07-14T19:29:55.183 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:55.167+0000 7f076e511b80 1 -- 172.21.15.98:0/4102957919 shutdown_connections 2023-07-14T19:29:55.183 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:55.167+0000 7f076e511b80 1 --2- 172.21.15.98:0/4102957919 >> [v2:172.21.15.101:6838/48393381,v1:172.21.15.101:6839/48393381] conn(0x560bc710a060 0x560bc71093d0 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:29:55.183 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:55.167+0000 7f076e511b80 1 --2- 172.21.15.98:0/4102957919 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x560bc70f2360 0x560bc7106270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:29:55.183 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:55.167+0000 7f076e511b80 1 --2- 172.21.15.98:0/4102957919 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560bc70f0c40 0x560bc70fecf0 unknown :-1 s=CLOSED pgs=334 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:29:55.184 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:55.167+0000 7f073b7fe700 0 client.15096 ms_handle_reset on v2:172.21.15.101:6838/48393381 2023-07-14T19:29:55.184 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:55.167+0000 7f076e511b80 1 --2- 172.21.15.98:0/4102957919 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x560bc70f15f0 0x560bc70ff230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:29:55.184 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:55.167+0000 7f076e511b80 1 -- 172.21.15.98:0/4102957919 >> 172.21.15.98:0/4102957919 conn(0x560bc70e6630 msgr2=0x560bc70ea600 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:29:55.184 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:55.167+0000 7f076e511b80 1 -- 172.21.15.98:0/4102957919 shutdown_connections 2023-07-14T19:29:55.184 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:55.167+0000 7f076e511b80 1 -- 172.21.15.98:0/4102957919 wait complete. 2023-07-14T19:29:55.184 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:55.167+0000 7f076e511b80 20 client.15096 trim_cache size 0 max 16384 2023-07-14T19:29:55.186 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:55.185+0000 7f076e511b80 1 Processor -- start 2023-07-14T19:29:55.186 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:55.185+0000 7f076e511b80 1 -- start start 2023-07-14T19:29:55.186 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:55.185+0000 7f076e511b80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560bc70f0c40 0x560bc70fcde0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:29:55.186 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:55.185+0000 7f076e511b80 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x560bc70f15f0 0x560bc7106230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:29:55.187 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:55.185+0000 7f076e511b80 1 --2- >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x560bc70f2360 0x560bc70fd320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:29:55.187 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:55.185+0000 7f076e511b80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x560bc70bfd50 con 0x560bc70f0c40 2023-07-14T19:29:55.187 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:55.185+0000 7f076e511b80 1 -- --> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] -- mon_getmap magic: 0 v1 -- 0x560bc6f9b4a0 con 0x560bc70f2360 2023-07-14T19:29:55.187 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:55.185+0000 7f076e511b80 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 v1 -- 0x560bc6feb470 con 0x560bc70f15f0 2023-07-14T19:29:55.187 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:55.185+0000 7f075bfff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560bc70f0c40 0x560bc70fcde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:29:55.187 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:55.185+0000 7f075bfff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560bc70f0c40 0x560bc70fcde0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:46584/0 (socket says 172.21.15.98:46584) 2023-07-14T19:29:55.188 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:55.185+0000 7f075bfff700 1 -- 172.21.15.98:0/154279683 learned_addr learned my addr 172.21.15.98:0/154279683 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-07-14T19:29:55.188 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:55.185+0000 7f0760916700 1 --2- 172.21.15.98:0/154279683 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x560bc70f2360 0x560bc70fd320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:29:55.188 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:55.185+0000 7f075bfff700 1 -- 172.21.15.98:0/154279683 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x560bc70f15f0 msgr2=0x560bc7106230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-14T19:29:55.188 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:55.185+0000 7f075b7fe700 1 --2- 172.21.15.98:0/154279683 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x560bc70f15f0 0x560bc7106230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:29:55.188 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:55.185+0000 7f075bfff700 1 --2- 172.21.15.98:0/154279683 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x560bc70f15f0 0x560bc7106230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:29:55.188 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:55.185+0000 7f075bfff700 1 -- 172.21.15.98:0/154279683 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x560bc70f2360 msgr2=0x560bc70fd320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-14T19:29:55.189 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:55.185+0000 7f075bfff700 1 --2- 172.21.15.98:0/154279683 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x560bc70f2360 0x560bc70fd320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:29:55.189 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:55.185+0000 7f075bfff700 1 -- 172.21.15.98:0/154279683 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x560bc6e96200 con 0x560bc70f0c40 2023-07-14T19:29:55.189 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:55.185+0000 7f0760916700 1 --2- 172.21.15.98:0/154279683 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x560bc70f2360 0x560bc70fd320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-07-14T19:29:55.189 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:55.185+0000 7f075b7fe700 1 --2- 172.21.15.98:0/154279683 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x560bc70f15f0 0x560bc7106230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-07-14T19:29:55.189 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:55.185+0000 7f075bfff700 1 --2- 172.21.15.98:0/154279683 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560bc70f0c40 0x560bc70fcde0 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7f074c005950 tx=0x7f074c0ca5e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d0c82e2ec982cd3c server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:29:55.189 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:55.186+0000 7f0742ffd700 1 -- 172.21.15.98:0/154279683 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f074c0a23f0 con 0x560bc70f0c40 2023-07-14T19:29:55.189 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:55.186+0000 7f0742ffd700 1 -- 172.21.15.98:0/154279683 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f074c0b80d0 con 0x560bc70f0c40 2023-07-14T19:29:55.189 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:55.186+0000 7f0742ffd700 1 -- 172.21.15.98:0/154279683 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f074c094bb0 con 0x560bc70f0c40 2023-07-14T19:29:55.193 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:55.186+0000 7f076e511b80 1 -- 172.21.15.98:0/154279683 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560bc70f0c40 msgr2=0x560bc70fcde0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:29:55.194 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:55.186+0000 7f076e511b80 1 --2- 172.21.15.98:0/154279683 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560bc70f0c40 0x560bc70fcde0 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7f074c005950 tx=0x7f074c0ca5e0 comp rx=0 tx=0).stop 2023-07-14T19:29:55.194 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:55.186+0000 7f076e511b80 1 -- 172.21.15.98:0/154279683 shutdown_connections 2023-07-14T19:29:55.194 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:55.186+0000 7f076e511b80 1 --2- 172.21.15.98:0/154279683 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x560bc70f2360 0x560bc70fd320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:29:55.194 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:55.186+0000 7f076e511b80 1 --2- 172.21.15.98:0/154279683 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560bc70f0c40 0x560bc70fcde0 unknown :-1 s=CLOSED pgs=335 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:29:55.194 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:55.186+0000 7f076e511b80 1 --2- 172.21.15.98:0/154279683 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x560bc70f15f0 0x560bc7106230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:29:55.194 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:55.186+0000 7f076e511b80 1 -- 172.21.15.98:0/154279683 >> 172.21.15.98:0/154279683 conn(0x560bc70e6630 msgr2=0x560bc70f9910 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:29:55.194 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:55.186+0000 7f076e511b80 1 -- 172.21.15.98:0/154279683 shutdown_connections 2023-07-14T19:29:55.195 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:55.186+0000 7f076e511b80 1 -- 172.21.15.98:0/154279683 wait complete. 2023-07-14T19:29:55.195 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:55.186+0000 7f076e511b80 1 -- start start 2023-07-14T19:29:55.195 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:55.187+0000 7f076e511b80 1 Processor -- start 2023-07-14T19:29:55.195 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:55.187+0000 7f076e511b80 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x560bc70f0c40 0x560bc7103520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:29:55.196 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:55.187+0000 7f076e511b80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560bc70f15f0 0x560bc7103a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:29:55.196 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:55.187+0000 7f076e511b80 1 --2- >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x560bc70f2360 0x560bc7104a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:29:55.196 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:55.187+0000 7f076e511b80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x560bc7104f40 con 0x560bc70f15f0 2023-07-14T19:29:55.196 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:55.187+0000 7f076e511b80 1 -- --> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] -- mon_getmap magic: 0 v1 -- 0x560bc71050b0 con 0x560bc70f2360 2023-07-14T19:29:55.197 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:55.187+0000 7f076e511b80 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 v1 -- 0x560bc7105270 con 0x560bc70f0c40 2023-07-14T19:29:55.197 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:55.187+0000 7f076e511b80 10 client.0 _ll_register_callbacks cb 0x7f074c099900 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-07-14T19:29:55.197 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:55.187+0000 7f075b7fe700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560bc70f15f0 0x560bc7103a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:29:55.197 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:55.187+0000 7f075b7fe700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560bc70f15f0 0x560bc7103a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:46592/0 (socket says 172.21.15.98:46592) 2023-07-14T19:29:55.198 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:55.187+0000 7f075b7fe700 1 -- 172.21.15.98:0/4226179038 learned_addr learned my addr 172.21.15.98:0/4226179038 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-07-14T19:29:55.198 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:55.187+0000 7f075bfff700 1 --2- 172.21.15.98:0/4226179038 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x560bc70f0c40 0x560bc7103520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:29:55.198 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:55.187+0000 7f0760916700 1 --2- 172.21.15.98:0/4226179038 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x560bc70f2360 0x560bc7104a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:29:55.199 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:55.187+0000 7f075b7fe700 1 -- 172.21.15.98:0/4226179038 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x560bc70f0c40 msgr2=0x560bc7103520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-14T19:29:55.199 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:55.187+0000 7f075b7fe700 1 --2- 172.21.15.98:0/4226179038 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x560bc70f0c40 0x560bc7103520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:29:55.199 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:55.187+0000 7f075b7fe700 1 -- 172.21.15.98:0/4226179038 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x560bc70f2360 msgr2=0x560bc7104a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-14T19:29:55.199 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:55.187+0000 7f075b7fe700 1 --2- 172.21.15.98:0/4226179038 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x560bc70f2360 0x560bc7104a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:29:55.200 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:55.187+0000 7f075b7fe700 1 -- 172.21.15.98:0/4226179038 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x560bc6e58c10 con 0x560bc70f15f0 2023-07-14T19:29:55.200 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:55.187+0000 7f075bfff700 1 --2- 172.21.15.98:0/4226179038 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x560bc70f0c40 0x560bc7103520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-07-14T19:29:55.200 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:55.187+0000 7f075b7fe700 1 --2- 172.21.15.98:0/4226179038 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560bc70f15f0 0x560bc7103a60 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7f075000ff90 tx=0x7f0750003de0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3844a4d9e433f208 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:29:55.200 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:55.187+0000 7f073b7fe700 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2023-07-14T19:29:55.201 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:55.188+0000 7f073b7fe700 1 -- 172.21.15.98:0/4226179038 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f075002b040 con 0x560bc70f15f0 2023-07-14T19:29:55.201 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:55.188+0000 7f073b7fe700 1 -- 172.21.15.98:0/4226179038 <== mon.0 v2:172.21.15.98:3300/0 2 ==== osd_map(83..83 src has 1..83) v4 ==== 7979+0+0 (secure 0 0 0) 0x7f0750005040 con 0x560bc70f15f0 2023-07-14T19:29:55.201 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:55.188+0000 7f076e511b80 1 -- 172.21.15.98:0/4226179038 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f0750003bf0 con 0x560bc70f15f0 2023-07-14T19:29:55.201 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:55.188+0000 7f073b7fe700 1 -- 172.21.15.98:0/4226179038 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=22) v2 ==== 24+0+0 (secure 0 0 0) 0x7f075002f670 con 0x560bc70f15f0 2023-07-14T19:29:55.201 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:55.188+0000 7f076e511b80 10 client.15102 fetch_fsmap learned FSMap version 22 2023-07-14T19:29:55.202 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:55.188+0000 7f076e511b80 1 -- 172.21.15.98:0/4226179038 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=22,monmap=4+}) v3 -- 0x560bc6e6e260 con 0x560bc70f15f0 2023-07-14T19:29:55.202 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:55.188+0000 7f073b7fe700 1 -- 172.21.15.98:0/4226179038 <== mon.0 v2:172.21.15.98:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f0750003bf0 con 0x560bc70f15f0 2023-07-14T19:29:55.202 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:55.188+0000 7f073b7fe700 1 -- 172.21.15.98:0/4226179038 <== mon.0 v2:172.21.15.98:3300/0 5 ==== fsmap(e 22) v1 ==== 7462+0+0 (secure 0 0 0) 0x7f075002e030 con 0x560bc70f15f0 2023-07-14T19:29:55.202 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:55.188+0000 7f076e511b80 10 client.15102 fetch_fsmap finished waiting for FSMap version 22 2023-07-14T19:29:55.202 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:55.188+0000 7f076e511b80 10 client.15102 resolve_mds: resolved name 'b' to daemon mds.b 2023-07-14T19:29:55.203 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:55.188+0000 7f076e511b80 20 client.15102 populate_metadata read hostname 'smithi098' 2023-07-14T19:29:55.203 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:55.188+0000 7f076e511b80 1 --2- 172.21.15.98:0/4226179038 >> [v2:172.21.15.101:6838/48393381,v1:172.21.15.101:6839/48393381] conn(0x560bc71633d0 0x560bc715fd60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:29:55.203 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:55.188+0000 7f076e511b80 4 client.15102 mds_command: new command op to 24521 tid=0[{"prefix": "status"}] 2023-07-14T19:29:55.203 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:55.188+0000 7f076e511b80 1 -- 172.21.15.98:0/4226179038 --> [v2:172.21.15.101:6838/48393381,v1:172.21.15.101:6839/48393381] -- command(tid 0: {"prefix": "status"}) v1 -- 0x560bc703ae60 con 0x560bc71633d0 2023-07-14T19:29:55.203 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:55.188+0000 7f075bfff700 1 --2- 172.21.15.98:0/4226179038 >> [v2:172.21.15.101:6838/48393381,v1:172.21.15.101:6839/48393381] conn(0x560bc71633d0 0x560bc715fd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:29:55.203 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:55.189+0000 7f075bfff700 1 --2- 172.21.15.98:0/4226179038 >> [v2:172.21.15.101:6838/48393381,v1:172.21.15.101:6839/48393381] conn(0x560bc71633d0 0x560bc715fd60 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9da7d4d87470bd57 server_cookie=613f5c2d887d99a6 in_seq=0 out_seq=0 2023-07-14T19:29:55.203 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:55.189+0000 7f073b7fe700 10 client.15102 ms_handle_connect on v2:172.21.15.101:6838/48393381 2023-07-14T19:29:55.203 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:55.189+0000 7f073b7fe700 1 -- 172.21.15.98:0/4226179038 <== mds.0 v2:172.21.15.101:6838/48393381 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+332 (crc 0 0 0) 0x560bc703ae60 con 0x560bc71633d0 2023-07-14T19:29:55.204 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:55.189+0000 7f073b7fe700 10 client.15102 handle_command_reply: tid=0 2023-07-14T19:29:55.204 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:55.189+0000 7f076e511b80 1 client.15102 shutdown 2023-07-14T19:29:55.204 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:55.189+0000 7f076e511b80 1 -- 172.21.15.98:0/4226179038 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560bc70f15f0 msgr2=0x560bc7103a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:29:55.204 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:55.189+0000 7f076e511b80 1 --2- 172.21.15.98:0/4226179038 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560bc70f15f0 0x560bc7103a60 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7f075000ff90 tx=0x7f0750003de0 comp rx=0 tx=0).stop 2023-07-14T19:29:55.204 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:55.189+0000 7f076e511b80 1 -- 172.21.15.98:0/4226179038 shutdown_connections 2023-07-14T19:29:55.204 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:55.189+0000 7f076e511b80 1 --2- 172.21.15.98:0/4226179038 >> [v2:172.21.15.101:6838/48393381,v1:172.21.15.101:6839/48393381] conn(0x560bc71633d0 0x560bc715fd60 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:29:55.204 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:55.189+0000 7f076e511b80 1 --2- 172.21.15.98:0/4226179038 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x560bc70f0c40 0x560bc7103520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:29:55.205 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:55.189+0000 7f073b7fe700 0 client.15102 ms_handle_reset on v2:172.21.15.101:6838/48393381 2023-07-14T19:29:55.205 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:55.189+0000 7f076e511b80 1 --2- 172.21.15.98:0/4226179038 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x560bc70f2360 0x560bc7104a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:29:55.205 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:55.189+0000 7f076e511b80 1 --2- 172.21.15.98:0/4226179038 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560bc70f15f0 0x560bc7103a60 unknown :-1 s=CLOSED pgs=336 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:29:55.205 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:55.189+0000 7f076e511b80 1 -- 172.21.15.98:0/4226179038 >> 172.21.15.98:0/4226179038 conn(0x560bc70e6630 msgr2=0x560bc70fcde0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:29:55.205 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:55.189+0000 7f076e511b80 1 -- 172.21.15.98:0/4226179038 shutdown_connections 2023-07-14T19:29:55.205 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:55.189+0000 7f076e511b80 1 -- 172.21.15.98:0/4226179038 wait complete. 2023-07-14T19:29:55.205 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:55.189+0000 7f076e511b80 20 client.15102 trim_cache size 0 max 16384 2023-07-14T19:29:55.206 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2023-07-14T19:29:55.206 INFO:teuthology.orchestra.run.smithi098.stdout: "cluster_fsid": "9adea648-227b-11ee-9b33-001a4aab830c", 2023-07-14T19:29:55.206 INFO:teuthology.orchestra.run.smithi098.stdout: "whoami": 0, 2023-07-14T19:29:55.206 INFO:teuthology.orchestra.run.smithi098.stdout: "id": 24521, 2023-07-14T19:29:55.206 INFO:teuthology.orchestra.run.smithi098.stdout: "want_state": "up:active", 2023-07-14T19:29:55.207 INFO:teuthology.orchestra.run.smithi098.stdout: "state": "up:active", 2023-07-14T19:29:55.207 INFO:teuthology.orchestra.run.smithi098.stdout: "fs_name": "cephfs", 2023-07-14T19:29:55.207 INFO:teuthology.orchestra.run.smithi098.stdout: "rank_uptime": 53.558796471999997, 2023-07-14T19:29:55.207 INFO:teuthology.orchestra.run.smithi098.stdout: "mdsmap_epoch": 22, 2023-07-14T19:29:55.207 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap_epoch": 83, 2023-07-14T19:29:55.207 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap_epoch_barrier": 83, 2023-07-14T19:29:55.207 INFO:teuthology.orchestra.run.smithi098.stdout: "uptime": 59.896651955999999 2023-07-14T19:29:55.208 INFO:teuthology.orchestra.run.smithi098.stdout:} 2023-07-14T19:29:55.209 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:55.190+0000 7f0762379700 1 -- 172.21.15.98:0/2939289279 >> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] conn(0x7f07440a11e0 msgr2=0x7f07440a3690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:29:55.209 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:55.190+0000 7f0762379700 1 --2- 172.21.15.98:0/2939289279 >> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] conn(0x7f07440a11e0 0x7f07440a3690 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f075000ac50 tx=0x7f075000a380 comp rx=0 tx=0).stop 2023-07-14T19:29:55.209 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:55.190+0000 7f0762379700 1 -- 172.21.15.98:0/2939289279 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f075c0feb10 msgr2=0x7f075c194890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:29:55.209 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:55.190+0000 7f0762379700 1 --2- 172.21.15.98:0/2939289279 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f075c0feb10 0x7f075c194890 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7f074c00e970 tx=0x7f074c00ec80 comp rx=0 tx=0).stop 2023-07-14T19:29:55.209 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:55.190+0000 7f0762379700 1 -- 172.21.15.98:0/2939289279 shutdown_connections 2023-07-14T19:29:55.210 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:55.190+0000 7f0762379700 1 --2- 172.21.15.98:0/2939289279 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f075c100690 0x7f075c19be40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:29:55.210 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:55.190+0000 7f0762379700 1 --2- 172.21.15.98:0/2939289279 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f075c0feb10 0x7f075c194890 unknown :-1 s=CLOSED pgs=332 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:29:55.210 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:55.190+0000 7f0762379700 1 --2- 172.21.15.98:0/2939289279 >> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] conn(0x7f07440a11e0 0x7f07440a3690 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:29:55.210 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:55.190+0000 7f0762379700 1 --2- 172.21.15.98:0/2939289279 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f075c0ffd00 0x7f075c194dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:29:55.210 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:55.190+0000 7f0762379700 1 -- 172.21.15.98:0/2939289279 >> 172.21.15.98:0/2939289279 conn(0x7f075c0fa130 msgr2=0x7f075c0fc2a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:29:55.210 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:55.190+0000 7f0762379700 1 -- 172.21.15.98:0/2939289279 shutdown_connections 2023-07-14T19:29:55.210 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:55.190+0000 7f0762379700 1 -- 172.21.15.98:0/2939289279 wait complete. 2023-07-14T19:29:55.211 INFO:tasks.cephadm:Setup complete, yielding 2023-07-14T19:29:55.211 INFO:teuthology.run_tasks:Running task cephadm.shell... 2023-07-14T19:29:55.221 INFO:tasks.cephadm:Running commands on role mon.a host ubuntu@smithi098.front.sepia.ceph.com 2023-07-14T19:29:55.221 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81c9387d2e7576da065747ec9c16d1c7d69016ec shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 9adea648-227b-11ee-9b33-001a4aab830c -- bash -c 'ceph orch status' 2023-07-14T19:29:55.769 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/9adea648-227b-11ee-9b33-001a4aab830c/mon.a/config 2023-07-14T19:29:55.941 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:29:55 smithi101 ceph-mon[128807]: from='client.? 172.21.15.98:0/3625700843' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-07-14T19:29:55.961 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:29:55 smithi098 ceph-mon[127486]: from='client.? 172.21.15.98:0/3625700843' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-07-14T19:29:55.970 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:29:55 smithi139 ceph-mon[129293]: from='client.? 172.21.15.98:0/3625700843' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-07-14T19:29:56.929 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:56.927+0000 7eff63f73700 1 -- 172.21.15.98:0/1490461184 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7eff5c100740 msgr2=0x7eff5c107000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:29:56.929 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:56.927+0000 7eff63f73700 1 --2- 172.21.15.98:0/1490461184 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7eff5c100740 0x7eff5c107000 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7eff5400b0a0 tx=0x7eff5400b3b0 comp rx=0 tx=0).stop 2023-07-14T19:29:56.929 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:56.927+0000 7eff63f73700 1 -- 172.21.15.98:0/1490461184 shutdown_connections 2023-07-14T19:29:56.930 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:56.927+0000 7eff63f73700 1 --2- 172.21.15.98:0/1490461184 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7eff5c100740 0x7eff5c107000 unknown :-1 s=CLOSED pgs=337 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:29:56.930 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:56.927+0000 7eff63f73700 1 --2- 172.21.15.98:0/1490461184 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7eff5c0febb0 0x7eff5c0fefc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:29:56.930 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:56.927+0000 7eff63f73700 1 --2- 172.21.15.98:0/1490461184 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7eff5c0ffdb0 0x7eff5c100200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:29:56.930 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:56.927+0000 7eff63f73700 1 -- 172.21.15.98:0/1490461184 >> 172.21.15.98:0/1490461184 conn(0x7eff5c0fa1e0 msgr2=0x7eff5c0fc630 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:29:56.930 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:56.927+0000 7eff63f73700 1 -- 172.21.15.98:0/1490461184 shutdown_connections 2023-07-14T19:29:56.930 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:56.927+0000 7eff63f73700 1 -- 172.21.15.98:0/1490461184 wait complete. 2023-07-14T19:29:56.930 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:56.927+0000 7eff63f73700 1 Processor -- start 2023-07-14T19:29:56.931 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:56.928+0000 7eff63f73700 1 -- start start 2023-07-14T19:29:56.931 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:56.928+0000 7eff63f73700 1 --2- >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7eff5c0febb0 0x7eff5c068c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:29:56.931 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:56.928+0000 7eff63f73700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7eff5c0ffdb0 0x7eff5c19cbf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:29:56.931 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:56.928+0000 7eff63f73700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7eff5c100740 0x7eff5c19f140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:29:56.931 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:56.928+0000 7eff63f73700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7eff5c19f7f0 con 0x7eff5c100740 2023-07-14T19:29:56.931 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:56.928+0000 7eff63f73700 1 -- --> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] -- mon_getmap magic: 0 v1 -- 0x7eff5c19f960 con 0x7eff5c0febb0 2023-07-14T19:29:56.931 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:56.928+0000 7eff63f73700 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 v1 -- 0x7eff5c19fb20 con 0x7eff5c0ffdb0 2023-07-14T19:29:56.932 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:56.928+0000 7eff62510700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7eff5c100740 0x7eff5c19f140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:29:56.932 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:56.928+0000 7eff62510700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7eff5c100740 0x7eff5c19f140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:46608/0 (socket says 172.21.15.98:46608) 2023-07-14T19:29:56.932 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:56.928+0000 7eff62510700 1 -- 172.21.15.98:0/3860061355 learned_addr learned my addr 172.21.15.98:0/3860061355 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-07-14T19:29:56.932 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:56.928+0000 7eff61d0f700 1 --2- 172.21.15.98:0/3860061355 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7eff5c0febb0 0x7eff5c068c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:29:56.932 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:56.928+0000 7eff6150e700 1 --2- 172.21.15.98:0/3860061355 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7eff5c0ffdb0 0x7eff5c19cbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:29:56.932 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:56.928+0000 7eff62510700 1 -- 172.21.15.98:0/3860061355 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7eff5c0ffdb0 msgr2=0x7eff5c19cbf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:29:56.932 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:56.928+0000 7eff62510700 1 --2- 172.21.15.98:0/3860061355 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7eff5c0ffdb0 0x7eff5c19cbf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:29:56.933 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:56.928+0000 7eff62510700 1 -- 172.21.15.98:0/3860061355 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7eff5c0febb0 msgr2=0x7eff5c068c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:29:56.933 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:56.928+0000 7eff62510700 1 --2- 172.21.15.98:0/3860061355 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7eff5c0febb0 0x7eff5c068c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:29:56.933 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:56.928+0000 7eff62510700 1 -- 172.21.15.98:0/3860061355 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7eff54009d00 con 0x7eff5c100740 2023-07-14T19:29:56.933 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:56.928+0000 7eff6150e700 1 --2- 172.21.15.98:0/3860061355 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7eff5c0ffdb0 0x7eff5c19cbf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-07-14T19:29:56.933 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:56.928+0000 7eff62510700 1 --2- 172.21.15.98:0/3860061355 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7eff5c100740 0x7eff5c19f140 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7eff540062a0 tx=0x7eff54004db0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:29:56.933 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:56.928+0000 7eff52ffd700 1 -- 172.21.15.98:0/3860061355 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7eff5400e040 con 0x7eff5c100740 2023-07-14T19:29:56.933 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:56.928+0000 7eff52ffd700 1 -- 172.21.15.98:0/3860061355 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7eff5401db50 con 0x7eff5c100740 2023-07-14T19:29:56.933 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:56.928+0000 7eff63f73700 1 -- 172.21.15.98:0/3860061355 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7eff5c19fda0 con 0x7eff5c100740 2023-07-14T19:29:56.934 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:56.929+0000 7eff52ffd700 1 -- 172.21.15.98:0/3860061355 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7eff5401bc40 con 0x7eff5c100740 2023-07-14T19:29:56.934 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:56.929+0000 7eff63f73700 1 -- 172.21.15.98:0/3860061355 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7eff5c1a0210 con 0x7eff5c100740 2023-07-14T19:29:56.935 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:56.929+0000 7eff63f73700 1 -- 172.21.15.98:0/3860061355 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7eff5c106150 con 0x7eff5c100740 2023-07-14T19:29:56.935 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:56.931+0000 7eff52ffd700 1 -- 172.21.15.98:0/3860061355 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 17) v1 ==== 135766+0+0 (secure 0 0 0) 0x7eff5401d690 con 0x7eff5c100740 2023-07-14T19:29:56.935 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:56.931+0000 7eff52ffd700 1 --2- 172.21.15.98:0/3860061355 >> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] conn(0x7eff480a11e0 0x7eff480a3690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:29:56.935 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:56.932+0000 7eff61d0f700 1 --2- 172.21.15.98:0/3860061355 >> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] conn(0x7eff480a11e0 0x7eff480a3690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:29:56.935 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:56.932+0000 7eff52ffd700 1 -- 172.21.15.98:0/3860061355 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 7979+0+0 (secure 0 0 0) 0x7eff540cec10 con 0x7eff5c100740 2023-07-14T19:29:56.935 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:56.932+0000 7eff61d0f700 1 --2- 172.21.15.98:0/3860061355 >> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] conn(0x7eff480a11e0 0x7eff480a3690 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7eff4c005fd0 tx=0x7eff4c005dd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:29:56.936 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:56.934+0000 7eff52ffd700 1 -- 172.21.15.98:0/3860061355 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7eff54098440 con 0x7eff5c100740 2023-07-14T19:29:56.941 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:29:56 smithi101 ceph-mon[128807]: pgmap v280: 97 pgs: 97 active+clean; 634 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2023-07-14T19:29:56.970 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:29:56 smithi139 ceph-mon[129293]: pgmap v280: 97 pgs: 97 active+clean; 634 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2023-07-14T19:29:57.031 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:29:56 smithi098 ceph-mon[127486]: pgmap v280: 97 pgs: 97 active+clean; 634 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2023-07-14T19:29:57.136 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:57.136+0000 7eff63f73700 1 -- 172.21.15.98:0/3860061355 --> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] -- mgr_command(tid 0: {"prefix": "orch status", "target": ["mon-mgr", ""]}) v1 -- 0x7eff5c060a80 con 0x7eff480a11e0 2023-07-14T19:29:57.137 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:57.137+0000 7eff52ffd700 1 -- 172.21.15.98:0/3860061355 <== mgr.14152 v2:172.21.15.98:6800/452042164 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+43 (secure 0 0 0) 0x7eff5c060a80 con 0x7eff480a11e0 2023-07-14T19:29:57.137 INFO:teuthology.orchestra.run.smithi098.stdout:Backend: cephadm 2023-07-14T19:29:57.138 INFO:teuthology.orchestra.run.smithi098.stdout:Available: Yes 2023-07-14T19:29:57.138 INFO:teuthology.orchestra.run.smithi098.stdout:Paused: No 2023-07-14T19:29:57.140 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:57.139+0000 7eff63f73700 1 -- 172.21.15.98:0/3860061355 >> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] conn(0x7eff480a11e0 msgr2=0x7eff480a3690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:29:57.140 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:57.139+0000 7eff63f73700 1 --2- 172.21.15.98:0/3860061355 >> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] conn(0x7eff480a11e0 0x7eff480a3690 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7eff4c005fd0 tx=0x7eff4c005dd0 comp rx=0 tx=0).stop 2023-07-14T19:29:57.140 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:57.139+0000 7eff63f73700 1 -- 172.21.15.98:0/3860061355 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7eff5c100740 msgr2=0x7eff5c19f140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:29:57.141 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:57.139+0000 7eff63f73700 1 --2- 172.21.15.98:0/3860061355 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7eff5c100740 0x7eff5c19f140 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7eff540062a0 tx=0x7eff54004db0 comp rx=0 tx=0).stop 2023-07-14T19:29:57.141 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:57.139+0000 7eff63f73700 1 -- 172.21.15.98:0/3860061355 shutdown_connections 2023-07-14T19:29:57.141 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:57.139+0000 7eff63f73700 1 --2- 172.21.15.98:0/3860061355 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7eff5c100740 0x7eff5c19f140 unknown :-1 s=CLOSED pgs=338 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:29:57.141 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:57.139+0000 7eff63f73700 1 --2- 172.21.15.98:0/3860061355 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7eff5c0febb0 0x7eff5c068c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:29:57.142 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:57.139+0000 7eff63f73700 1 --2- 172.21.15.98:0/3860061355 >> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] conn(0x7eff480a11e0 0x7eff480a3690 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:29:57.142 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:57.139+0000 7eff63f73700 1 --2- 172.21.15.98:0/3860061355 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7eff5c0ffdb0 0x7eff5c19cbf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:29:57.142 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:57.139+0000 7eff63f73700 1 -- 172.21.15.98:0/3860061355 >> 172.21.15.98:0/3860061355 conn(0x7eff5c0fa1e0 msgr2=0x7eff5c103290 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:29:57.142 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:57.139+0000 7eff63f73700 1 -- 172.21.15.98:0/3860061355 shutdown_connections 2023-07-14T19:29:57.142 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:57.139+0000 7eff63f73700 1 -- 172.21.15.98:0/3860061355 wait complete. 2023-07-14T19:29:57.526 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81c9387d2e7576da065747ec9c16d1c7d69016ec shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 9adea648-227b-11ee-9b33-001a4aab830c -- bash -c 'ceph orch ps' 2023-07-14T19:29:58.078 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/9adea648-227b-11ee-9b33-001a4aab830c/mon.a/config 2023-07-14T19:29:58.941 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:29:58 smithi101 ceph-mon[128807]: from='client.15108 -' entity='client.admin' cmd=[{"prefix": "orch status", "target": ["mon-mgr", ""]}]: dispatch 2023-07-14T19:29:58.941 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:29:58 smithi101 ceph-mon[128807]: pgmap v281: 97 pgs: 97 active+clean; 634 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2023-07-14T19:29:58.970 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:29:58 smithi139 ceph-mon[129293]: from='client.15108 -' entity='client.admin' cmd=[{"prefix": "orch status", "target": ["mon-mgr", ""]}]: dispatch 2023-07-14T19:29:58.971 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:29:58 smithi139 ceph-mon[129293]: pgmap v281: 97 pgs: 97 active+clean; 634 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2023-07-14T19:29:59.031 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:29:58 smithi098 ceph-mon[127486]: from='client.15108 -' entity='client.admin' cmd=[{"prefix": "orch status", "target": ["mon-mgr", ""]}]: dispatch 2023-07-14T19:29:59.031 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:29:58 smithi098 ceph-mon[127486]: pgmap v281: 97 pgs: 97 active+clean; 634 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2023-07-14T19:29:59.283 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:59.282+0000 7f6570989700 1 -- 172.21.15.98:0/2656548016 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f656c0febb0 msgr2=0x7f656c0fefc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:29:59.283 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:59.282+0000 7f6570989700 1 --2- 172.21.15.98:0/2656548016 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f656c0febb0 0x7f656c0fefc0 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7f6554009990 tx=0x7f6554009ca0 comp rx=0 tx=0).stop 2023-07-14T19:29:59.283 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:59.282+0000 7f6570989700 1 -- 172.21.15.98:0/2656548016 shutdown_connections 2023-07-14T19:29:59.284 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:59.282+0000 7f6570989700 1 --2- 172.21.15.98:0/2656548016 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f656c100740 0x7f656c107000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:29:59.284 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:59.282+0000 7f6570989700 1 --2- 172.21.15.98:0/2656548016 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f656c0febb0 0x7f656c0fefc0 unknown :-1 s=CLOSED pgs=339 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:29:59.284 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:59.282+0000 7f6570989700 1 --2- 172.21.15.98:0/2656548016 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f656c0ffdb0 0x7f656c100200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:29:59.284 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:59.282+0000 7f6570989700 1 -- 172.21.15.98:0/2656548016 >> 172.21.15.98:0/2656548016 conn(0x7f656c0fa1e0 msgr2=0x7f656c0fc630 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:29:59.284 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:59.282+0000 7f6570989700 1 -- 172.21.15.98:0/2656548016 shutdown_connections 2023-07-14T19:29:59.284 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:59.282+0000 7f6570989700 1 -- 172.21.15.98:0/2656548016 wait complete. 2023-07-14T19:29:59.286 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:59.283+0000 7f6570989700 1 Processor -- start 2023-07-14T19:29:59.286 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:59.283+0000 7f6570989700 1 -- start start 2023-07-14T19:29:59.286 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:59.283+0000 7f6570989700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f656c0febb0 0x7f656c1948d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:29:59.286 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:59.283+0000 7f6570989700 1 --2- >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f656c0ffdb0 0x7f656c194e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:29:59.287 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:59.283+0000 7f6570989700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f656c100740 0x7f656c19be90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:29:59.287 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:59.283+0000 7f6570989700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f656c19c460 con 0x7f656c0febb0 2023-07-14T19:29:59.287 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:59.283+0000 7f6570989700 1 -- --> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f656c19c5a0 con 0x7f656c0ffdb0 2023-07-14T19:29:59.287 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:59.283+0000 7f6570989700 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f656c19c730 con 0x7f656c100740 2023-07-14T19:29:59.287 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:59.283+0000 7f656a59c700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f656c0febb0 0x7f656c1948d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:29:59.287 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:59.283+0000 7f656a59c700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f656c0febb0 0x7f656c1948d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:46614/0 (socket says 172.21.15.98:46614) 2023-07-14T19:29:59.287 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:59.283+0000 7f656a59c700 1 -- 172.21.15.98:0/2624603951 learned_addr learned my addr 172.21.15.98:0/2624603951 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-07-14T19:29:59.288 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:59.284+0000 7f656ad9d700 1 --2- 172.21.15.98:0/2624603951 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f656c100740 0x7f656c19be90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:29:59.288 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:59.284+0000 7f6569d9b700 1 --2- 172.21.15.98:0/2624603951 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f656c0ffdb0 0x7f656c194e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:29:59.288 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:59.284+0000 7f656a59c700 1 -- 172.21.15.98:0/2624603951 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f656c100740 msgr2=0x7f656c19be90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:29:59.288 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:59.284+0000 7f656a59c700 1 --2- 172.21.15.98:0/2624603951 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f656c100740 0x7f656c19be90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:29:59.288 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:59.284+0000 7f656a59c700 1 -- 172.21.15.98:0/2624603951 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f656c0ffdb0 msgr2=0x7f656c194e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:29:59.288 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:59.284+0000 7f656a59c700 1 --2- 172.21.15.98:0/2624603951 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f656c0ffdb0 0x7f656c194e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:29:59.288 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:59.284+0000 7f656a59c700 1 -- 172.21.15.98:0/2624603951 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f65540095f0 con 0x7f656c0febb0 2023-07-14T19:29:59.289 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:59.284+0000 7f656a59c700 1 --2- 172.21.15.98:0/2624603951 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f656c0febb0 0x7f656c1948d0 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7f655400b5c0 tx=0x7f655400f690 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:29:59.289 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:59.284+0000 7f65637fe700 1 -- 172.21.15.98:0/2624603951 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f655401d070 con 0x7f656c0febb0 2023-07-14T19:29:59.289 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:59.284+0000 7f65637fe700 1 -- 172.21.15.98:0/2624603951 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f6554005030 con 0x7f656c0febb0 2023-07-14T19:29:59.289 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:59.284+0000 7f65637fe700 1 -- 172.21.15.98:0/2624603951 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6554004ab0 con 0x7f656c0febb0 2023-07-14T19:29:59.289 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:59.284+0000 7f6570989700 1 -- 172.21.15.98:0/2624603951 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f656c19c9b0 con 0x7f656c0febb0 2023-07-14T19:29:59.289 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:59.284+0000 7f6570989700 1 -- 172.21.15.98:0/2624603951 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f656c19cd70 con 0x7f656c0febb0 2023-07-14T19:29:59.289 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:59.285+0000 7f6570989700 1 -- 172.21.15.98:0/2624603951 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f656c10a370 con 0x7f656c0febb0 2023-07-14T19:29:59.290 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:59.286+0000 7f65637fe700 1 -- 172.21.15.98:0/2624603951 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 17) v1 ==== 135766+0+0 (secure 0 0 0) 0x7f65540051a0 con 0x7f656c0febb0 2023-07-14T19:29:59.290 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:59.287+0000 7f65637fe700 1 --2- 172.21.15.98:0/2624603951 >> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] conn(0x7f65580a11e0 0x7f65580a3690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:29:59.290 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:59.287+0000 7f6569d9b700 1 --2- 172.21.15.98:0/2624603951 >> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] conn(0x7f65580a11e0 0x7f65580a3690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:29:59.291 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:59.287+0000 7f65637fe700 1 -- 172.21.15.98:0/2624603951 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 7979+0+0 (secure 0 0 0) 0x7f65540cdd20 con 0x7f656c0febb0 2023-07-14T19:29:59.291 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:59.287+0000 7f6569d9b700 1 --2- 172.21.15.98:0/2624603951 >> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] conn(0x7f65580a11e0 0x7f65580a3690 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f655c005950 tx=0x7f655c00b410 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:29:59.291 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:59.290+0000 7f65637fe700 1 -- 172.21.15.98:0/2624603951 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f6554097450 con 0x7f656c0febb0 2023-07-14T19:29:59.486 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:59.486+0000 7f6570989700 1 -- 172.21.15.98:0/2624603951 --> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7f656c060a80 con 0x7f65580a11e0 2023-07-14T19:29:59.499 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:59.498+0000 7f65637fe700 1 -- 172.21.15.98:0/2624603951 <== mgr.14152 v2:172.21.15.98:6800/452042164 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+4185 (secure 0 0 0) 0x7f656c060a80 con 0x7f65580a11e0 2023-07-14T19:29:59.499 INFO:teuthology.orchestra.run.smithi098.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2023-07-14T19:29:59.499 INFO:teuthology.orchestra.run.smithi098.stdout:mds.a smithi098 running (66s) - 66s 13.4M - 18.0.0-4940-g81c9387d 219b180907a8 72b8a2739e10 2023-07-14T19:29:59.500 INFO:teuthology.orchestra.run.smithi098.stdout:mds.b smithi101 running (64s) - 64s 15.7M - 18.0.0-4940-g81c9387d 219b180907a8 3ff95e443806 2023-07-14T19:29:59.500 INFO:teuthology.orchestra.run.smithi098.stdout:mds.c smithi139 running (75s) - 75s 16.8M - 18.0.0-4940-g81c9387d 219b180907a8 5ee99378ab7a 2023-07-14T19:29:59.500 INFO:teuthology.orchestra.run.smithi098.stdout:mds.d smithi098 running (82s) - 81s 14.5M - 18.0.0-4940-g81c9387d 219b180907a8 37123ac74367 2023-07-14T19:29:59.500 INFO:teuthology.orchestra.run.smithi098.stdout:mds.e smithi101 running (79s) - 79s 16.5M - 18.0.0-4940-g81c9387d 219b180907a8 852dc3d2dd89 2023-07-14T19:29:59.500 INFO:teuthology.orchestra.run.smithi098.stdout:mds.f smithi139 running (77s) - 77s 16.9M - 18.0.0-4940-g81c9387d 219b180907a8 98c4e4395a70 2023-07-14T19:29:59.500 INFO:teuthology.orchestra.run.smithi098.stdout:mds.g smithi098 running (69s) - 68s 16.8M - 18.0.0-4940-g81c9387d 219b180907a8 13afeea7158e 2023-07-14T19:29:59.500 INFO:teuthology.orchestra.run.smithi098.stdout:mds.h smithi101 running (73s) - 72s 16.9M - 18.0.0-4940-g81c9387d 219b180907a8 72be8a5e632b 2023-07-14T19:29:59.501 INFO:teuthology.orchestra.run.smithi098.stdout:mds.i smithi139 running (71s) - 70s 16.7M - 18.0.0-4940-g81c9387d 219b180907a8 d07608af6635 2023-07-14T19:29:59.501 INFO:teuthology.orchestra.run.smithi098.stdout:mds.j smithi098 running (84s) - 83s 14.4M - 18.0.0-4940-g81c9387d 219b180907a8 76133ab42514 2023-07-14T19:29:59.501 INFO:teuthology.orchestra.run.smithi098.stdout:mds.k smithi101 running (86s) - 85s 14.4M - 18.0.0-4940-g81c9387d 219b180907a8 d9f2148b1993 2023-07-14T19:29:59.501 INFO:teuthology.orchestra.run.smithi098.stdout:mds.l smithi139 running (88s) - 88s 13.4M - 18.0.0-4940-g81c9387d 219b180907a8 44462b89c5cd 2023-07-14T19:29:59.501 INFO:teuthology.orchestra.run.smithi098.stdout:mgr.x smithi098 *:9283,8765 running (7m) - 7m 511M - 18.0.0-4940-g81c9387d 219b180907a8 b3f6c0ebf294 2023-07-14T19:29:59.501 INFO:teuthology.orchestra.run.smithi098.stdout:mgr.y smithi101 *:8443,8765 running (5m) - 5m 446M - 18.0.0-4940-g81c9387d 219b180907a8 615bd99c6c7f 2023-07-14T19:29:59.501 INFO:teuthology.orchestra.run.smithi098.stdout:mgr.z smithi139 *:8443,8765 running (5m) - 5m 448M - 18.0.0-4940-g81c9387d 219b180907a8 faa8fef8c923 2023-07-14T19:29:59.501 INFO:teuthology.orchestra.run.smithi098.stdout:mon.a smithi098 running (7m) - 7m 82.9M 2048M 18.0.0-4940-g81c9387d 219b180907a8 99ff082ac7c0 2023-07-14T19:29:59.502 INFO:teuthology.orchestra.run.smithi098.stdout:mon.b smithi101 running (6m) - 6m 56.9M 2048M 18.0.0-4940-g81c9387d 219b180907a8 cee07623756b 2023-07-14T19:29:59.502 INFO:teuthology.orchestra.run.smithi098.stdout:mon.c smithi139 running (6m) - 6m 57.5M 2048M 18.0.0-4940-g81c9387d 219b180907a8 fa3a6a6f918f 2023-07-14T19:29:59.502 INFO:teuthology.orchestra.run.smithi098.stdout:osd.0 smithi098 running (5m) - 5m 53.8M 4292M 18.0.0-4940-g81c9387d 219b180907a8 7d54ed5366b0 2023-07-14T19:29:59.502 INFO:teuthology.orchestra.run.smithi098.stdout:osd.1 smithi101 running (5m) - 5m 60.6M 4292M 18.0.0-4940-g81c9387d 219b180907a8 d31ce0c27ae0 2023-07-14T19:29:59.502 INFO:teuthology.orchestra.run.smithi098.stdout:osd.2 smithi139 running (4m) - 4m 54.1M 4292M 18.0.0-4940-g81c9387d 219b180907a8 eee509688fa1 2023-07-14T19:29:59.502 INFO:teuthology.orchestra.run.smithi098.stdout:osd.3 smithi098 running (4m) - 4m 54.5M 4292M 18.0.0-4940-g81c9387d 219b180907a8 7b1ecef4162e 2023-07-14T19:29:59.502 INFO:teuthology.orchestra.run.smithi098.stdout:osd.4 smithi101 running (4m) - 4m 50.7M 4292M 18.0.0-4940-g81c9387d 219b180907a8 67c8deff38c5 2023-07-14T19:29:59.502 INFO:teuthology.orchestra.run.smithi098.stdout:osd.5 smithi139 running (3m) - 3m 54.4M 4292M 18.0.0-4940-g81c9387d 219b180907a8 08d19304d0c9 2023-07-14T19:29:59.503 INFO:teuthology.orchestra.run.smithi098.stdout:osd.6 smithi098 running (3m) - 3m 47.8M 4292M 18.0.0-4940-g81c9387d 219b180907a8 b1164d8cce22 2023-07-14T19:29:59.503 INFO:teuthology.orchestra.run.smithi098.stdout:osd.7 smithi101 running (3m) - 3m 54.7M 4292M 18.0.0-4940-g81c9387d 219b180907a8 5aa1d959c637 2023-07-14T19:29:59.503 INFO:teuthology.orchestra.run.smithi098.stdout:osd.8 smithi139 running (2m) - 2m 48.6M 4292M 18.0.0-4940-g81c9387d 219b180907a8 b293788bd01c 2023-07-14T19:29:59.503 INFO:teuthology.orchestra.run.smithi098.stdout:osd.9 smithi098 running (2m) - 2m 45.5M 4292M 18.0.0-4940-g81c9387d 219b180907a8 d515531ddd9e 2023-07-14T19:29:59.503 INFO:teuthology.orchestra.run.smithi098.stdout:osd.10 smithi101 running (2m) - 2m 48.5M 4292M 18.0.0-4940-g81c9387d 219b180907a8 f2a320100ca9 2023-07-14T19:29:59.503 INFO:teuthology.orchestra.run.smithi098.stdout:osd.11 smithi139 running (106s) - 106s 44.9M 4292M 18.0.0-4940-g81c9387d 219b180907a8 07c7327ae9e5 2023-07-14T19:29:59.504 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:59.500+0000 7f6570989700 1 -- 172.21.15.98:0/2624603951 >> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] conn(0x7f65580a11e0 msgr2=0x7f65580a3690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:29:59.504 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:59.500+0000 7f6570989700 1 --2- 172.21.15.98:0/2624603951 >> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] conn(0x7f65580a11e0 0x7f65580a3690 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f655c005950 tx=0x7f655c00b410 comp rx=0 tx=0).stop 2023-07-14T19:29:59.504 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:59.500+0000 7f6570989700 1 -- 172.21.15.98:0/2624603951 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f656c0febb0 msgr2=0x7f656c1948d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:29:59.505 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:59.500+0000 7f6570989700 1 --2- 172.21.15.98:0/2624603951 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f656c0febb0 0x7f656c1948d0 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7f655400b5c0 tx=0x7f655400f690 comp rx=0 tx=0).stop 2023-07-14T19:29:59.505 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:59.501+0000 7f6570989700 1 -- 172.21.15.98:0/2624603951 shutdown_connections 2023-07-14T19:29:59.505 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:59.501+0000 7f6570989700 1 --2- 172.21.15.98:0/2624603951 >> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] conn(0x7f65580a11e0 0x7f65580a3690 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:29:59.505 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:59.501+0000 7f6570989700 1 --2- 172.21.15.98:0/2624603951 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f656c100740 0x7f656c19be90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:29:59.505 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:59.501+0000 7f6570989700 1 --2- 172.21.15.98:0/2624603951 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f656c0febb0 0x7f656c1948d0 unknown :-1 s=CLOSED pgs=340 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:29:59.505 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:59.501+0000 7f6570989700 1 --2- 172.21.15.98:0/2624603951 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f656c0ffdb0 0x7f656c194e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:29:59.505 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:59.501+0000 7f6570989700 1 -- 172.21.15.98:0/2624603951 >> 172.21.15.98:0/2624603951 conn(0x7f656c0fa1e0 msgr2=0x7f656c0fc4c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:29:59.506 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:59.501+0000 7f6570989700 1 -- 172.21.15.98:0/2624603951 shutdown_connections 2023-07-14T19:29:59.506 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:29:59.501+0000 7f6570989700 1 -- 172.21.15.98:0/2624603951 wait complete. 2023-07-14T19:29:59.834 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81c9387d2e7576da065747ec9c16d1c7d69016ec shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 9adea648-227b-11ee-9b33-001a4aab830c -- bash -c 'ceph orch ls' 2023-07-14T19:30:00.388 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/9adea648-227b-11ee-9b33-001a4aab830c/mon.a/config 2023-07-14T19:30:00.896 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:30:00 smithi098 ceph-mon[127486]: from='client.15114 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2023-07-14T19:30:00.897 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:30:00 smithi098 ceph-mon[127486]: pgmap v282: 97 pgs: 97 active+clean; 634 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2023-07-14T19:30:00.897 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:30:00 smithi098 ceph-mon[127486]: overall HEALTH_OK 2023-07-14T19:30:00.941 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:30:00 smithi101 ceph-mon[128807]: from='client.15114 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2023-07-14T19:30:00.941 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:30:00 smithi101 ceph-mon[128807]: pgmap v282: 97 pgs: 97 active+clean; 634 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2023-07-14T19:30:00.942 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:30:00 smithi101 ceph-mon[128807]: overall HEALTH_OK 2023-07-14T19:30:00.970 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:30:00 smithi139 ceph-mon[129293]: from='client.15114 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2023-07-14T19:30:00.970 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:30:00 smithi139 ceph-mon[129293]: pgmap v282: 97 pgs: 97 active+clean; 634 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2023-07-14T19:30:00.970 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:30:00 smithi139 ceph-mon[129293]: overall HEALTH_OK 2023-07-14T19:30:01.592 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:01.590+0000 7eff6cff5700 1 -- 172.21.15.98:0/410746808 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7eff68100740 msgr2=0x7eff68107000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:30:01.593 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:01.590+0000 7eff6cff5700 1 --2- 172.21.15.98:0/410746808 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7eff68100740 0x7eff68107000 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7eff6000b0a0 tx=0x7eff6000b3b0 comp rx=0 tx=0).stop 2023-07-14T19:30:01.593 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:01.590+0000 7eff6cff5700 1 -- 172.21.15.98:0/410746808 shutdown_connections 2023-07-14T19:30:01.593 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:01.590+0000 7eff6cff5700 1 --2- 172.21.15.98:0/410746808 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7eff680febb0 0x7eff680fefc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:30:01.593 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:01.590+0000 7eff6cff5700 1 --2- 172.21.15.98:0/410746808 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7eff68100740 0x7eff68107000 unknown :-1 s=CLOSED pgs=341 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:30:01.593 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:01.590+0000 7eff6cff5700 1 --2- 172.21.15.98:0/410746808 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7eff680ffdb0 0x7eff68100200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:30:01.594 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:01.590+0000 7eff6cff5700 1 -- 172.21.15.98:0/410746808 >> 172.21.15.98:0/410746808 conn(0x7eff680fa1e0 msgr2=0x7eff680fc630 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:30:01.594 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:01.590+0000 7eff6cff5700 1 -- 172.21.15.98:0/410746808 shutdown_connections 2023-07-14T19:30:01.594 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:01.590+0000 7eff6cff5700 1 -- 172.21.15.98:0/410746808 wait complete. 2023-07-14T19:30:01.594 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:01.591+0000 7eff6cff5700 1 Processor -- start 2023-07-14T19:30:01.594 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:01.591+0000 7eff6cff5700 1 -- start start 2023-07-14T19:30:01.594 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:01.591+0000 7eff6cff5700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7eff680febb0 0x7eff68194980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:30:01.594 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:01.591+0000 7eff6cff5700 1 --2- >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7eff680ffdb0 0x7eff68194ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:30:01.594 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:01.591+0000 7eff6cff5700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7eff68100740 0x7eff6819bf40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:30:01.595 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:01.591+0000 7eff6cff5700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7eff6819c510 con 0x7eff68100740 2023-07-14T19:30:01.595 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:01.591+0000 7eff6cff5700 1 -- --> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] -- mon_getmap magic: 0 v1 -- 0x7eff6819c650 con 0x7eff680ffdb0 2023-07-14T19:30:01.595 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:01.591+0000 7eff6cff5700 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 v1 -- 0x7eff6819c7e0 con 0x7eff680febb0 2023-07-14T19:30:01.595 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:01.591+0000 7eff6659c700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7eff680febb0 0x7eff68194980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:30:01.595 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:01.591+0000 7eff66d9d700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7eff68100740 0x7eff6819bf40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:30:01.595 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:01.592+0000 7eff6659c700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7eff680febb0 0x7eff68194980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.139:3300/0 says I am v2:172.21.15.98:53580/0 (socket says 172.21.15.98:53580) 2023-07-14T19:30:01.595 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:01.592+0000 7eff6659c700 1 -- 172.21.15.98:0/1429772109 learned_addr learned my addr 172.21.15.98:0/1429772109 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-07-14T19:30:01.596 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:01.592+0000 7eff65d9b700 1 --2- 172.21.15.98:0/1429772109 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7eff680ffdb0 0x7eff68194ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:30:01.596 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:01.592+0000 7eff66d9d700 1 -- 172.21.15.98:0/1429772109 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7eff680febb0 msgr2=0x7eff68194980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:30:01.596 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:01.592+0000 7eff66d9d700 1 --2- 172.21.15.98:0/1429772109 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7eff680febb0 0x7eff68194980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:30:01.596 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:01.592+0000 7eff66d9d700 1 -- 172.21.15.98:0/1429772109 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7eff680ffdb0 msgr2=0x7eff68194ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:30:01.596 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:01.592+0000 7eff66d9d700 1 --2- 172.21.15.98:0/1429772109 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7eff680ffdb0 0x7eff68194ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:30:01.596 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:01.592+0000 7eff66d9d700 1 -- 172.21.15.98:0/1429772109 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7eff60009d00 con 0x7eff68100740 2023-07-14T19:30:01.596 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:01.592+0000 7eff66d9d700 1 --2- 172.21.15.98:0/1429772109 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7eff68100740 0x7eff6819bf40 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7eff6000b0a0 tx=0x7eff60004c80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:30:01.597 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:01.592+0000 7eff5f7fe700 1 -- 172.21.15.98:0/1429772109 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7eff6000e040 con 0x7eff68100740 2023-07-14T19:30:01.597 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:01.592+0000 7eff6cff5700 1 -- 172.21.15.98:0/1429772109 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7eff6819ca60 con 0x7eff68100740 2023-07-14T19:30:01.597 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:01.592+0000 7eff5f7fe700 1 -- 172.21.15.98:0/1429772109 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7eff6000cec0 con 0x7eff68100740 2023-07-14T19:30:01.597 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:01.592+0000 7eff5f7fe700 1 -- 172.21.15.98:0/1429772109 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7eff6001baf0 con 0x7eff68100740 2023-07-14T19:30:01.597 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:01.592+0000 7eff6cff5700 1 -- 172.21.15.98:0/1429772109 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7eff6819ce80 con 0x7eff68100740 2023-07-14T19:30:01.598 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:01.593+0000 7eff6cff5700 1 -- 172.21.15.98:0/1429772109 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7eff68106160 con 0x7eff68100740 2023-07-14T19:30:01.599 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:01.594+0000 7eff5f7fe700 1 -- 172.21.15.98:0/1429772109 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 17) v1 ==== 135766+0+0 (secure 0 0 0) 0x7eff6001bc50 con 0x7eff68100740 2023-07-14T19:30:01.599 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:01.595+0000 7eff5f7fe700 1 --2- 172.21.15.98:0/1429772109 >> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] conn(0x7eff540a11e0 0x7eff540a3690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:30:01.599 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:01.595+0000 7eff6659c700 1 --2- 172.21.15.98:0/1429772109 >> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] conn(0x7eff540a11e0 0x7eff540a3690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:30:01.599 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:01.595+0000 7eff5f7fe700 1 -- 172.21.15.98:0/1429772109 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 7979+0+0 (secure 0 0 0) 0x7eff600ce750 con 0x7eff68100740 2023-07-14T19:30:01.599 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:01.595+0000 7eff6659c700 1 --2- 172.21.15.98:0/1429772109 >> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] conn(0x7eff540a11e0 0x7eff540a3690 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7eff5800bd70 tx=0x7eff5800b4a0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:30:01.600 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:01.600+0000 7eff5f7fe700 1 -- 172.21.15.98:0/1429772109 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7eff60097f80 con 0x7eff68100740 2023-07-14T19:30:01.798 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:01.797+0000 7eff6cff5700 1 -- 172.21.15.98:0/1429772109 --> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] -- mgr_command(tid 0: {"prefix": "orch ls", "target": ["mon-mgr", ""]}) v1 -- 0x7eff68060a80 con 0x7eff540a11e0 2023-07-14T19:30:01.802 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:01.802+0000 7eff5f7fe700 1 -- 172.21.15.98:0/1429772109 <== mgr.14152 v2:172.21.15.98:6800/452042164 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+980 (secure 0 0 0) 0x7eff68060a80 con 0x7eff540a11e0 2023-07-14T19:30:01.803 INFO:teuthology.orchestra.run.smithi098.stdout:NAME PORTS RUNNING REFRESHED AGE PLACEMENT 2023-07-14T19:30:01.803 INFO:teuthology.orchestra.run.smithi098.stdout:mds.all 12/12 - 92s smithi098=a;smithi098=d;smithi098=g;smithi098=j;smithi101=b;smithi101=e;smithi101=h;smithi101=k;smithi139=c;smithi139=f;smithi139=i;smithi139=l;count:12 2023-07-14T19:30:01.803 INFO:teuthology.orchestra.run.smithi098.stdout:mgr 3/3 - 6m smithi098=x;smithi101=y;smithi139=z;count:3 2023-07-14T19:30:01.803 INFO:teuthology.orchestra.run.smithi098.stdout:mon 3/3 - 6m smithi098:172.21.15.98=a;smithi101:172.21.15.101=b;smithi139:172.21.15.139=c;count:3 2023-07-14T19:30:01.803 INFO:teuthology.orchestra.run.smithi098.stdout:osd 12 - - 2023-07-14T19:30:01.805 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:01.804+0000 7eff6cff5700 1 -- 172.21.15.98:0/1429772109 >> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] conn(0x7eff540a11e0 msgr2=0x7eff540a3690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:30:01.805 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:01.804+0000 7eff6cff5700 1 --2- 172.21.15.98:0/1429772109 >> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] conn(0x7eff540a11e0 0x7eff540a3690 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7eff5800bd70 tx=0x7eff5800b4a0 comp rx=0 tx=0).stop 2023-07-14T19:30:01.805 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:01.804+0000 7eff6cff5700 1 -- 172.21.15.98:0/1429772109 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7eff68100740 msgr2=0x7eff6819bf40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:30:01.805 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:01.804+0000 7eff6cff5700 1 --2- 172.21.15.98:0/1429772109 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7eff68100740 0x7eff6819bf40 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7eff6000b0a0 tx=0x7eff60004c80 comp rx=0 tx=0).stop 2023-07-14T19:30:01.805 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:01.804+0000 7eff6cff5700 1 -- 172.21.15.98:0/1429772109 shutdown_connections 2023-07-14T19:30:01.805 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:01.804+0000 7eff6cff5700 1 --2- 172.21.15.98:0/1429772109 >> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] conn(0x7eff540a11e0 0x7eff540a3690 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:30:01.806 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:01.804+0000 7eff6cff5700 1 --2- 172.21.15.98:0/1429772109 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7eff680febb0 0x7eff68194980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:30:01.806 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:01.804+0000 7eff6cff5700 1 --2- 172.21.15.98:0/1429772109 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7eff68100740 0x7eff6819bf40 unknown :-1 s=CLOSED pgs=342 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:30:01.806 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:01.804+0000 7eff6cff5700 1 --2- 172.21.15.98:0/1429772109 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7eff680ffdb0 0x7eff68194ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:30:01.806 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:01.804+0000 7eff6cff5700 1 -- 172.21.15.98:0/1429772109 >> 172.21.15.98:0/1429772109 conn(0x7eff680fa1e0 msgr2=0x7eff681032a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:30:01.806 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:01.804+0000 7eff6cff5700 1 -- 172.21.15.98:0/1429772109 shutdown_connections 2023-07-14T19:30:01.806 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:01.804+0000 7eff6cff5700 1 -- 172.21.15.98:0/1429772109 wait complete. 2023-07-14T19:30:02.348 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81c9387d2e7576da065747ec9c16d1c7d69016ec shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 9adea648-227b-11ee-9b33-001a4aab830c -- bash -c 'ceph orch host ls' 2023-07-14T19:30:02.471 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:30:02 smithi139 ceph-mon[129293]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.4", "id": [2, 11]}]: dispatch 2023-07-14T19:30:02.471 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:30:02 smithi139 ceph-mon[129293]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.f", "id": [8, 11]}]: dispatch 2023-07-14T19:30:02.472 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:30:02 smithi139 ceph-mon[129293]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.0", "id": [4, 8]}]: dispatch 2023-07-14T19:30:02.472 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:30:02 smithi139 ceph-mon[129293]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1", "id": [4, 9]}]: dispatch 2023-07-14T19:30:02.472 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:30:02 smithi139 ceph-mon[129293]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.c", "id": [3, 9]}]: dispatch 2023-07-14T19:30:02.473 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:30:02 smithi139 ceph-mon[129293]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.17", "id": [4, 6]}]: dispatch 2023-07-14T19:30:02.473 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:30:02 smithi139 ceph-mon[129293]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1e", "id": [3, 9]}]: dispatch 2023-07-14T19:30:02.473 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:30:02 smithi139 ceph-mon[129293]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.28", "id": [5, 7]}]: dispatch 2023-07-14T19:30:02.473 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:30:02 smithi139 ceph-mon[129293]: pgmap v283: 97 pgs: 97 active+clean; 634 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2023-07-14T19:30:02.474 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:30:02 smithi139 ceph-mon[129293]: from='client.15120 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2023-07-14T19:30:02.474 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:30:02 smithi139 ceph-mon[129293]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-14T19:30:02.531 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:30:02 smithi098 ceph-mon[127486]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.4", "id": [2, 11]}]: dispatch 2023-07-14T19:30:02.532 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:30:02 smithi098 ceph-mon[127486]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.f", "id": [8, 11]}]: dispatch 2023-07-14T19:30:02.532 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:30:02 smithi098 ceph-mon[127486]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.0", "id": [4, 8]}]: dispatch 2023-07-14T19:30:02.532 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:30:02 smithi098 ceph-mon[127486]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1", "id": [4, 9]}]: dispatch 2023-07-14T19:30:02.532 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:30:02 smithi098 ceph-mon[127486]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.c", "id": [3, 9]}]: dispatch 2023-07-14T19:30:02.532 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:30:02 smithi098 ceph-mon[127486]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.17", "id": [4, 6]}]: dispatch 2023-07-14T19:30:02.532 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:30:02 smithi098 ceph-mon[127486]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1e", "id": [3, 9]}]: dispatch 2023-07-14T19:30:02.532 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:30:02 smithi098 ceph-mon[127486]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.28", "id": [5, 7]}]: dispatch 2023-07-14T19:30:02.533 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:30:02 smithi098 ceph-mon[127486]: pgmap v283: 97 pgs: 97 active+clean; 634 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2023-07-14T19:30:02.533 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:30:02 smithi098 ceph-mon[127486]: from='client.15120 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2023-07-14T19:30:02.533 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:30:02 smithi098 ceph-mon[127486]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-14T19:30:02.691 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:30:02 smithi101 ceph-mon[128807]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.4", "id": [2, 11]}]: dispatch 2023-07-14T19:30:02.692 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:30:02 smithi101 ceph-mon[128807]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.f", "id": [8, 11]}]: dispatch 2023-07-14T19:30:02.692 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:30:02 smithi101 ceph-mon[128807]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.0", "id": [4, 8]}]: dispatch 2023-07-14T19:30:02.692 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:30:02 smithi101 ceph-mon[128807]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1", "id": [4, 9]}]: dispatch 2023-07-14T19:30:02.692 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:30:02 smithi101 ceph-mon[128807]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.c", "id": [3, 9]}]: dispatch 2023-07-14T19:30:02.692 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:30:02 smithi101 ceph-mon[128807]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.17", "id": [4, 6]}]: dispatch 2023-07-14T19:30:02.692 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:30:02 smithi101 ceph-mon[128807]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1e", "id": [3, 9]}]: dispatch 2023-07-14T19:30:02.693 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:30:02 smithi101 ceph-mon[128807]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.28", "id": [5, 7]}]: dispatch 2023-07-14T19:30:02.693 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:30:02 smithi101 ceph-mon[128807]: pgmap v283: 97 pgs: 97 active+clean; 634 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2023-07-14T19:30:02.693 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:30:02 smithi101 ceph-mon[128807]: from='client.15120 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2023-07-14T19:30:02.693 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:30:02 smithi101 ceph-mon[128807]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-14T19:30:02.904 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/9adea648-227b-11ee-9b33-001a4aab830c/mon.a/config 2023-07-14T19:30:03.531 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:30:03 smithi098 ceph-mon[127486]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.4", "id": [2, 11]}]': finished 2023-07-14T19:30:03.532 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:30:03 smithi098 ceph-mon[127486]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.f", "id": [8, 11]}]': finished 2023-07-14T19:30:03.532 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:30:03 smithi098 ceph-mon[127486]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.0", "id": [4, 8]}]': finished 2023-07-14T19:30:03.532 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:30:03 smithi098 ceph-mon[127486]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1", "id": [4, 9]}]': finished 2023-07-14T19:30:03.532 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:30:03 smithi098 ceph-mon[127486]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.c", "id": [3, 9]}]': finished 2023-07-14T19:30:03.532 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:30:03 smithi098 ceph-mon[127486]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.17", "id": [4, 6]}]': finished 2023-07-14T19:30:03.533 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:30:03 smithi098 ceph-mon[127486]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1e", "id": [3, 9]}]': finished 2023-07-14T19:30:03.533 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:30:03 smithi098 ceph-mon[127486]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.28", "id": [5, 7]}]': finished 2023-07-14T19:30:03.533 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:30:03 smithi098 ceph-mon[127486]: osdmap e84: 12 total, 12 up, 12 in 2023-07-14T19:30:03.691 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:30:03 smithi101 ceph-mon[128807]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.4", "id": [2, 11]}]': finished 2023-07-14T19:30:03.691 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:30:03 smithi101 ceph-mon[128807]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.f", "id": [8, 11]}]': finished 2023-07-14T19:30:03.691 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:30:03 smithi101 ceph-mon[128807]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.0", "id": [4, 8]}]': finished 2023-07-14T19:30:03.692 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:30:03 smithi101 ceph-mon[128807]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1", "id": [4, 9]}]': finished 2023-07-14T19:30:03.692 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:30:03 smithi101 ceph-mon[128807]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.c", "id": [3, 9]}]': finished 2023-07-14T19:30:03.692 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:30:03 smithi101 ceph-mon[128807]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.17", "id": [4, 6]}]': finished 2023-07-14T19:30:03.692 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:30:03 smithi101 ceph-mon[128807]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1e", "id": [3, 9]}]': finished 2023-07-14T19:30:03.692 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:30:03 smithi101 ceph-mon[128807]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.28", "id": [5, 7]}]': finished 2023-07-14T19:30:03.692 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:30:03 smithi101 ceph-mon[128807]: osdmap e84: 12 total, 12 up, 12 in 2023-07-14T19:30:03.720 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:30:03 smithi139 ceph-mon[129293]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.4", "id": [2, 11]}]': finished 2023-07-14T19:30:03.721 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:30:03 smithi139 ceph-mon[129293]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.f", "id": [8, 11]}]': finished 2023-07-14T19:30:03.721 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:30:03 smithi139 ceph-mon[129293]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.0", "id": [4, 8]}]': finished 2023-07-14T19:30:03.721 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:30:03 smithi139 ceph-mon[129293]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1", "id": [4, 9]}]': finished 2023-07-14T19:30:03.721 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:30:03 smithi139 ceph-mon[129293]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.c", "id": [3, 9]}]': finished 2023-07-14T19:30:03.721 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:30:03 smithi139 ceph-mon[129293]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.17", "id": [4, 6]}]': finished 2023-07-14T19:30:03.722 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:30:03 smithi139 ceph-mon[129293]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1e", "id": [3, 9]}]': finished 2023-07-14T19:30:03.722 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:30:03 smithi139 ceph-mon[129293]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.28", "id": [5, 7]}]': finished 2023-07-14T19:30:03.722 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:30:03 smithi139 ceph-mon[129293]: osdmap e84: 12 total, 12 up, 12 in 2023-07-14T19:30:04.211 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:04.209+0000 7fd1a73ef700 1 -- 172.21.15.98:0/2588156025 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd1a00fe350 msgr2=0x7fd1a00fe760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:30:04.211 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:04.209+0000 7fd1a73ef700 1 --2- 172.21.15.98:0/2588156025 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd1a00fe350 0x7fd1a00fe760 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7fd19c009990 tx=0x7fd19c009ca0 comp rx=0 tx=0).stop 2023-07-14T19:30:04.211 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:04.209+0000 7fd1a73ef700 1 -- 172.21.15.98:0/2588156025 shutdown_connections 2023-07-14T19:30:04.211 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:04.209+0000 7fd1a73ef700 1 --2- 172.21.15.98:0/2588156025 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fd1a00fffa0 0x7fd1a0106840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:30:04.211 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:04.209+0000 7fd1a73ef700 1 --2- 172.21.15.98:0/2588156025 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd1a00fe350 0x7fd1a00fe760 unknown :-1 s=CLOSED pgs=343 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:30:04.212 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:04.209+0000 7fd1a73ef700 1 --2- 172.21.15.98:0/2588156025 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7fd1a00ff5f0 0x7fd1a00ffa60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:30:04.212 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:04.209+0000 7fd1a73ef700 1 -- 172.21.15.98:0/2588156025 >> 172.21.15.98:0/2588156025 conn(0x7fd1a00f99c0 msgr2=0x7fd1a00fbdd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:30:04.212 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:04.209+0000 7fd1a73ef700 1 -- 172.21.15.98:0/2588156025 shutdown_connections 2023-07-14T19:30:04.212 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:04.209+0000 7fd1a73ef700 1 -- 172.21.15.98:0/2588156025 wait complete. 2023-07-14T19:30:04.212 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:04.209+0000 7fd1a73ef700 1 Processor -- start 2023-07-14T19:30:04.212 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:04.209+0000 7fd1a73ef700 1 -- start start 2023-07-14T19:30:04.213 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:04.210+0000 7fd1a73ef700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd1a00fe350 0x7fd1a0101680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:30:04.213 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:04.210+0000 7fd1a73ef700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fd1a00ff5f0 0x7fd1a019a730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:30:04.213 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:04.210+0000 7fd1a73ef700 1 --2- >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7fd1a00fffa0 0x7fd1a019cd70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:30:04.213 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:04.210+0000 7fd1a73ef700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd1a019d510 con 0x7fd1a00fe350 2023-07-14T19:30:04.213 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:04.210+0000 7fd1a73ef700 1 -- --> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd1a019d680 con 0x7fd1a00fffa0 2023-07-14T19:30:04.213 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:04.210+0000 7fd1a73ef700 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd1a019d840 con 0x7fd1a00ff5f0 2023-07-14T19:30:04.213 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:04.210+0000 7fd1a598c700 1 --2- >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7fd1a00fffa0 0x7fd1a019cd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:30:04.213 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:04.210+0000 7fd1a598c700 1 --2- >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7fd1a00fffa0 0x7fd1a019cd70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.101:3300/0 says I am v2:172.21.15.98:55994/0 (socket says 172.21.15.98:55994) 2023-07-14T19:30:04.214 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:04.210+0000 7fd1a598c700 1 -- 172.21.15.98:0/1395715827 learned_addr learned my addr 172.21.15.98:0/1395715827 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-07-14T19:30:04.214 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:04.210+0000 7fd1a518b700 1 --2- 172.21.15.98:0/1395715827 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd1a00fe350 0x7fd1a0101680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:30:04.214 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:04.210+0000 7fd1a498a700 1 --2- 172.21.15.98:0/1395715827 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fd1a00ff5f0 0x7fd1a019a730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:30:04.214 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:04.210+0000 7fd1a598c700 1 -- 172.21.15.98:0/1395715827 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fd1a00ff5f0 msgr2=0x7fd1a019a730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:30:04.214 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:04.210+0000 7fd1a598c700 1 --2- 172.21.15.98:0/1395715827 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fd1a00ff5f0 0x7fd1a019a730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:30:04.214 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:04.210+0000 7fd1a598c700 1 -- 172.21.15.98:0/1395715827 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd1a00fe350 msgr2=0x7fd1a0101680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:30:04.214 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:04.210+0000 7fd1a598c700 1 --2- 172.21.15.98:0/1395715827 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd1a00fe350 0x7fd1a0101680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:30:04.215 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:04.210+0000 7fd1a598c700 1 -- 172.21.15.98:0/1395715827 --> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd19c0095f0 con 0x7fd1a00fffa0 2023-07-14T19:30:04.215 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:04.210+0000 7fd1a598c700 1 --2- 172.21.15.98:0/1395715827 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7fd1a00fffa0 0x7fd1a019cd70 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7fd198007ee0 tx=0x7fd19800d3b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:30:04.215 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:04.210+0000 7fd1967fc700 1 -- 172.21.15.98:0/1395715827 <== mon.2 v2:172.21.15.101:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd198017070 con 0x7fd1a00fffa0 2023-07-14T19:30:04.215 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:04.210+0000 7fd1a73ef700 1 -- 172.21.15.98:0/1395715827 --> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd1a019db20 con 0x7fd1a00fffa0 2023-07-14T19:30:04.215 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:04.210+0000 7fd1967fc700 1 -- 172.21.15.98:0/1395715827 <== mon.2 v2:172.21.15.101:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fd19800f040 con 0x7fd1a00fffa0 2023-07-14T19:30:04.215 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:04.211+0000 7fd1967fc700 1 -- 172.21.15.98:0/1395715827 <== mon.2 v2:172.21.15.101:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd198005580 con 0x7fd1a00fffa0 2023-07-14T19:30:04.215 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:04.211+0000 7fd1a73ef700 1 -- 172.21.15.98:0/1395715827 --> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd1a019dfd0 con 0x7fd1a00fffa0 2023-07-14T19:30:04.216 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:04.211+0000 7fd1a73ef700 1 -- 172.21.15.98:0/1395715827 --> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd1a0109b40 con 0x7fd1a00fffa0 2023-07-14T19:30:04.216 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:04.212+0000 7fd1967fc700 1 -- 172.21.15.98:0/1395715827 <== mon.2 v2:172.21.15.101:3300/0 4 ==== mgrmap(e 17) v1 ==== 135766+0+0 (secure 0 0 0) 0x7fd19800db20 con 0x7fd1a00fffa0 2023-07-14T19:30:04.217 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:04.213+0000 7fd1967fc700 1 --2- 172.21.15.98:0/1395715827 >> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] conn(0x7fd18c0a11e0 0x7fd18c0a3690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:30:04.217 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:04.213+0000 7fd1a518b700 1 --2- 172.21.15.98:0/1395715827 >> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] conn(0x7fd18c0a11e0 0x7fd18c0a3690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:30:04.217 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:04.213+0000 7fd1967fc700 1 -- 172.21.15.98:0/1395715827 <== mon.2 v2:172.21.15.101:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 8211+0+0 (secure 0 0 0) 0x7fd1980cc1b0 con 0x7fd1a00fffa0 2023-07-14T19:30:04.217 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:04.213+0000 7fd1a518b700 1 --2- 172.21.15.98:0/1395715827 >> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] conn(0x7fd18c0a11e0 0x7fd18c0a3690 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7fd19c00b5c0 tx=0x7fd19c005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:30:04.217 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:04.217+0000 7fd1967fc700 1 -- 172.21.15.98:0/1395715827 <== mon.2 v2:172.21.15.101:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7fd1980957b0 con 0x7fd1a00fffa0 2023-07-14T19:30:04.416 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:04.416+0000 7fd1a73ef700 1 -- 172.21.15.98:0/1395715827 --> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] -- mgr_command(tid 0: {"prefix": "orch host ls", "target": ["mon-mgr", ""]}) v1 -- 0x7fd1a0068190 con 0x7fd18c0a11e0 2023-07-14T19:30:04.418 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:04.418+0000 7fd1967fc700 1 -- 172.21.15.98:0/1395715827 <== mgr.14152 v2:172.21.15.98:6800/452042164 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+191 (secure 0 0 0) 0x7fd1a0068190 con 0x7fd18c0a11e0 2023-07-14T19:30:04.419 INFO:teuthology.orchestra.run.smithi098.stdout:HOST ADDR LABELS STATUS 2023-07-14T19:30:04.419 INFO:teuthology.orchestra.run.smithi098.stdout:smithi098 172.21.15.98 2023-07-14T19:30:04.419 INFO:teuthology.orchestra.run.smithi098.stdout:smithi101 172.21.15.101 2023-07-14T19:30:04.419 INFO:teuthology.orchestra.run.smithi098.stdout:smithi139 172.21.15.139 2023-07-14T19:30:04.419 INFO:teuthology.orchestra.run.smithi098.stdout:3 hosts in cluster 2023-07-14T19:30:04.420 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:04.420+0000 7fd1a73ef700 1 -- 172.21.15.98:0/1395715827 >> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] conn(0x7fd18c0a11e0 msgr2=0x7fd18c0a3690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:30:04.420 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:04.420+0000 7fd1a73ef700 1 --2- 172.21.15.98:0/1395715827 >> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] conn(0x7fd18c0a11e0 0x7fd18c0a3690 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7fd19c00b5c0 tx=0x7fd19c005fd0 comp rx=0 tx=0).stop 2023-07-14T19:30:04.421 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:04.420+0000 7fd1a73ef700 1 -- 172.21.15.98:0/1395715827 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7fd1a00fffa0 msgr2=0x7fd1a019cd70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:30:04.421 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:04.420+0000 7fd1a73ef700 1 --2- 172.21.15.98:0/1395715827 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7fd1a00fffa0 0x7fd1a019cd70 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7fd198007ee0 tx=0x7fd19800d3b0 comp rx=0 tx=0).stop 2023-07-14T19:30:04.421 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:04.420+0000 7fd1a73ef700 1 -- 172.21.15.98:0/1395715827 shutdown_connections 2023-07-14T19:30:04.421 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:04.420+0000 7fd1a73ef700 1 --2- 172.21.15.98:0/1395715827 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7fd1a00fffa0 0x7fd1a019cd70 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:30:04.421 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:04.420+0000 7fd1a73ef700 1 --2- 172.21.15.98:0/1395715827 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd1a00fe350 0x7fd1a0101680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:30:04.421 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:04.420+0000 7fd1a73ef700 1 --2- 172.21.15.98:0/1395715827 >> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] conn(0x7fd18c0a11e0 0x7fd18c0a3690 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:30:04.422 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:04.420+0000 7fd1a73ef700 1 --2- 172.21.15.98:0/1395715827 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fd1a00ff5f0 0x7fd1a019a730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:30:04.422 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:04.420+0000 7fd1a73ef700 1 -- 172.21.15.98:0/1395715827 >> 172.21.15.98:0/1395715827 conn(0x7fd1a00f99c0 msgr2=0x7fd1a00fbbd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:30:04.422 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:04.420+0000 7fd1a73ef700 1 -- 172.21.15.98:0/1395715827 shutdown_connections 2023-07-14T19:30:04.422 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:04.420+0000 7fd1a73ef700 1 -- 172.21.15.98:0/1395715827 wait complete. 2023-07-14T19:30:04.691 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:30:04 smithi101 ceph-mon[128807]: osdmap e85: 12 total, 12 up, 12 in 2023-07-14T19:30:04.691 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:30:04 smithi101 ceph-mon[128807]: pgmap v286: 97 pgs: 97 active+clean; 634 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 4 op/s 2023-07-14T19:30:04.712 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:30:04 smithi098 ceph-mon[127486]: osdmap e85: 12 total, 12 up, 12 in 2023-07-14T19:30:04.712 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:30:04 smithi098 ceph-mon[127486]: pgmap v286: 97 pgs: 97 active+clean; 634 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 4 op/s 2023-07-14T19:30:04.720 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:30:04 smithi139 ceph-mon[129293]: osdmap e85: 12 total, 12 up, 12 in 2023-07-14T19:30:04.720 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:30:04 smithi139 ceph-mon[129293]: pgmap v286: 97 pgs: 97 active+clean; 634 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 4 op/s 2023-07-14T19:30:04.760 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81c9387d2e7576da065747ec9c16d1c7d69016ec shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 9adea648-227b-11ee-9b33-001a4aab830c -- bash -c 'ceph orch device ls' 2023-07-14T19:30:05.315 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/9adea648-227b-11ee-9b33-001a4aab830c/mon.a/config 2023-07-14T19:30:05.781 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:30:05 smithi098 ceph-mon[127486]: from='client.24836 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""]}]: dispatch 2023-07-14T19:30:05.781 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:30:05 smithi098 ceph-mon[127486]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' 2023-07-14T19:30:05.781 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:30:05 smithi098 ceph-mon[127486]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' 2023-07-14T19:30:05.782 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:30:05 smithi098 ceph-mon[127486]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' 2023-07-14T19:30:05.782 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:30:05 smithi098 ceph-mon[127486]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' 2023-07-14T19:30:05.782 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:30:05 smithi098 ceph-mon[127486]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' 2023-07-14T19:30:05.782 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:30:05 smithi098 ceph-mon[127486]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' 2023-07-14T19:30:06.191 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:30:05 smithi101 ceph-mon[128807]: from='client.24836 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""]}]: dispatch 2023-07-14T19:30:06.191 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:30:05 smithi101 ceph-mon[128807]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' 2023-07-14T19:30:06.191 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:30:05 smithi101 ceph-mon[128807]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' 2023-07-14T19:30:06.192 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:30:05 smithi101 ceph-mon[128807]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' 2023-07-14T19:30:06.192 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:30:05 smithi101 ceph-mon[128807]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' 2023-07-14T19:30:06.192 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:30:05 smithi101 ceph-mon[128807]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' 2023-07-14T19:30:06.192 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:30:05 smithi101 ceph-mon[128807]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' 2023-07-14T19:30:06.224 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:30:05 smithi139 ceph-mon[129293]: from='client.24836 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""]}]: dispatch 2023-07-14T19:30:06.224 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:30:05 smithi139 ceph-mon[129293]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' 2023-07-14T19:30:06.224 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:30:05 smithi139 ceph-mon[129293]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' 2023-07-14T19:30:06.224 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:30:05 smithi139 ceph-mon[129293]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' 2023-07-14T19:30:06.224 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:30:05 smithi139 ceph-mon[129293]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' 2023-07-14T19:30:06.225 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:30:05 smithi139 ceph-mon[129293]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' 2023-07-14T19:30:06.225 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:30:05 smithi139 ceph-mon[129293]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' 2023-07-14T19:30:06.385 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:06.385+0000 7f71a252e700 1 -- 172.21.15.98:0/2373482791 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f719c0ffcd0 msgr2=0x7f719c1000e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:30:06.386 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:06.385+0000 7f71a252e700 1 --2- 172.21.15.98:0/2373482791 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f719c0ffcd0 0x7f719c1000e0 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7f7190009990 tx=0x7f7190009ca0 comp rx=0 tx=0).stop 2023-07-14T19:30:06.386 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:06.385+0000 7f71a252e700 1 -- 172.21.15.98:0/2373482791 shutdown_connections 2023-07-14T19:30:06.386 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:06.385+0000 7f71a252e700 1 --2- 172.21.15.98:0/2373482791 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f719c101920 0x7f719c1081c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:30:06.386 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:06.385+0000 7f71a252e700 1 --2- 172.21.15.98:0/2373482791 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f719c0ffcd0 0x7f719c1000e0 unknown :-1 s=CLOSED pgs=344 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:30:06.386 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:06.385+0000 7f71a252e700 1 --2- 172.21.15.98:0/2373482791 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f719c100f70 0x7f719c1013e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:30:06.387 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:06.385+0000 7f71a252e700 1 -- 172.21.15.98:0/2373482791 >> 172.21.15.98:0/2373482791 conn(0x7f719c0fb320 msgr2=0x7f719c0fd750 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:30:06.387 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:06.385+0000 7f71a252e700 1 -- 172.21.15.98:0/2373482791 shutdown_connections 2023-07-14T19:30:06.387 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:06.385+0000 7f71a252e700 1 -- 172.21.15.98:0/2373482791 wait complete. 2023-07-14T19:30:06.389 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:06.385+0000 7f71a252e700 1 Processor -- start 2023-07-14T19:30:06.389 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:06.385+0000 7f71a252e700 1 -- start start 2023-07-14T19:30:06.389 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:06.386+0000 7f71a252e700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f719c0ffcd0 0x7f719c195a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:30:06.389 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:06.386+0000 7f71a252e700 1 --2- >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f719c100f70 0x7f719c195f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:30:06.389 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:06.386+0000 7f71a252e700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f719c101920 0x7f719c19cfd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:30:06.390 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:06.386+0000 7f71a252e700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f719c19d5a0 con 0x7f719c101920 2023-07-14T19:30:06.390 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:06.386+0000 7f71a252e700 1 -- --> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f719c19d6e0 con 0x7f719c100f70 2023-07-14T19:30:06.390 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:06.386+0000 7f71a252e700 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f719c19d870 con 0x7f719c0ffcd0 2023-07-14T19:30:06.390 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:06.386+0000 7f71a0acb700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f719c101920 0x7f719c19cfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:30:06.390 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:06.386+0000 7f71a0acb700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f719c101920 0x7f719c19cfd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:46658/0 (socket says 172.21.15.98:46658) 2023-07-14T19:30:06.391 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:06.386+0000 7f71a0acb700 1 -- 172.21.15.98:0/3760470961 learned_addr learned my addr 172.21.15.98:0/3760470961 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-07-14T19:30:06.391 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:06.386+0000 7f719b7fe700 1 --2- 172.21.15.98:0/3760470961 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f719c100f70 0x7f719c195f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:30:06.391 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:06.386+0000 7f719bfff700 1 --2- 172.21.15.98:0/3760470961 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f719c0ffcd0 0x7f719c195a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:30:06.391 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:06.386+0000 7f71a0acb700 1 -- 172.21.15.98:0/3760470961 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f719c0ffcd0 msgr2=0x7f719c195a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:30:06.391 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:06.386+0000 7f71a0acb700 1 --2- 172.21.15.98:0/3760470961 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f719c0ffcd0 0x7f719c195a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:30:06.391 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:06.386+0000 7f71a0acb700 1 -- 172.21.15.98:0/3760470961 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f719c100f70 msgr2=0x7f719c195f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:30:06.392 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:06.386+0000 7f71a0acb700 1 --2- 172.21.15.98:0/3760470961 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f719c100f70 0x7f719c195f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:30:06.392 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:06.386+0000 7f71a0acb700 1 -- 172.21.15.98:0/3760470961 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f71900095f0 con 0x7f719c101920 2023-07-14T19:30:06.392 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:06.386+0000 7f71a0acb700 1 --2- 172.21.15.98:0/3760470961 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f719c101920 0x7f719c19cfd0 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7f719400b9a0 tx=0x7f719400bcb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:30:06.392 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:06.387+0000 7f71997fa700 1 -- 172.21.15.98:0/3760470961 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f719400c930 con 0x7f719c101920 2023-07-14T19:30:06.392 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:06.387+0000 7f71a252e700 1 -- 172.21.15.98:0/3760470961 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f719c19db50 con 0x7f719c101920 2023-07-14T19:30:06.392 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:06.387+0000 7f71997fa700 1 -- 172.21.15.98:0/3760470961 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f7194014440 con 0x7f719c101920 2023-07-14T19:30:06.392 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:06.387+0000 7f71997fa700 1 -- 172.21.15.98:0/3760470961 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f71940075d0 con 0x7f719c101920 2023-07-14T19:30:06.393 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:06.387+0000 7f71a252e700 1 -- 172.21.15.98:0/3760470961 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f719c19e050 con 0x7f719c101920 2023-07-14T19:30:06.393 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:06.388+0000 7f71a252e700 1 -- 172.21.15.98:0/3760470961 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f719c10b520 con 0x7f719c101920 2023-07-14T19:30:06.393 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:06.389+0000 7f71997fa700 1 -- 172.21.15.98:0/3760470961 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 17) v1 ==== 135766+0+0 (secure 0 0 0) 0x7f71940145b0 con 0x7f719c101920 2023-07-14T19:30:06.394 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:06.389+0000 7f71997fa700 1 --2- 172.21.15.98:0/3760470961 >> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] conn(0x7f71840a11e0 0x7f71840a3690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:30:06.394 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:06.389+0000 7f719bfff700 1 --2- 172.21.15.98:0/3760470961 >> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] conn(0x7f71840a11e0 0x7f71840a3690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:30:06.394 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:06.389+0000 7f71997fa700 1 -- 172.21.15.98:0/3760470961 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 8211+0+0 (secure 0 0 0) 0x7f7194006e40 con 0x7f719c101920 2023-07-14T19:30:06.394 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:06.389+0000 7f719bfff700 1 --2- 172.21.15.98:0/3760470961 >> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] conn(0x7f71840a11e0 0x7f71840a3690 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f7190006010 tx=0x7f719000b540 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:30:06.394 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:06.393+0000 7f71997fa700 1 -- 172.21.15.98:0/3760470961 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f719404e780 con 0x7f719c101920 2023-07-14T19:30:06.589 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:06.588+0000 7f71a252e700 1 -- 172.21.15.98:0/3760470961 --> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] -- mgr_command(tid 0: {"prefix": "orch device ls", "target": ["mon-mgr", ""]}) v1 -- 0x7f719c10b710 con 0x7f71840a11e0 2023-07-14T19:30:06.590 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:06.590+0000 7f71997fa700 1 -- 172.21.15.98:0/3760470961 <== mgr.14152 v2:172.21.15.98:6800/452042164 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+492 (secure 0 0 0) 0x7f719c10b710 con 0x7f71840a11e0 2023-07-14T19:30:06.590 INFO:teuthology.orchestra.run.smithi098.stdout:HOST PATH TYPE DEVICE ID SIZE AVAILABLE REFRESHED REJECT REASONS 2023-07-14T19:30:06.591 INFO:teuthology.orchestra.run.smithi098.stdout:smithi098 /dev/nvme0n1 ssd INTEL_SSDPEDMD400G4_PHFT6204010H400BGN 372G No 2m ago LVM detected, locked 2023-07-14T19:30:06.591 INFO:teuthology.orchestra.run.smithi098.stdout:smithi101 /dev/nvme0n1 ssd INTEL_SSDPEDMD400G4_PHFT6204017B400BGN 372G No 2m ago LVM detected, locked 2023-07-14T19:30:06.591 INFO:teuthology.orchestra.run.smithi098.stdout:smithi139 /dev/nvme0n1 ssd INTEL_SSDPEDMD400G4_PHFT620500N6400BGN 372G No 107s ago LVM detected, locked 2023-07-14T19:30:06.592 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:06.592+0000 7f71a252e700 1 -- 172.21.15.98:0/3760470961 >> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] conn(0x7f71840a11e0 msgr2=0x7f71840a3690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:30:06.593 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:06.592+0000 7f71a252e700 1 --2- 172.21.15.98:0/3760470961 >> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] conn(0x7f71840a11e0 0x7f71840a3690 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f7190006010 tx=0x7f719000b540 comp rx=0 tx=0).stop 2023-07-14T19:30:06.593 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:06.592+0000 7f71a252e700 1 -- 172.21.15.98:0/3760470961 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f719c101920 msgr2=0x7f719c19cfd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:30:06.593 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:06.592+0000 7f71a252e700 1 --2- 172.21.15.98:0/3760470961 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f719c101920 0x7f719c19cfd0 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7f719400b9a0 tx=0x7f719400bcb0 comp rx=0 tx=0).stop 2023-07-14T19:30:06.593 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:06.592+0000 7f71a252e700 1 -- 172.21.15.98:0/3760470961 shutdown_connections 2023-07-14T19:30:06.593 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:06.592+0000 7f71a252e700 1 --2- 172.21.15.98:0/3760470961 >> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] conn(0x7f71840a11e0 0x7f71840a3690 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:30:06.593 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:06.592+0000 7f71a252e700 1 --2- 172.21.15.98:0/3760470961 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f719c0ffcd0 0x7f719c195a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:30:06.593 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:06.592+0000 7f71a252e700 1 --2- 172.21.15.98:0/3760470961 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f719c101920 0x7f719c19cfd0 unknown :-1 s=CLOSED pgs=345 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:30:06.594 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:06.592+0000 7f71a252e700 1 --2- 172.21.15.98:0/3760470961 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f719c100f70 0x7f719c195f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:30:06.594 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:06.592+0000 7f71a252e700 1 -- 172.21.15.98:0/3760470961 >> 172.21.15.98:0/3760470961 conn(0x7f719c0fb320 msgr2=0x7f719c0fd520 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:30:06.594 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:06.592+0000 7f71a252e700 1 -- 172.21.15.98:0/3760470961 shutdown_connections 2023-07-14T19:30:06.594 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:06.592+0000 7f71a252e700 1 -- 172.21.15.98:0/3760470961 wait complete. 2023-07-14T19:30:06.928 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:30:06 smithi098 ceph-mon[127486]: pgmap v287: 97 pgs: 97 active+clean; 634 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s; 15 B/s, 0 objects/s recovering 2023-07-14T19:30:06.929 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:30:06 smithi098 ceph-mon[127486]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T19:30:06.929 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:30:06 smithi098 ceph-mon[127486]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-14T19:30:06.929 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:30:06 smithi098 ceph-mon[127486]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' 2023-07-14T19:30:06.960 INFO:teuthology.run_tasks:Running task cephadm.shell... 2023-07-14T19:30:06.971 INFO:tasks.cephadm:Running commands on role mon.a host ubuntu@smithi098.front.sepia.ceph.com 2023-07-14T19:30:06.972 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81c9387d2e7576da065747ec9c16d1c7d69016ec shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 9adea648-227b-11ee-9b33-001a4aab830c -- bash -c 'ceph fs dump' 2023-07-14T19:30:07.191 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:30:06 smithi101 ceph-mon[128807]: pgmap v287: 97 pgs: 97 active+clean; 634 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s; 15 B/s, 0 objects/s recovering 2023-07-14T19:30:07.192 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:30:06 smithi101 ceph-mon[128807]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T19:30:07.192 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:30:06 smithi101 ceph-mon[128807]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-14T19:30:07.192 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:30:06 smithi101 ceph-mon[128807]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' 2023-07-14T19:30:07.220 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:30:06 smithi139 ceph-mon[129293]: pgmap v287: 97 pgs: 97 active+clean; 634 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s; 15 B/s, 0 objects/s recovering 2023-07-14T19:30:07.221 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:30:06 smithi139 ceph-mon[129293]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T19:30:07.221 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:30:06 smithi139 ceph-mon[129293]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-14T19:30:07.222 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:30:06 smithi139 ceph-mon[129293]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' 2023-07-14T19:30:07.523 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/9adea648-227b-11ee-9b33-001a4aab830c/mon.a/config 2023-07-14T19:30:08.012 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:30:07 smithi098 ceph-mon[127486]: from='client.15132 -' entity='client.admin' cmd=[{"prefix": "orch device ls", "target": ["mon-mgr", ""]}]: dispatch 2023-07-14T19:30:08.191 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:30:07 smithi101 ceph-mon[128807]: from='client.15132 -' entity='client.admin' cmd=[{"prefix": "orch device ls", "target": ["mon-mgr", ""]}]: dispatch 2023-07-14T19:30:08.220 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:30:07 smithi139 ceph-mon[129293]: from='client.15132 -' entity='client.admin' cmd=[{"prefix": "orch device ls", "target": ["mon-mgr", ""]}]: dispatch 2023-07-14T19:30:08.660 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:08.659+0000 7fb9a72a8700 1 -- 172.21.15.98:0/2356862323 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb9a0100740 msgr2=0x7fb9a0107000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:30:08.661 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:08.659+0000 7fb9a72a8700 1 --2- 172.21.15.98:0/2356862323 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb9a0100740 0x7fb9a0107000 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7fb99800b0a0 tx=0x7fb99800b3b0 comp rx=0 tx=0).stop 2023-07-14T19:30:08.661 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:08.659+0000 7fb9a72a8700 1 -- 172.21.15.98:0/2356862323 shutdown_connections 2023-07-14T19:30:08.661 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:08.659+0000 7fb9a72a8700 1 --2- 172.21.15.98:0/2356862323 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb9a0100740 0x7fb9a0107000 unknown :-1 s=CLOSED pgs=346 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:30:08.661 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:08.659+0000 7fb9a72a8700 1 --2- 172.21.15.98:0/2356862323 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7fb9a00febb0 0x7fb9a00fefc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:30:08.661 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:08.659+0000 7fb9a72a8700 1 --2- 172.21.15.98:0/2356862323 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fb9a00ffdb0 0x7fb9a0100200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:30:08.662 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:08.659+0000 7fb9a72a8700 1 -- 172.21.15.98:0/2356862323 >> 172.21.15.98:0/2356862323 conn(0x7fb9a00fa1e0 msgr2=0x7fb9a00fc630 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:30:08.662 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:08.660+0000 7fb9a72a8700 1 -- 172.21.15.98:0/2356862323 shutdown_connections 2023-07-14T19:30:08.662 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:08.660+0000 7fb9a72a8700 1 -- 172.21.15.98:0/2356862323 wait complete. 2023-07-14T19:30:08.662 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:08.660+0000 7fb9a72a8700 1 Processor -- start 2023-07-14T19:30:08.662 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:08.660+0000 7fb9a72a8700 1 -- start start 2023-07-14T19:30:08.663 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:08.661+0000 7fb9a72a8700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fb9a00febb0 0x7fb9a0194990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:30:08.664 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:08.661+0000 7fb9a72a8700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb9a00ffdb0 0x7fb9a0194ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:30:08.664 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:08.661+0000 7fb9a72a8700 1 --2- >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7fb9a0100740 0x7fb9a019bf50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:30:08.664 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:08.661+0000 7fb9a72a8700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb9a019c520 con 0x7fb9a00ffdb0 2023-07-14T19:30:08.664 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:08.661+0000 7fb9a72a8700 1 -- --> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb9a019c660 con 0x7fb9a0100740 2023-07-14T19:30:08.664 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:08.661+0000 7fb9a72a8700 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb9a019c7f0 con 0x7fb9a00febb0 2023-07-14T19:30:08.664 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:08.661+0000 7fb9a4843700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb9a00ffdb0 0x7fb9a0194ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:30:08.665 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:08.661+0000 7fb9a5044700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fb9a00febb0 0x7fb9a0194990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:30:08.665 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:08.661+0000 7fb9a4843700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb9a00ffdb0 0x7fb9a0194ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:46668/0 (socket says 172.21.15.98:46668) 2023-07-14T19:30:08.665 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:08.661+0000 7fb9a4843700 1 -- 172.21.15.98:0/3592597354 learned_addr learned my addr 172.21.15.98:0/3592597354 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-07-14T19:30:08.665 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:08.661+0000 7fb9a5845700 1 --2- 172.21.15.98:0/3592597354 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7fb9a0100740 0x7fb9a019bf50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:30:08.665 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:08.661+0000 7fb9a4843700 1 -- 172.21.15.98:0/3592597354 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fb9a00febb0 msgr2=0x7fb9a0194990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:30:08.665 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:08.661+0000 7fb9a4843700 1 --2- 172.21.15.98:0/3592597354 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fb9a00febb0 0x7fb9a0194990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:30:08.665 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:08.661+0000 7fb9a4843700 1 -- 172.21.15.98:0/3592597354 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7fb9a0100740 msgr2=0x7fb9a019bf50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:30:08.665 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:08.661+0000 7fb9a4843700 1 --2- 172.21.15.98:0/3592597354 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7fb9a0100740 0x7fb9a019bf50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:30:08.666 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:08.661+0000 7fb9a4843700 1 -- 172.21.15.98:0/3592597354 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb998009d00 con 0x7fb9a00ffdb0 2023-07-14T19:30:08.666 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:08.661+0000 7fb9a4843700 1 --2- 172.21.15.98:0/3592597354 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb9a00ffdb0 0x7fb9a0194ed0 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7fb99400d980 tx=0x7fb99400dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:30:08.666 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:08.661+0000 7fb9927fc700 1 -- 172.21.15.98:0/3592597354 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb9940098d0 con 0x7fb9a00ffdb0 2023-07-14T19:30:08.666 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:08.661+0000 7fb9927fc700 1 -- 172.21.15.98:0/3592597354 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fb994010460 con 0x7fb9a00ffdb0 2023-07-14T19:30:08.666 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:08.661+0000 7fb9a72a8700 1 -- 172.21.15.98:0/3592597354 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb9a019cad0 con 0x7fb9a00ffdb0 2023-07-14T19:30:08.666 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:08.661+0000 7fb9927fc700 1 -- 172.21.15.98:0/3592597354 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb99400f620 con 0x7fb9a00ffdb0 2023-07-14T19:30:08.666 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:08.661+0000 7fb9a72a8700 1 -- 172.21.15.98:0/3592597354 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb9a019cfd0 con 0x7fb9a00ffdb0 2023-07-14T19:30:08.667 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:08.662+0000 7fb9a72a8700 1 -- 172.21.15.98:0/3592597354 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb9a0106150 con 0x7fb9a00ffdb0 2023-07-14T19:30:08.667 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:08.664+0000 7fb9927fc700 1 -- 172.21.15.98:0/3592597354 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 17) v1 ==== 135766+0+0 (secure 0 0 0) 0x7fb994010860 con 0x7fb9a00ffdb0 2023-07-14T19:30:08.667 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:08.664+0000 7fb9927fc700 1 --2- 172.21.15.98:0/3592597354 >> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] conn(0x7fb98c0a11e0 0x7fb98c0a3690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:30:08.667 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:08.665+0000 7fb9a5044700 1 --2- 172.21.15.98:0/3592597354 >> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] conn(0x7fb98c0a11e0 0x7fb98c0a3690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:30:08.668 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:08.665+0000 7fb9927fc700 1 -- 172.21.15.98:0/3592597354 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 8211+0+0 (secure 0 0 0) 0x7fb9940cc820 con 0x7fb9a00ffdb0 2023-07-14T19:30:08.668 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:08.665+0000 7fb9a5044700 1 --2- 172.21.15.98:0/3592597354 >> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] conn(0x7fb98c0a11e0 0x7fb98c0a3690 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7fb9a00ffc10 tx=0x7fb99c00a380 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:30:08.668 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:08.668+0000 7fb9927fc700 1 -- 172.21.15.98:0/3592597354 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7fb994095dc0 con 0x7fb9a00ffdb0 2023-07-14T19:30:08.898 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:08.898+0000 7fb9a72a8700 1 -- 172.21.15.98:0/3592597354 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7fb9a002c8e0 con 0x7fb9a00ffdb0 2023-07-14T19:30:08.899 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:08.898+0000 7fb9927fc700 1 -- 172.21.15.98:0/3592597354 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 22 v22) v1 ==== 76+0+2895 (secure 0 0 0) 0x7fb994020b60 con 0x7fb9a00ffdb0 2023-07-14T19:30:08.899 INFO:teuthology.orchestra.run.smithi098.stdout:e22 2023-07-14T19:30:08.899 INFO:teuthology.orchestra.run.smithi098.stdout:enable_multiple, ever_enabled_multiple: 1,1 2023-07-14T19:30:08.900 INFO:teuthology.orchestra.run.smithi098.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2023-07-14T19:30:08.900 INFO:teuthology.orchestra.run.smithi098.stdout:legacy client fscid: 1 2023-07-14T19:30:08.900 INFO:teuthology.orchestra.run.smithi098.stdout: 2023-07-14T19:30:08.900 INFO:teuthology.orchestra.run.smithi098.stdout:Filesystem 'cephfs' (1) 2023-07-14T19:30:08.900 INFO:teuthology.orchestra.run.smithi098.stdout:fs_name cephfs 2023-07-14T19:30:08.900 INFO:teuthology.orchestra.run.smithi098.stdout:epoch 22 2023-07-14T19:30:08.900 INFO:teuthology.orchestra.run.smithi098.stdout:flags 32 joinable allow_snaps allow_multimds_snaps allow_standby_replay 2023-07-14T19:30:08.901 INFO:teuthology.orchestra.run.smithi098.stdout:created 2023-07-14T19:29:01.033156+0000 2023-07-14T19:30:08.901 INFO:teuthology.orchestra.run.smithi098.stdout:modified 2023-07-14T19:29:06.291916+0000 2023-07-14T19:30:08.901 INFO:teuthology.orchestra.run.smithi098.stdout:tableserver 0 2023-07-14T19:30:08.901 INFO:teuthology.orchestra.run.smithi098.stdout:root 0 2023-07-14T19:30:08.901 INFO:teuthology.orchestra.run.smithi098.stdout:session_timeout 300 2023-07-14T19:30:08.901 INFO:teuthology.orchestra.run.smithi098.stdout:session_autoclose 300 2023-07-14T19:30:08.901 INFO:teuthology.orchestra.run.smithi098.stdout:max_file_size 1099511627776 2023-07-14T19:30:08.902 INFO:teuthology.orchestra.run.smithi098.stdout:max_xattr_size 65536 2023-07-14T19:30:08.902 INFO:teuthology.orchestra.run.smithi098.stdout:required_client_features {} 2023-07-14T19:30:08.902 INFO:teuthology.orchestra.run.smithi098.stdout:last_failure 0 2023-07-14T19:30:08.902 INFO:teuthology.orchestra.run.smithi098.stdout:last_failure_osd_epoch 0 2023-07-14T19:30:08.902 INFO:teuthology.orchestra.run.smithi098.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2023-07-14T19:30:08.902 INFO:teuthology.orchestra.run.smithi098.stdout:max_mds 3 2023-07-14T19:30:08.902 INFO:teuthology.orchestra.run.smithi098.stdout:in 0,1,2 2023-07-14T19:30:08.903 INFO:teuthology.orchestra.run.smithi098.stdout:up {0=24521,1=24493,2=24479} 2023-07-14T19:30:08.903 INFO:teuthology.orchestra.run.smithi098.stdout:failed 2023-07-14T19:30:08.903 INFO:teuthology.orchestra.run.smithi098.stdout:damaged 2023-07-14T19:30:08.903 INFO:teuthology.orchestra.run.smithi098.stdout:stopped 2023-07-14T19:30:08.903 INFO:teuthology.orchestra.run.smithi098.stdout:data_pools [3] 2023-07-14T19:30:08.904 INFO:teuthology.orchestra.run.smithi098.stdout:metadata_pool 2 2023-07-14T19:30:08.904 INFO:teuthology.orchestra.run.smithi098.stdout:inline_data disabled 2023-07-14T19:30:08.904 INFO:teuthology.orchestra.run.smithi098.stdout:balancer 2023-07-14T19:30:08.904 INFO:teuthology.orchestra.run.smithi098.stdout:bal_rank_mask -1 2023-07-14T19:30:08.904 INFO:teuthology.orchestra.run.smithi098.stdout:standby_count_wanted 0 2023-07-14T19:30:08.904 INFO:teuthology.orchestra.run.smithi098.stdout:[mds.b{0:24521} state up:active seq 3 addr [v2:172.21.15.101:6838/48393381,v1:172.21.15.101:6839/48393381] compat {c=[1],r=[1],i=[7ff]}] 2023-07-14T19:30:08.904 INFO:teuthology.orchestra.run.smithi098.stdout:[mds.c{0:24469} state up:standby-replay seq 1 addr [v2:172.21.15.139:6836/3597660552,v1:172.21.15.139:6837/3597660552] compat {c=[1],r=[1],i=[7ff]}] 2023-07-14T19:30:08.905 INFO:teuthology.orchestra.run.smithi098.stdout:[mds.i{1:24493} state up:active seq 5 addr [v2:172.21.15.139:6838/1258854143,v1:172.21.15.139:6839/1258854143] compat {c=[1],r=[1],i=[7ff]}] 2023-07-14T19:30:08.905 INFO:teuthology.orchestra.run.smithi098.stdout:[mds.f{1:24460} state up:standby-replay seq 1 addr [v2:172.21.15.139:6834/2315281724,v1:172.21.15.139:6835/2315281724] compat {c=[1],r=[1],i=[7ff]}] 2023-07-14T19:30:08.905 INFO:teuthology.orchestra.run.smithi098.stdout:[mds.h{2:24479} state up:active seq 6 addr [v2:172.21.15.101:6836/485741528,v1:172.21.15.101:6837/485741528] compat {c=[1],r=[1],i=[7ff]}] 2023-07-14T19:30:08.905 INFO:teuthology.orchestra.run.smithi098.stdout:[mds.e{2:24455} state up:standby-replay seq 1 addr [v2:172.21.15.101:6834/1583264648,v1:172.21.15.101:6835/1583264648] compat {c=[1],r=[1],i=[7ff]}] 2023-07-14T19:30:08.905 INFO:teuthology.orchestra.run.smithi098.stdout: 2023-07-14T19:30:08.905 INFO:teuthology.orchestra.run.smithi098.stdout: 2023-07-14T19:30:08.905 INFO:teuthology.orchestra.run.smithi098.stdout:Standby daemons: 2023-07-14T19:30:08.905 INFO:teuthology.orchestra.run.smithi098.stdout: 2023-07-14T19:30:08.906 INFO:teuthology.orchestra.run.smithi098.stdout:[mds.j{-1:14565} state up:standby seq 1 addr [v2:172.21.15.98:6834/586280675,v1:172.21.15.98:6835/586280675] compat {c=[1],r=[1],i=[7ff]}] 2023-07-14T19:30:08.906 INFO:teuthology.orchestra.run.smithi098.stdout:[mds.d{-1:14577} state up:standby seq 1 addr [v2:172.21.15.98:6836/487577257,v1:172.21.15.98:6837/487577257] compat {c=[1],r=[1],i=[7ff]}] 2023-07-14T19:30:08.906 INFO:teuthology.orchestra.run.smithi098.stdout:[mds.g{-1:14634} state up:standby seq 1 addr [v2:172.21.15.98:6838/4085731209,v1:172.21.15.98:6839/4085731209] compat {c=[1],r=[1],i=[7ff]}] 2023-07-14T19:30:08.906 INFO:teuthology.orchestra.run.smithi098.stdout:[mds.a{-1:14646} state up:standby seq 1 addr [v2:172.21.15.98:6840/1679318259,v1:172.21.15.98:6841/1679318259] compat {c=[1],r=[1],i=[7ff]}] 2023-07-14T19:30:08.906 INFO:teuthology.orchestra.run.smithi098.stdout:[mds.l{-1:24421} state up:standby seq 1 addr [v2:172.21.15.139:6832/3505477820,v1:172.21.15.139:6833/3505477820] compat {c=[1],r=[1],i=[7ff]}] 2023-07-14T19:30:08.906 INFO:teuthology.orchestra.run.smithi098.stdout:[mds.k{-1:24437} state up:standby seq 1 addr [v2:172.21.15.101:6832/3474460171,v1:172.21.15.101:6833/3474460171] compat {c=[1],r=[1],i=[7ff]}] 2023-07-14T19:30:08.907 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:08.900+0000 7fb9a72a8700 1 -- 172.21.15.98:0/3592597354 >> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] conn(0x7fb98c0a11e0 msgr2=0x7fb98c0a3690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:30:08.907 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:08.900+0000 7fb9a72a8700 1 --2- 172.21.15.98:0/3592597354 >> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] conn(0x7fb98c0a11e0 0x7fb98c0a3690 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7fb9a00ffc10 tx=0x7fb99c00a380 comp rx=0 tx=0).stop 2023-07-14T19:30:08.907 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:08.900+0000 7fb9a72a8700 1 -- 172.21.15.98:0/3592597354 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb9a00ffdb0 msgr2=0x7fb9a0194ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:30:08.907 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:08.900+0000 7fb9a72a8700 1 --2- 172.21.15.98:0/3592597354 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb9a00ffdb0 0x7fb9a0194ed0 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7fb99400d980 tx=0x7fb99400dc90 comp rx=0 tx=0).stop 2023-07-14T19:30:08.908 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:08.900+0000 7fb9a72a8700 1 -- 172.21.15.98:0/3592597354 shutdown_connections 2023-07-14T19:30:08.908 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:08.900+0000 7fb9a72a8700 1 --2- 172.21.15.98:0/3592597354 >> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] conn(0x7fb98c0a11e0 0x7fb98c0a3690 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:30:08.908 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:08.900+0000 7fb9a72a8700 1 --2- 172.21.15.98:0/3592597354 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fb9a00febb0 0x7fb9a0194990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:30:08.908 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:08.900+0000 7fb9a72a8700 1 --2- 172.21.15.98:0/3592597354 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7fb9a0100740 0x7fb9a019bf50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:30:08.908 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:08.900+0000 7fb9a72a8700 1 --2- 172.21.15.98:0/3592597354 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb9a00ffdb0 0x7fb9a0194ed0 unknown :-1 s=CLOSED pgs=347 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:30:08.908 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:08.900+0000 7fb9a72a8700 1 -- 172.21.15.98:0/3592597354 >> 172.21.15.98:0/3592597354 conn(0x7fb9a00fa1e0 msgr2=0x7fb9a0103290 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:30:08.908 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:08.901+0000 7fb9a72a8700 1 -- 172.21.15.98:0/3592597354 shutdown_connections 2023-07-14T19:30:08.908 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:08.901+0000 7fb9a72a8700 1 -- 172.21.15.98:0/3592597354 wait complete. 2023-07-14T19:30:08.909 INFO:teuthology.orchestra.run.smithi098.stderr:dumped fsmap epoch 22 2023-07-14T19:30:09.028 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:30:08 smithi098 ceph-mon[127486]: pgmap v288: 97 pgs: 97 active+clean; 634 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s; 18 B/s, 0 objects/s recovering 2023-07-14T19:30:09.191 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:30:08 smithi101 ceph-mon[128807]: pgmap v288: 97 pgs: 97 active+clean; 634 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s; 18 B/s, 0 objects/s recovering 2023-07-14T19:30:09.220 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:30:08 smithi139 ceph-mon[129293]: pgmap v288: 97 pgs: 97 active+clean; 634 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s; 18 B/s, 0 objects/s recovering 2023-07-14T19:30:09.263 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81c9387d2e7576da065747ec9c16d1c7d69016ec shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 9adea648-227b-11ee-9b33-001a4aab830c -- bash -c 'ceph osd dump' 2023-07-14T19:30:09.816 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/9adea648-227b-11ee-9b33-001a4aab830c/mon.a/config 2023-07-14T19:30:10.017 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:30:09 smithi098 ceph-mon[127486]: from='client.? 172.21.15.98:0/3592597354' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2023-07-14T19:30:10.191 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:30:09 smithi101 ceph-mon[128807]: from='client.? 172.21.15.98:0/3592597354' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2023-07-14T19:30:10.220 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:30:09 smithi139 ceph-mon[129293]: from='client.? 172.21.15.98:0/3592597354' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2023-07-14T19:30:10.924 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:10.922+0000 7f2e94cea700 1 -- 172.21.15.98:0/2919385261 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2e900ffed0 msgr2=0x7f2e90106790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:30:10.924 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:10.922+0000 7f2e94cea700 1 --2- 172.21.15.98:0/2919385261 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2e900ffed0 0x7f2e90106790 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7f2e8800c210 tx=0x7f2e8800c520 comp rx=0 tx=0).stop 2023-07-14T19:30:10.925 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:10.922+0000 7f2e94cea700 1 -- 172.21.15.98:0/2919385261 shutdown_connections 2023-07-14T19:30:10.925 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:10.922+0000 7f2e94cea700 1 --2- 172.21.15.98:0/2919385261 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f2e900fe340 0x7f2e900fe750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:30:10.925 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:10.922+0000 7f2e94cea700 1 --2- 172.21.15.98:0/2919385261 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2e900ffed0 0x7f2e90106790 unknown :-1 s=CLOSED pgs=348 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:30:10.925 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:10.922+0000 7f2e94cea700 1 --2- 172.21.15.98:0/2919385261 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f2e900ff540 0x7f2e900ff990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:30:10.925 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:10.922+0000 7f2e94cea700 1 -- 172.21.15.98:0/2919385261 >> 172.21.15.98:0/2919385261 conn(0x7f2e900f9990 msgr2=0x7f2e900fbdc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:30:10.925 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:10.922+0000 7f2e94cea700 1 -- 172.21.15.98:0/2919385261 shutdown_connections 2023-07-14T19:30:10.925 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:10.922+0000 7f2e94cea700 1 -- 172.21.15.98:0/2919385261 wait complete. 2023-07-14T19:30:10.925 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:10.922+0000 7f2e94cea700 1 Processor -- start 2023-07-14T19:30:10.926 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:10.923+0000 7f2e94cea700 1 -- start start 2023-07-14T19:30:10.926 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:10.923+0000 7f2e94cea700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f2e900fe340 0x7f2e901950a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:30:10.926 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:10.923+0000 7f2e94cea700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2e900ff540 0x7f2e901936f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:30:10.926 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:10.923+0000 7f2e94cea700 1 --2- >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f2e900ffed0 0x7f2e90193c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:30:10.926 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:10.923+0000 7f2e94cea700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2e901943b0 con 0x7f2e900ff540 2023-07-14T19:30:10.926 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:10.923+0000 7f2e94cea700 1 -- --> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2e90194520 con 0x7f2e900ffed0 2023-07-14T19:30:10.926 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:10.923+0000 7f2e94cea700 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2e9019fb80 con 0x7f2e900fe340 2023-07-14T19:30:10.927 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:10.923+0000 7f2e8dd9b700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2e900ff540 0x7f2e901936f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:30:10.927 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:10.923+0000 7f2e8dd9b700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2e900ff540 0x7f2e901936f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:46682/0 (socket says 172.21.15.98:46682) 2023-07-14T19:30:10.927 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:10.923+0000 7f2e8dd9b700 1 -- 172.21.15.98:0/2117490361 learned_addr learned my addr 172.21.15.98:0/2117490361 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-07-14T19:30:10.927 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:10.923+0000 7f2e8ed9d700 1 --2- 172.21.15.98:0/2117490361 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f2e900ffed0 0x7f2e90193c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:30:10.927 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:10.923+0000 7f2e8e59c700 1 --2- 172.21.15.98:0/2117490361 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f2e900fe340 0x7f2e901950a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:30:10.927 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:10.923+0000 7f2e8dd9b700 1 -- 172.21.15.98:0/2117490361 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f2e900fe340 msgr2=0x7f2e901950a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:30:10.927 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:10.923+0000 7f2e8dd9b700 1 --2- 172.21.15.98:0/2117490361 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f2e900fe340 0x7f2e901950a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:30:10.927 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:10.923+0000 7f2e8dd9b700 1 -- 172.21.15.98:0/2117490361 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f2e900ffed0 msgr2=0x7f2e90193c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:30:10.928 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:10.923+0000 7f2e8dd9b700 1 --2- 172.21.15.98:0/2117490361 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f2e900ffed0 0x7f2e90193c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:30:10.928 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:10.923+0000 7f2e8dd9b700 1 -- 172.21.15.98:0/2117490361 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2e8800ae10 con 0x7f2e900ff540 2023-07-14T19:30:10.928 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:10.923+0000 7f2e8dd9b700 1 --2- 172.21.15.98:0/2117490361 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2e900ff540 0x7f2e901936f0 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7f2e8400b530 tx=0x7f2e8400b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:30:10.928 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:10.923+0000 7f2e7f7fe700 1 -- 172.21.15.98:0/2117490361 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2e84011830 con 0x7f2e900ff540 2023-07-14T19:30:10.928 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:10.923+0000 7f2e7f7fe700 1 -- 172.21.15.98:0/2117490361 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f2e84011e50 con 0x7f2e900ff540 2023-07-14T19:30:10.928 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:10.923+0000 7f2e7f7fe700 1 -- 172.21.15.98:0/2117490361 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2e8400f2e0 con 0x7f2e900ff540 2023-07-14T19:30:10.928 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:10.924+0000 7f2e94cea700 1 -- 172.21.15.98:0/2117490361 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2e9019fe60 con 0x7f2e900ff540 2023-07-14T19:30:10.929 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:10.924+0000 7f2e94cea700 1 -- 172.21.15.98:0/2117490361 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2e901a0300 con 0x7f2e900ff540 2023-07-14T19:30:10.929 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:10.924+0000 7f2e94cea700 1 -- 172.21.15.98:0/2117490361 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2e90109310 con 0x7f2e900ff540 2023-07-14T19:30:10.930 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:10.925+0000 7f2e7f7fe700 1 -- 172.21.15.98:0/2117490361 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 17) v1 ==== 135766+0+0 (secure 0 0 0) 0x7f2e840105a0 con 0x7f2e900ff540 2023-07-14T19:30:10.930 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:10.926+0000 7f2e7f7fe700 1 --2- 172.21.15.98:0/2117490361 >> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] conn(0x7f2e780a11e0 0x7f2e780a3690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:30:10.930 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:10.926+0000 7f2e8e59c700 1 --2- 172.21.15.98:0/2117490361 >> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] conn(0x7f2e780a11e0 0x7f2e780a3690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:30:10.930 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:10.926+0000 7f2e7f7fe700 1 -- 172.21.15.98:0/2117490361 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 8211+0+0 (secure 0 0 0) 0x7f2e840cc820 con 0x7f2e900ff540 2023-07-14T19:30:10.930 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:10.926+0000 7f2e8e59c700 1 --2- 172.21.15.98:0/2117490361 >> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] conn(0x7f2e780a11e0 0x7f2e780a3690 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f2e900ff3a0 tx=0x7f2e8000a380 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:30:10.930 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:10.929+0000 7f2e7f7fe700 1 -- 172.21.15.98:0/2117490361 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f2e84095f30 con 0x7f2e900ff540 2023-07-14T19:30:11.031 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:30:10 smithi098 ceph-mon[127486]: pgmap v289: 97 pgs: 97 active+clean; 634 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 6 op/s; 18 B/s, 0 objects/s recovering 2023-07-14T19:30:11.121 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:11.120+0000 7f2e94cea700 1 -- 172.21.15.98:0/2117490361 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd dump"} v 0) v1 -- 0x7f2e901051b0 con 0x7f2e900ff540 2023-07-14T19:30:11.122 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:11.121+0000 7f2e7f7fe700 1 -- 172.21.15.98:0/2117490361 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump"}]=0 v85) v1 ==== 56+0+5278 (secure 0 0 0) 0x7f2e84099550 con 0x7f2e900ff540 2023-07-14T19:30:11.122 INFO:teuthology.orchestra.run.smithi098.stdout:epoch 85 2023-07-14T19:30:11.122 INFO:teuthology.orchestra.run.smithi098.stdout:fsid 9adea648-227b-11ee-9b33-001a4aab830c 2023-07-14T19:30:11.122 INFO:teuthology.orchestra.run.smithi098.stdout:created 2023-07-14T19:22:07.178854+0000 2023-07-14T19:30:11.122 INFO:teuthology.orchestra.run.smithi098.stdout:modified 2023-07-14T19:30:03.314948+0000 2023-07-14T19:30:11.123 INFO:teuthology.orchestra.run.smithi098.stdout:flags sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit 2023-07-14T19:30:11.123 INFO:teuthology.orchestra.run.smithi098.stdout:crush_version 26 2023-07-14T19:30:11.123 INFO:teuthology.orchestra.run.smithi098.stdout:full_ratio 0.95 2023-07-14T19:30:11.123 INFO:teuthology.orchestra.run.smithi098.stdout:backfillfull_ratio 0.9 2023-07-14T19:30:11.123 INFO:teuthology.orchestra.run.smithi098.stdout:nearfull_ratio 0.85 2023-07-14T19:30:11.123 INFO:teuthology.orchestra.run.smithi098.stdout:require_min_compat_client luminous 2023-07-14T19:30:11.123 INFO:teuthology.orchestra.run.smithi098.stdout:min_compat_client luminous 2023-07-14T19:30:11.123 INFO:teuthology.orchestra.run.smithi098.stdout:require_osd_release reef 2023-07-14T19:30:11.124 INFO:teuthology.orchestra.run.smithi098.stdout:stretch_mode_enabled false 2023-07-14T19:30:11.124 INFO:teuthology.orchestra.run.smithi098.stdout:pool 1 '.mgr' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 1 pgp_num 1 autoscale_mode off last_change 22 flags hashpspool stripe_width 0 pg_num_max 32 pg_num_min 1 application mgr read_balance_score 12.00 2023-07-14T19:30:11.124 INFO:teuthology.orchestra.run.smithi098.stdout:pool 2 'cephfs_metadata' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode off last_change 82 flags hashpspool stripe_width 0 pg_autoscale_bias 4 pg_num_min 16 recovery_priority 5 application cephfs read_balance_score 1.88 2023-07-14T19:30:11.124 INFO:teuthology.orchestra.run.smithi098.stdout:pool 3 'cephfs_data' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 64 pgp_num 64 autoscale_mode off last_change 83 flags hashpspool stripe_width 0 pg_num_min 64 target_size_ratio 0.9 application cephfs read_balance_score 1.69 2023-07-14T19:30:11.124 INFO:teuthology.orchestra.run.smithi098.stdout:max_osd 12 2023-07-14T19:30:11.124 INFO:teuthology.orchestra.run.smithi098.stdout:osd.0 up in weight 1 up_from 8 up_thru 84 down_at 0 last_clean_interval [0,0) [v2:172.21.15.98:6802/120507228,v1:172.21.15.98:6803/120507228] [v2:172.21.15.98:6804/120507228,v1:172.21.15.98:6805/120507228] exists,up a3797e89-1754-41cf-9517-227e06ffdd72 2023-07-14T19:30:11.125 INFO:teuthology.orchestra.run.smithi098.stdout:osd.1 up in weight 1 up_from 13 up_thru 81 down_at 0 last_clean_interval [0,0) [v2:172.21.15.101:6800/2450584584,v1:172.21.15.101:6801/2450584584] [v2:172.21.15.101:6802/2450584584,v1:172.21.15.101:6803/2450584584] exists,up 2b001add-1aaa-4496-a403-7ef9725b80a0 2023-07-14T19:30:11.125 INFO:teuthology.orchestra.run.smithi098.stdout:osd.2 up in weight 1 up_from 18 up_thru 81 down_at 0 last_clean_interval [0,0) [v2:172.21.15.139:6800/945038995,v1:172.21.15.139:6801/945038995] [v2:172.21.15.139:6802/945038995,v1:172.21.15.139:6803/945038995] exists,up 48e6d499-3ca9-424b-b0fb-3ba5f33b9fc9 2023-07-14T19:30:11.125 INFO:teuthology.orchestra.run.smithi098.stdout:osd.3 up in weight 1 up_from 25 up_thru 81 down_at 0 last_clean_interval [0,0) [v2:172.21.15.98:6810/3095249665,v1:172.21.15.98:6811/3095249665] [v2:172.21.15.98:6812/3095249665,v1:172.21.15.98:6813/3095249665] exists,up 5530b265-31a5-447b-a1ea-17ebae9cfddf 2023-07-14T19:30:11.125 INFO:teuthology.orchestra.run.smithi098.stdout:osd.4 up in weight 1 up_from 32 up_thru 81 down_at 0 last_clean_interval [0,0) [v2:172.21.15.101:6808/2844761406,v1:172.21.15.101:6809/2844761406] [v2:172.21.15.101:6810/2844761406,v1:172.21.15.101:6811/2844761406] exists,up 615d4642-a997-4f95-9e38-9fbdaa831400 2023-07-14T19:30:11.125 INFO:teuthology.orchestra.run.smithi098.stdout:osd.5 up in weight 1 up_from 37 up_thru 84 down_at 0 last_clean_interval [0,0) [v2:172.21.15.139:6808/3532263042,v1:172.21.15.139:6809/3532263042] [v2:172.21.15.139:6810/3532263042,v1:172.21.15.139:6811/3532263042] exists,up 415de5fa-607a-45a8-b732-28daf3be6dd3 2023-07-14T19:30:11.125 INFO:teuthology.orchestra.run.smithi098.stdout:osd.6 up in weight 1 up_from 45 up_thru 84 down_at 0 last_clean_interval [0,0) [v2:172.21.15.98:6818/658948743,v1:172.21.15.98:6819/658948743] [v2:172.21.15.98:6820/658948743,v1:172.21.15.98:6821/658948743] exists,up d7f9d4ba-c309-4b75-a836-5a36384b528e 2023-07-14T19:30:11.125 INFO:teuthology.orchestra.run.smithi098.stdout:osd.7 up in weight 1 up_from 50 up_thru 84 down_at 0 last_clean_interval [0,0) [v2:172.21.15.101:6816/2881214066,v1:172.21.15.101:6817/2881214066] [v2:172.21.15.101:6818/2881214066,v1:172.21.15.101:6819/2881214066] exists,up eedb94d9-c5cd-4a93-8a8c-2648804b7195 2023-07-14T19:30:11.125 INFO:teuthology.orchestra.run.smithi098.stdout:osd.8 up in weight 1 up_from 57 up_thru 81 down_at 0 last_clean_interval [0,0) [v2:172.21.15.139:6816/3669514524,v1:172.21.15.139:6817/3669514524] [v2:172.21.15.139:6818/3669514524,v1:172.21.15.139:6819/3669514524] exists,up 6bfd77c2-5770-466e-aa6f-84b50c74ffb6 2023-07-14T19:30:11.126 INFO:teuthology.orchestra.run.smithi098.stdout:osd.9 up in weight 1 up_from 65 up_thru 84 down_at 0 last_clean_interval [0,0) [v2:172.21.15.98:6826/2893994080,v1:172.21.15.98:6827/2893994080] [v2:172.21.15.98:6828/2893994080,v1:172.21.15.98:6829/2893994080] exists,up 19c5b54a-2242-4b23-b721-63be9559653d 2023-07-14T19:30:11.126 INFO:teuthology.orchestra.run.smithi098.stdout:osd.10 up in weight 1 up_from 70 up_thru 81 down_at 0 last_clean_interval [0,0) [v2:172.21.15.101:6824/3545244225,v1:172.21.15.101:6825/3545244225] [v2:172.21.15.101:6826/3545244225,v1:172.21.15.101:6827/3545244225] exists,up 326f0209-ba9d-4b59-97cc-d61a758a8f29 2023-07-14T19:30:11.126 INFO:teuthology.orchestra.run.smithi098.stdout:osd.11 up in weight 1 up_from 78 up_thru 84 down_at 0 last_clean_interval [0,0) [v2:172.21.15.139:6824/2630023454,v1:172.21.15.139:6825/2630023454] [v2:172.21.15.139:6826/2630023454,v1:172.21.15.139:6827/2630023454] exists,up c61c0320-1057-4f17-9bf1-7ad306ed7ed3 2023-07-14T19:30:11.126 INFO:teuthology.orchestra.run.smithi098.stdout:pg_upmap_items 2.4 [2,11] 2023-07-14T19:30:11.126 INFO:teuthology.orchestra.run.smithi098.stdout:pg_upmap_items 2.f [8,11] 2023-07-14T19:30:11.126 INFO:teuthology.orchestra.run.smithi098.stdout:pg_upmap_items 3.0 [4,8] 2023-07-14T19:30:11.126 INFO:teuthology.orchestra.run.smithi098.stdout:pg_upmap_items 3.1 [4,9] 2023-07-14T19:30:11.127 INFO:teuthology.orchestra.run.smithi098.stdout:pg_upmap_items 3.c [3,9] 2023-07-14T19:30:11.127 INFO:teuthology.orchestra.run.smithi098.stdout:pg_upmap_items 3.17 [4,6] 2023-07-14T19:30:11.127 INFO:teuthology.orchestra.run.smithi098.stdout:pg_upmap_items 3.1e [3,9] 2023-07-14T19:30:11.127 INFO:teuthology.orchestra.run.smithi098.stdout:pg_upmap_items 3.28 [5,7] 2023-07-14T19:30:11.127 INFO:teuthology.orchestra.run.smithi098.stdout:blocklist 172.21.15.98:0/2164590253 expires 2023-07-15T19:23:01.520302+0000 2023-07-14T19:30:11.127 INFO:teuthology.orchestra.run.smithi098.stdout:blocklist 172.21.15.98:0/2094734896 expires 2023-07-15T19:23:01.520302+0000 2023-07-14T19:30:11.127 INFO:teuthology.orchestra.run.smithi098.stdout:blocklist 172.21.15.98:0/784745009 expires 2023-07-15T19:22:39.481502+0000 2023-07-14T19:30:11.127 INFO:teuthology.orchestra.run.smithi098.stdout:blocklist 172.21.15.98:0/3657190818 expires 2023-07-15T19:22:39.481502+0000 2023-07-14T19:30:11.128 INFO:teuthology.orchestra.run.smithi098.stdout:blocklist 172.21.15.98:6800/1732769087 expires 2023-07-15T19:23:01.520302+0000 2023-07-14T19:30:11.128 INFO:teuthology.orchestra.run.smithi098.stdout:blocklist 172.21.15.98:0/3393542419 expires 2023-07-15T19:22:39.481502+0000 2023-07-14T19:30:11.128 INFO:teuthology.orchestra.run.smithi098.stdout:blocklist 172.21.15.98:6800/749234665 expires 2023-07-15T19:22:39.481502+0000 2023-07-14T19:30:11.128 INFO:teuthology.orchestra.run.smithi098.stdout:blocklist 172.21.15.98:6801/1732769087 expires 2023-07-15T19:23:01.520302+0000 2023-07-14T19:30:11.128 INFO:teuthology.orchestra.run.smithi098.stdout:blocklist 172.21.15.98:6801/749234665 expires 2023-07-15T19:22:39.481502+0000 2023-07-14T19:30:11.128 INFO:teuthology.orchestra.run.smithi098.stdout:blocklist 172.21.15.98:0/187822873 expires 2023-07-15T19:23:01.520302+0000 2023-07-14T19:30:11.129 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:11.123+0000 7f2e94cea700 1 -- 172.21.15.98:0/2117490361 >> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] conn(0x7f2e780a11e0 msgr2=0x7f2e780a3690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:30:11.129 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:11.123+0000 7f2e94cea700 1 --2- 172.21.15.98:0/2117490361 >> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] conn(0x7f2e780a11e0 0x7f2e780a3690 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f2e900ff3a0 tx=0x7f2e8000a380 comp rx=0 tx=0).stop 2023-07-14T19:30:11.129 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:11.123+0000 7f2e94cea700 1 -- 172.21.15.98:0/2117490361 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2e900ff540 msgr2=0x7f2e901936f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:30:11.129 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:11.123+0000 7f2e94cea700 1 --2- 172.21.15.98:0/2117490361 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2e900ff540 0x7f2e901936f0 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7f2e8400b530 tx=0x7f2e8400b840 comp rx=0 tx=0).stop 2023-07-14T19:30:11.129 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:11.123+0000 7f2e94cea700 1 -- 172.21.15.98:0/2117490361 shutdown_connections 2023-07-14T19:30:11.130 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:11.123+0000 7f2e94cea700 1 --2- 172.21.15.98:0/2117490361 >> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] conn(0x7f2e780a11e0 0x7f2e780a3690 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:30:11.130 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:11.123+0000 7f2e94cea700 1 --2- 172.21.15.98:0/2117490361 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f2e900fe340 0x7f2e901950a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:30:11.130 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:11.123+0000 7f2e94cea700 1 --2- 172.21.15.98:0/2117490361 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f2e900ffed0 0x7f2e90193c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:30:11.130 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:11.123+0000 7f2e94cea700 1 --2- 172.21.15.98:0/2117490361 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2e900ff540 0x7f2e901936f0 unknown :-1 s=CLOSED pgs=349 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:30:11.130 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:11.123+0000 7f2e94cea700 1 -- 172.21.15.98:0/2117490361 >> 172.21.15.98:0/2117490361 conn(0x7f2e900f9990 msgr2=0x7f2e900fbb40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:30:11.130 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:11.124+0000 7f2e94cea700 1 -- 172.21.15.98:0/2117490361 shutdown_connections 2023-07-14T19:30:11.130 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:11.124+0000 7f2e94cea700 1 -- 172.21.15.98:0/2117490361 wait complete. 2023-07-14T19:30:11.191 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:30:10 smithi101 ceph-mon[128807]: pgmap v289: 97 pgs: 97 active+clean; 634 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 6 op/s; 18 B/s, 0 objects/s recovering 2023-07-14T19:30:11.220 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:30:10 smithi139 ceph-mon[129293]: pgmap v289: 97 pgs: 97 active+clean; 634 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 6 op/s; 18 B/s, 0 objects/s recovering 2023-07-14T19:30:11.560 INFO:teuthology.run_tasks:Running task fs.ready... 2023-07-14T19:30:11.571 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-07-14T19:30:11.700 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:11.696+0000 7fc610692700 1 -- 172.21.15.98:0/1861825925 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc6080feb00 msgr2=0x7fc6080fef10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:30:11.701 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:11.696+0000 7fc610692700 1 --2- 172.21.15.98:0/1861825925 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc6080feb00 0x7fc6080fef10 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7fc604009990 tx=0x7fc604009ca0 comp rx=0 tx=0).stop 2023-07-14T19:30:11.701 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:11.697+0000 7fc610692700 1 -- 172.21.15.98:0/1861825925 shutdown_connections 2023-07-14T19:30:11.701 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:11.697+0000 7fc610692700 1 --2- 172.21.15.98:0/1861825925 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7fc608100680 0x7fc608106f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:30:11.702 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:11.697+0000 7fc610692700 1 --2- 172.21.15.98:0/1861825925 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc6080feb00 0x7fc6080fef10 unknown :-1 s=CLOSED pgs=350 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:30:11.702 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:11.697+0000 7fc610692700 1 --2- 172.21.15.98:0/1861825925 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fc6080ffcf0 0x7fc608100140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:30:11.702 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:11.697+0000 7fc610692700 1 -- 172.21.15.98:0/1861825925 >> 172.21.15.98:0/1861825925 conn(0x7fc6080fa100 msgr2=0x7fc6080fc550 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:30:11.702 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:11.697+0000 7fc610692700 1 -- 172.21.15.98:0/1861825925 shutdown_connections 2023-07-14T19:30:11.703 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:11.697+0000 7fc610692700 1 -- 172.21.15.98:0/1861825925 wait complete. 2023-07-14T19:30:11.703 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:11.697+0000 7fc610692700 1 Processor -- start 2023-07-14T19:30:11.703 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:11.697+0000 7fc610692700 1 -- start start 2023-07-14T19:30:11.703 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:11.697+0000 7fc610692700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fc6080feb00 0x7fc608194860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:30:11.704 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:11.697+0000 7fc610692700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc6080ffcf0 0x7fc608194da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:30:11.704 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:11.697+0000 7fc610692700 1 --2- >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7fc608100680 0x7fc60819be10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:30:11.704 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:11.697+0000 7fc610692700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc60819c3e0 con 0x7fc6080ffcf0 2023-07-14T19:30:11.705 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:11.697+0000 7fc610692700 1 -- --> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc60819c520 con 0x7fc608100680 2023-07-14T19:30:11.705 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:11.697+0000 7fc610692700 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc60819c6b0 con 0x7fc6080feb00 2023-07-14T19:30:11.705 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:11.697+0000 7fc60dc2d700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc6080ffcf0 0x7fc608194da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:30:11.705 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:11.698+0000 7fc60dc2d700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc6080ffcf0 0x7fc608194da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:46694/0 (socket says 172.21.15.98:46694) 2023-07-14T19:30:11.705 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:11.698+0000 7fc60dc2d700 1 -- 172.21.15.98:0/756378664 learned_addr learned my addr 172.21.15.98:0/756378664 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-07-14T19:30:11.705 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:11.698+0000 7fc60e42e700 1 --2- 172.21.15.98:0/756378664 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fc6080feb00 0x7fc608194860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:30:11.705 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:11.698+0000 7fc60ec2f700 1 --2- 172.21.15.98:0/756378664 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7fc608100680 0x7fc60819be10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:30:11.705 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:11.698+0000 7fc60dc2d700 1 -- 172.21.15.98:0/756378664 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fc6080feb00 msgr2=0x7fc608194860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:30:11.706 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:11.698+0000 7fc60dc2d700 1 --2- 172.21.15.98:0/756378664 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fc6080feb00 0x7fc608194860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:30:11.706 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:11.698+0000 7fc60dc2d700 1 -- 172.21.15.98:0/756378664 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7fc608100680 msgr2=0x7fc60819be10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:30:11.706 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:11.698+0000 7fc60dc2d700 1 --2- 172.21.15.98:0/756378664 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7fc608100680 0x7fc60819be10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:30:11.706 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:11.698+0000 7fc60dc2d700 1 -- 172.21.15.98:0/756378664 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc6040095f0 con 0x7fc6080ffcf0 2023-07-14T19:30:11.706 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:11.698+0000 7fc60ec2f700 1 --2- 172.21.15.98:0/756378664 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7fc608100680 0x7fc60819be10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-07-14T19:30:11.706 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:11.698+0000 7fc60dc2d700 1 --2- 172.21.15.98:0/756378664 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc6080ffcf0 0x7fc608194da0 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7fc5f800d980 tx=0x7fc5f800dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:30:11.706 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:11.698+0000 7fc5ff7fe700 1 -- 172.21.15.98:0/756378664 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc5f8004220 con 0x7fc6080ffcf0 2023-07-14T19:30:11.707 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:11.698+0000 7fc5ff7fe700 1 -- 172.21.15.98:0/756378664 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fc5f8007640 con 0x7fc6080ffcf0 2023-07-14T19:30:11.707 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:11.698+0000 7fc610692700 1 -- 172.21.15.98:0/756378664 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc60819c990 con 0x7fc6080ffcf0 2023-07-14T19:30:11.707 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:11.698+0000 7fc5ff7fe700 1 -- 172.21.15.98:0/756378664 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc5f8005610 con 0x7fc6080ffcf0 2023-07-14T19:30:11.707 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:11.698+0000 7fc610692700 1 -- 172.21.15.98:0/756378664 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc60819cea0 con 0x7fc6080ffcf0 2023-07-14T19:30:11.707 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:11.699+0000 7fc610692700 1 -- 172.21.15.98:0/756378664 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc6081060a0 con 0x7fc6080ffcf0 2023-07-14T19:30:11.708 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:11.701+0000 7fc5ff7fe700 1 -- 172.21.15.98:0/756378664 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 17) v1 ==== 135766+0+0 (secure 0 0 0) 0x7fc5f8004750 con 0x7fc6080ffcf0 2023-07-14T19:30:11.708 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:11.701+0000 7fc5ff7fe700 1 --2- 172.21.15.98:0/756378664 >> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] conn(0x7fc5f40a1220 0x7fc5f40a36d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:30:11.708 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:11.701+0000 7fc60e42e700 1 --2- 172.21.15.98:0/756378664 >> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] conn(0x7fc5f40a1220 0x7fc5f40a36d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:30:11.708 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:11.701+0000 7fc5ff7fe700 1 -- 172.21.15.98:0/756378664 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 8211+0+0 (secure 0 0 0) 0x7fc5f80cce20 con 0x7fc6080ffcf0 2023-07-14T19:30:11.709 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:11.702+0000 7fc60e42e700 1 --2- 172.21.15.98:0/756378664 >> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] conn(0x7fc5f40a1220 0x7fc5f40a36d0 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7fc60400b5c0 tx=0x7fc604005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:30:11.709 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:11.704+0000 7fc5ff7fe700 1 -- 172.21.15.98:0/756378664 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7fc5f8096420 con 0x7fc6080ffcf0 2023-07-14T19:30:11.781 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:30:11 smithi098 ceph-mon[127486]: from='client.? 172.21.15.98:0/2117490361' entity='client.admin' cmd=[{"prefix": "osd dump"}]: dispatch 2023-07-14T19:30:11.898 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:11.898+0000 7fc610692700 1 -- 172.21.15.98:0/756378664 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fc60802cb30 con 0x7fc6080ffcf0 2023-07-14T19:30:11.899 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:11.899+0000 7fc5ff7fe700 1 -- 172.21.15.98:0/756378664 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v85) v1 ==== 74+0+22604 (secure 0 0 0) 0x7fc5f8016090 con 0x7fc6080ffcf0 2023-07-14T19:30:11.900 INFO:teuthology.orchestra.run.smithi098.stdout: 2023-07-14T19:30:11.900 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":85,"fsid":"9adea648-227b-11ee-9b33-001a4aab830c","created":"2023-07-14T19:22:07.178854+0000","modified":"2023-07-14T19:30:03.314948+0000","last_up_change":"2023-07-14T19:28:19.646827+0000","last_in_change":"2023-07-14T19:28:02.018325+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-07-14T19:25:09.845827+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2023-07-14T19:28:58.952921+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"82","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2023-07-14T19:29:00.228272+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"83","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.690000057220459,"score_stable":1.690000057220459,"optimal_score":1,"raw_score_acting":1.690000057220459,"raw_score_stable":1.690000057220459,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"a3797e89-1754-41cf-9517-227e06ffdd72","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6802","nonce":120507228},{"type":"v1","addr":"172.21.15.98:6803","nonce":120507228}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6804","nonce":120507228},{"type":"v1","addr":"172.21.15.98:6805","nonce":120507228}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6808","nonce":120507228},{"type":"v1","addr":"172.21.15.98:6809","nonce":120507228}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6806","nonce":120507228},{"type":"v1","addr":"172.21.15.98:6807","nonce":120507228}]},"public_addr":"172.21.15.98:6803/120507228","cluster_addr":"172.21.15.98:6805/120507228","heartbeat_back_addr":"172.21.15.98:6809/120507228","heartbeat_front_addr":"172.21.15.98:6807/120507228","state":["exists","up"]},{"osd":1,"uuid":"2b001add-1aaa-4496-a403-7ef9725b80a0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.101:6800","nonce":2450584584},{"type":"v1","addr":"172.21.15.101:6801","nonce":2450584584}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.101:6802","nonce":2450584584},{"type":"v1","addr":"172.21.15.101:6803","nonce":2450584584}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.101:6806","nonce":2450584584},{"type":"v1","addr":"172.21.15.101:6807","nonce":2450584584}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.101:6804","nonce":2450584584},{"type":"v1","addr":"172.21.15.101:6805","nonce":2450584584}]},"public_addr":"172.21.15.101:6801/2450584584","cluster_addr":"172.21.15.101:6803/2450584584","heartbeat_back_addr":"172.21.15.101:6807/2450584584","heartbeat_front_addr":"172.21.15.101:6805/2450584584","state":["exists","up"]},{"osd":2,"uuid":"48e6d499-3ca9-424b-b0fb-3ba5f33b9fc9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":945038995},{"type":"v1","addr":"172.21.15.139:6801","nonce":945038995}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":945038995},{"type":"v1","addr":"172.21.15.139:6803","nonce":945038995}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":945038995},{"type":"v1","addr":"172.21.15.139:6807","nonce":945038995}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":945038995},{"type":"v1","addr":"172.21.15.139:6805","nonce":945038995}]},"public_addr":"172.21.15.139:6801/945038995","cluster_addr":"172.21.15.139:6803/945038995","heartbeat_back_addr":"172.21.15.139:6807/945038995","heartbeat_front_addr":"172.21.15.139:6805/945038995","state":["exists","up"]},{"osd":3,"uuid":"5530b265-31a5-447b-a1ea-17ebae9cfddf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6810","nonce":3095249665},{"type":"v1","addr":"172.21.15.98:6811","nonce":3095249665}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6812","nonce":3095249665},{"type":"v1","addr":"172.21.15.98:6813","nonce":3095249665}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6816","nonce":3095249665},{"type":"v1","addr":"172.21.15.98:6817","nonce":3095249665}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6814","nonce":3095249665},{"type":"v1","addr":"172.21.15.98:6815","nonce":3095249665}]},"public_addr":"172.21.15.98:6811/3095249665","cluster_addr":"172.21.15.98:6813/3095249665","heartbeat_back_addr":"172.21.15.98:6817/3095249665","heartbeat_front_addr":"172.21.15.98:6815/3095249665","state":["exists","up"]},{"osd":4,"uuid":"615d4642-a997-4f95-9e38-9fbdaa831400","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":32,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.101:6808","nonce":2844761406},{"type":"v1","addr":"172.21.15.101:6809","nonce":2844761406}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.101:6810","nonce":2844761406},{"type":"v1","addr":"172.21.15.101:6811","nonce":2844761406}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.101:6814","nonce":2844761406},{"type":"v1","addr":"172.21.15.101:6815","nonce":2844761406}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.101:6812","nonce":2844761406},{"type":"v1","addr":"172.21.15.101:6813","nonce":2844761406}]},"public_addr":"172.21.15.101:6809/2844761406","cluster_addr":"172.21.15.101:6811/2844761406","heartbeat_back_addr":"172.21.15.101:6815/2844761406","heartbeat_front_addr":"172.21.15.101:6813/2844761406","state":["exists","up"]},{"osd":5,"uuid":"415de5fa-607a-45a8-b732-28daf3be6dd3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":37,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":3532263042},{"type":"v1","addr":"172.21.15.139:6809","nonce":3532263042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6810","nonce":3532263042},{"type":"v1","addr":"172.21.15.139:6811","nonce":3532263042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6814","nonce":3532263042},{"type":"v1","addr":"172.21.15.139:6815","nonce":3532263042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":3532263042},{"type":"v1","addr":"172.21.15.139:6813","nonce":3532263042}]},"public_addr":"172.21.15.139:6809/3532263042","cluster_addr":"172.21.15.139:6811/3532263042","heartbeat_back_addr":"172.21.15.139:6815/3532263042","heartbeat_front_addr":"172.21.15.139:6813/3532263042","state":["exists","up"]},{"osd":6,"uuid":"d7f9d4ba-c309-4b75-a836-5a36384b528e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":45,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6818","nonce":658948743},{"type":"v1","addr":"172.21.15.98:6819","nonce":658948743}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6820","nonce":658948743},{"type":"v1","addr":"172.21.15.98:6821","nonce":658948743}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6824","nonce":658948743},{"type":"v1","addr":"172.21.15.98:6825","nonce":658948743}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6822","nonce":658948743},{"type":"v1","addr":"172.21.15.98:6823","nonce":658948743}]},"public_addr":"172.21.15.98:6819/658948743","cluster_addr":"172.21.15.98:6821/658948743","heartbeat_back_addr":"172.21.15.98:6825/658948743","heartbeat_front_addr":"172.21.15.98:6823/658948743","state":["exists","up"]},{"osd":7,"uuid":"eedb94d9-c5cd-4a93-8a8c-2648804b7195","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":50,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.101:6816","nonce":2881214066},{"type":"v1","addr":"172.21.15.101:6817","nonce":2881214066}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.101:6818","nonce":2881214066},{"type":"v1","addr":"172.21.15.101:6819","nonce":2881214066}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.101:6822","nonce":2881214066},{"type":"v1","addr":"172.21.15.101:6823","nonce":2881214066}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.101:6820","nonce":2881214066},{"type":"v1","addr":"172.21.15.101:6821","nonce":2881214066}]},"public_addr":"172.21.15.101:6817/2881214066","cluster_addr":"172.21.15.101:6819/2881214066","heartbeat_back_addr":"172.21.15.101:6823/2881214066","heartbeat_front_addr":"172.21.15.101:6821/2881214066","state":["exists","up"]},{"osd":8,"uuid":"6bfd77c2-5770-466e-aa6f-84b50c74ffb6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":57,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":3669514524},{"type":"v1","addr":"172.21.15.139:6817","nonce":3669514524}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":3669514524},{"type":"v1","addr":"172.21.15.139:6819","nonce":3669514524}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":3669514524},{"type":"v1","addr":"172.21.15.139:6823","nonce":3669514524}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6820","nonce":3669514524},{"type":"v1","addr":"172.21.15.139:6821","nonce":3669514524}]},"public_addr":"172.21.15.139:6817/3669514524","cluster_addr":"172.21.15.139:6819/3669514524","heartbeat_back_addr":"172.21.15.139:6823/3669514524","heartbeat_front_addr":"172.21.15.139:6821/3669514524","state":["exists","up"]},{"osd":9,"uuid":"19c5b54a-2242-4b23-b721-63be9559653d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":65,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6826","nonce":2893994080},{"type":"v1","addr":"172.21.15.98:6827","nonce":2893994080}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6828","nonce":2893994080},{"type":"v1","addr":"172.21.15.98:6829","nonce":2893994080}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6832","nonce":2893994080},{"type":"v1","addr":"172.21.15.98:6833","nonce":2893994080}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6830","nonce":2893994080},{"type":"v1","addr":"172.21.15.98:6831","nonce":2893994080}]},"public_addr":"172.21.15.98:6827/2893994080","cluster_addr":"172.21.15.98:6829/2893994080","heartbeat_back_addr":"172.21.15.98:6833/2893994080","heartbeat_front_addr":"172.21.15.98:6831/2893994080","state":["exists","up"]},{"osd":10,"uuid":"326f0209-ba9d-4b59-97cc-d61a758a8f29","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":70,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.101:6824","nonce":3545244225},{"type":"v1","addr":"172.21.15.101:6825","nonce":3545244225}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.101:6826","nonce":3545244225},{"type":"v1","addr":"172.21.15.101:6827","nonce":3545244225}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.101:6830","nonce":3545244225},{"type":"v1","addr":"172.21.15.101:6831","nonce":3545244225}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.101:6828","nonce":3545244225},{"type":"v1","addr":"172.21.15.101:6829","nonce":3545244225}]},"public_addr":"172.21.15.101:6825/3545244225","cluster_addr":"172.21.15.101:6827/3545244225","heartbeat_back_addr":"172.21.15.101:6831/3545244225","heartbeat_front_addr":"172.21.15.101:6829/3545244225","state":["exists","up"]},{"osd":11,"uuid":"c61c0320-1057-4f17-9bf1-7ad306ed7ed3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":78,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":2630023454},{"type":"v1","addr":"172.21.15.139:6825","nonce":2630023454}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6826","nonce":2630023454},{"type":"v1","addr":"172.21.15.139:6827","nonce":2630023454}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":2630023454},{"type":"v1","addr":"172.21.15.139:6831","nonce":2630023454}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":2630023454},{"type":"v1","addr":"172.21.15.139:6829","nonce":2630023454}]},"public_addr":"172.21.15.139:6825/2630023454","cluster_addr":"172.21.15.139:6827/2630023454","heartbeat_back_addr":"172.21.15.139:6831/2630023454","heartbeat_front_addr":"172.21.15.139:6829/2630023454","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T19:24:23.620028+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T19:24:45.779846+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T19:25:06.877949+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T19:25:27.994199+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T19:25:49.461139+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T19:26:10.409282+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T19:26:32.116390+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T19:26:53.536474+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T19:27:14.384839+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T19:27:35.635921+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T19:27:56.858744+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T19:28:18.342519+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.4","mappings":[{"from":2,"to":11}]},{"pgid":"2.f","mappings":[{"from":8,"to":11}]},{"pgid":"3.0","mappings":[{"from":4,"to":8}]},{"pgid":"3.1","mappings":[{"from":4,"to":9}]},{"pgid":"3.c","mappings":[{"from":3,"to":9}]},{"pgid":"3.17","mappings":[{"from":4,"to":6}]},{"pgid":"3.1e","mappings":[{"from":3,"to":9}]},{"pgid":"3.28","mappings":[{"from":5,"to":7}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.98:0/2164590253":"2023-07-15T19:23:01.520302+0000","172.21.15.98:0/2094734896":"2023-07-15T19:23:01.520302+0000","172.21.15.98:0/784745009":"2023-07-15T19:22:39.481502+0000","172.21.15.98:0/3657190818":"2023-07-15T19:22:39.481502+0000","172.21.15.98:6800/1732769087":"2023-07-15T19:23:01.520302+0000","172.21.15.98:0/3393542419":"2023-07-15T19:22:39.481502+0000","172.21.15.98:6800/749234665":"2023-07-15T19:22:39.481502+0000","172.21.15.98:6801/1732769087":"2023-07-15T19:23:01.520302+0000","172.21.15.98:6801/749234665":"2023-07-15T19:22:39.481502+0000","172.21.15.98:0/187822873":"2023-07-15T19:23:01.520302+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-07-14T19:30:11.902 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:11.901+0000 7fc610692700 1 -- 172.21.15.98:0/756378664 >> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] conn(0x7fc5f40a1220 msgr2=0x7fc5f40a36d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:30:11.902 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:11.901+0000 7fc610692700 1 --2- 172.21.15.98:0/756378664 >> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] conn(0x7fc5f40a1220 0x7fc5f40a36d0 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7fc60400b5c0 tx=0x7fc604005fd0 comp rx=0 tx=0).stop 2023-07-14T19:30:11.902 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:11.901+0000 7fc610692700 1 -- 172.21.15.98:0/756378664 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc6080ffcf0 msgr2=0x7fc608194da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:30:11.902 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:11.901+0000 7fc610692700 1 --2- 172.21.15.98:0/756378664 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc6080ffcf0 0x7fc608194da0 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7fc5f800d980 tx=0x7fc5f800dc90 comp rx=0 tx=0).stop 2023-07-14T19:30:11.903 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:11.901+0000 7fc610692700 1 -- 172.21.15.98:0/756378664 shutdown_connections 2023-07-14T19:30:11.903 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:11.901+0000 7fc610692700 1 --2- 172.21.15.98:0/756378664 >> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] conn(0x7fc5f40a1220 0x7fc5f40a36d0 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:30:11.903 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:11.901+0000 7fc610692700 1 --2- 172.21.15.98:0/756378664 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fc6080feb00 0x7fc608194860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:30:11.903 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:11.901+0000 7fc610692700 1 --2- 172.21.15.98:0/756378664 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7fc608100680 0x7fc60819be10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:30:11.903 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:11.901+0000 7fc610692700 1 --2- 172.21.15.98:0/756378664 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc6080ffcf0 0x7fc608194da0 unknown :-1 s=CLOSED pgs=351 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:30:11.903 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:11.901+0000 7fc610692700 1 -- 172.21.15.98:0/756378664 >> 172.21.15.98:0/756378664 conn(0x7fc6080fa100 msgr2=0x7fc6081031e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:30:11.903 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:11.901+0000 7fc610692700 1 -- 172.21.15.98:0/756378664 shutdown_connections 2023-07-14T19:30:11.904 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:11.901+0000 7fc610692700 1 -- 172.21.15.98:0/756378664 wait complete. 2023-07-14T19:30:11.915 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-07-14T19:25:09.845827+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2023-07-14T19:28:58.952921+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '82', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.8799999952316284, 'score_stable': 1.8799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2023-07-14T19:29:00.228272+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '83', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.690000057220459, 'score_stable': 1.690000057220459, 'optimal_score': 1, 'raw_score_acting': 1.690000057220459, 'raw_score_stable': 1.690000057220459, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-07-14T19:30:11.915 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-07-14T19:30:12.039 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:12.038+0000 7fcf6f978700 1 -- 172.21.15.98:0/962450306 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcf68100690 msgr2=0x7fcf68106f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:30:12.040 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:12.038+0000 7fcf6f978700 1 --2- 172.21.15.98:0/962450306 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcf68100690 0x7fcf68106f50 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7fcf6000b0a0 tx=0x7fcf6000b3b0 comp rx=0 tx=0).stop 2023-07-14T19:30:12.040 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:12.038+0000 7fcf6f978700 1 -- 172.21.15.98:0/962450306 shutdown_connections 2023-07-14T19:30:12.040 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:12.038+0000 7fcf6f978700 1 --2- 172.21.15.98:0/962450306 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fcf680feb10 0x7fcf680fef20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:30:12.040 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:12.038+0000 7fcf6f978700 1 --2- 172.21.15.98:0/962450306 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcf68100690 0x7fcf68106f50 unknown :-1 s=CLOSED pgs=352 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:30:12.040 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:12.038+0000 7fcf6f978700 1 --2- 172.21.15.98:0/962450306 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7fcf680ffd00 0x7fcf68100150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:30:12.040 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:12.038+0000 7fcf6f978700 1 -- 172.21.15.98:0/962450306 >> 172.21.15.98:0/962450306 conn(0x7fcf680fa130 msgr2=0x7fcf680fc560 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:30:12.040 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:12.039+0000 7fcf6f978700 1 -- 172.21.15.98:0/962450306 shutdown_connections 2023-07-14T19:30:12.041 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:12.039+0000 7fcf6f978700 1 -- 172.21.15.98:0/962450306 wait complete. 2023-07-14T19:30:12.041 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:12.039+0000 7fcf6f978700 1 Processor -- start 2023-07-14T19:30:12.041 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:12.039+0000 7fcf6f978700 1 -- start start 2023-07-14T19:30:12.043 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:12.039+0000 7fcf6f978700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fcf680feb10 0x7fcf68068310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:30:12.043 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:12.039+0000 7fcf6f978700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcf680ffd00 0x7fcf6819cb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:30:12.044 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:12.039+0000 7fcf6f978700 1 --2- >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7fcf68100690 0x7fcf6819f0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:30:12.044 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:12.039+0000 7fcf6f978700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcf6819f790 con 0x7fcf680ffd00 2023-07-14T19:30:12.044 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:12.039+0000 7fcf6f978700 1 -- --> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcf6819f900 con 0x7fcf68100690 2023-07-14T19:30:12.044 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:12.039+0000 7fcf6f978700 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcf6819fac0 con 0x7fcf680feb10 2023-07-14T19:30:12.044 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:12.039+0000 7fcf6cf13700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcf680ffd00 0x7fcf6819cb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:30:12.044 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:12.039+0000 7fcf6cf13700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcf680ffd00 0x7fcf6819cb90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:46708/0 (socket says 172.21.15.98:46708) 2023-07-14T19:30:12.044 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:12.039+0000 7fcf6cf13700 1 -- 172.21.15.98:0/3734407787 learned_addr learned my addr 172.21.15.98:0/3734407787 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-07-14T19:30:12.045 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:12.040+0000 7fcf6d714700 1 --2- 172.21.15.98:0/3734407787 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fcf680feb10 0x7fcf68068310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:30:12.045 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:12.040+0000 7fcf6df15700 1 --2- 172.21.15.98:0/3734407787 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7fcf68100690 0x7fcf6819f0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:30:12.045 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:12.040+0000 7fcf6cf13700 1 -- 172.21.15.98:0/3734407787 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fcf680feb10 msgr2=0x7fcf68068310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:30:12.045 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:12.040+0000 7fcf6cf13700 1 --2- 172.21.15.98:0/3734407787 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fcf680feb10 0x7fcf68068310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:30:12.045 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:12.040+0000 7fcf6cf13700 1 -- 172.21.15.98:0/3734407787 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7fcf68100690 msgr2=0x7fcf6819f0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:30:12.045 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:12.040+0000 7fcf6cf13700 1 --2- 172.21.15.98:0/3734407787 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7fcf68100690 0x7fcf6819f0e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:30:12.045 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:12.040+0000 7fcf6cf13700 1 -- 172.21.15.98:0/3734407787 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcf60009d00 con 0x7fcf680ffd00 2023-07-14T19:30:12.046 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:12.040+0000 7fcf6d714700 1 --2- 172.21.15.98:0/3734407787 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fcf680feb10 0x7fcf68068310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-07-14T19:30:12.046 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:12.040+0000 7fcf6df15700 1 --2- 172.21.15.98:0/3734407787 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7fcf68100690 0x7fcf6819f0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-07-14T19:30:12.046 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:12.040+0000 7fcf6cf13700 1 --2- 172.21.15.98:0/3734407787 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcf680ffd00 0x7fcf6819cb90 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7fcf5800c920 tx=0x7fcf5800cce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:30:12.046 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:12.040+0000 7fcf5e7fc700 1 -- 172.21.15.98:0/3734407787 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf58007b00 con 0x7fcf680ffd00 2023-07-14T19:30:12.046 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:12.040+0000 7fcf6f978700 1 -- 172.21.15.98:0/3734407787 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcf6819fda0 con 0x7fcf680ffd00 2023-07-14T19:30:12.046 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:12.040+0000 7fcf5e7fc700 1 -- 172.21.15.98:0/3734407787 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fcf58004cf0 con 0x7fcf680ffd00 2023-07-14T19:30:12.046 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:12.040+0000 7fcf5e7fc700 1 -- 172.21.15.98:0/3734407787 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf58005520 con 0x7fcf680ffd00 2023-07-14T19:30:12.046 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:12.040+0000 7fcf6f978700 1 -- 172.21.15.98:0/3734407787 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcf681a0250 con 0x7fcf680ffd00 2023-07-14T19:30:12.047 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:12.041+0000 7fcf6f978700 1 -- 172.21.15.98:0/3734407787 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcf681059a0 con 0x7fcf680ffd00 2023-07-14T19:30:12.047 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:12.042+0000 7fcf5e7fc700 1 -- 172.21.15.98:0/3734407787 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 17) v1 ==== 135766+0+0 (secure 0 0 0) 0x7fcf58004750 con 0x7fcf680ffd00 2023-07-14T19:30:12.047 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:12.042+0000 7fcf5e7fc700 1 --2- 172.21.15.98:0/3734407787 >> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] conn(0x7fcf540a11e0 0x7fcf540a3690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:30:12.047 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:12.043+0000 7fcf6d714700 1 --2- 172.21.15.98:0/3734407787 >> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] conn(0x7fcf540a11e0 0x7fcf540a3690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:30:12.047 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:12.043+0000 7fcf5e7fc700 1 -- 172.21.15.98:0/3734407787 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 8211+0+0 (secure 0 0 0) 0x7fcf580ccaa0 con 0x7fcf680ffd00 2023-07-14T19:30:12.048 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:12.043+0000 7fcf6d714700 1 --2- 172.21.15.98:0/3734407787 >> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] conn(0x7fcf540a11e0 0x7fcf540a3690 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7fcf680ffb60 tx=0x7fcf64005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:30:12.048 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:12.046+0000 7fcf5e7fc700 1 -- 172.21.15.98:0/3734407787 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7fcf580960a0 con 0x7fcf680ffd00 2023-07-14T19:30:12.191 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:30:11 smithi101 ceph-mon[128807]: from='client.? 172.21.15.98:0/2117490361' entity='client.admin' cmd=[{"prefix": "osd dump"}]: dispatch 2023-07-14T19:30:12.220 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:30:11 smithi139 ceph-mon[129293]: from='client.? 172.21.15.98:0/2117490361' entity='client.admin' cmd=[{"prefix": "osd dump"}]: dispatch 2023-07-14T19:30:12.239 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:12.238+0000 7fcf6f978700 1 -- 172.21.15.98:0/3734407787 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fcf6802ca90 con 0x7fcf680ffd00 2023-07-14T19:30:12.239 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:12.238+0000 7fcf5e7fc700 1 -- 172.21.15.98:0/3734407787 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v85) v1 ==== 93+0+10 (secure 0 0 0) 0x7fcf5801d020 con 0x7fcf680ffd00 2023-07-14T19:30:12.239 INFO:teuthology.orchestra.run.smithi098.stdout:pg_num: 1 2023-07-14T19:30:12.241 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:12.241+0000 7fcf6f978700 1 -- 172.21.15.98:0/3734407787 >> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] conn(0x7fcf540a11e0 msgr2=0x7fcf540a3690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:30:12.241 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:12.241+0000 7fcf6f978700 1 --2- 172.21.15.98:0/3734407787 >> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] conn(0x7fcf540a11e0 0x7fcf540a3690 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7fcf680ffb60 tx=0x7fcf64005f90 comp rx=0 tx=0).stop 2023-07-14T19:30:12.241 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:12.241+0000 7fcf6f978700 1 -- 172.21.15.98:0/3734407787 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcf680ffd00 msgr2=0x7fcf6819cb90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:30:12.241 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:12.241+0000 7fcf6f978700 1 --2- 172.21.15.98:0/3734407787 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcf680ffd00 0x7fcf6819cb90 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7fcf5800c920 tx=0x7fcf5800cce0 comp rx=0 tx=0).stop 2023-07-14T19:30:12.242 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:12.241+0000 7fcf6f978700 1 -- 172.21.15.98:0/3734407787 shutdown_connections 2023-07-14T19:30:12.242 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:12.241+0000 7fcf6f978700 1 --2- 172.21.15.98:0/3734407787 >> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] conn(0x7fcf540a11e0 0x7fcf540a3690 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:30:12.242 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:12.241+0000 7fcf6f978700 1 --2- 172.21.15.98:0/3734407787 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fcf680feb10 0x7fcf68068310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:30:12.242 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:12.241+0000 7fcf6f978700 1 --2- 172.21.15.98:0/3734407787 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7fcf68100690 0x7fcf6819f0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:30:12.242 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:12.241+0000 7fcf6f978700 1 --2- 172.21.15.98:0/3734407787 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcf680ffd00 0x7fcf6819cb90 unknown :-1 s=CLOSED pgs=353 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:30:12.242 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:12.241+0000 7fcf6f978700 1 -- 172.21.15.98:0/3734407787 >> 172.21.15.98:0/3734407787 conn(0x7fcf680fa130 msgr2=0x7fcf680fc2d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:30:12.242 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:12.241+0000 7fcf6f978700 1 -- 172.21.15.98:0/3734407787 shutdown_connections 2023-07-14T19:30:12.243 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:12.241+0000 7fcf6f978700 1 -- 172.21.15.98:0/3734407787 wait complete. 2023-07-14T19:30:12.253 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-07-14T19:30:12.381 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:12.377+0000 7f535cd17700 1 -- 172.21.15.98:0/1364663396 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f53580ffd20 msgr2=0x7f5358100170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:30:12.381 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:12.377+0000 7f535cd17700 1 --2- 172.21.15.98:0/1364663396 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f53580ffd20 0x7f5358100170 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7f5340009990 tx=0x7f5340009ca0 comp rx=0 tx=0).stop 2023-07-14T19:30:12.382 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:12.377+0000 7f535cd17700 1 -- 172.21.15.98:0/1364663396 shutdown_connections 2023-07-14T19:30:12.382 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:12.377+0000 7f535cd17700 1 --2- 172.21.15.98:0/1364663396 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f53581006b0 0x7f5358106f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:30:12.382 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:12.377+0000 7f535cd17700 1 --2- 172.21.15.98:0/1364663396 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f53580feb30 0x7f53580fef40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:30:12.383 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:12.377+0000 7f535cd17700 1 --2- 172.21.15.98:0/1364663396 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f53580ffd20 0x7f5358100170 unknown :-1 s=CLOSED pgs=354 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:30:12.383 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:12.377+0000 7f535cd17700 1 -- 172.21.15.98:0/1364663396 >> 172.21.15.98:0/1364663396 conn(0x7f53580fa190 msgr2=0x7f53580fc5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:30:12.383 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:12.377+0000 7f535cd17700 1 -- 172.21.15.98:0/1364663396 shutdown_connections 2023-07-14T19:30:12.383 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:12.377+0000 7f535cd17700 1 -- 172.21.15.98:0/1364663396 wait complete. 2023-07-14T19:30:12.383 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:12.378+0000 7f535cd17700 1 Processor -- start 2023-07-14T19:30:12.384 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:12.378+0000 7f535cd17700 1 -- start start 2023-07-14T19:30:12.384 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:12.378+0000 7f535cd17700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f53580feb30 0x7f5358195990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:30:12.384 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:12.378+0000 7f535cd17700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f53580ffd20 0x7f5358195ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:30:12.384 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:12.378+0000 7f535cd17700 1 --2- >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f53581006b0 0x7f535819cf40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:30:12.385 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:12.378+0000 7f535cd17700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f535819d510 con 0x7f53580ffd20 2023-07-14T19:30:12.385 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:12.378+0000 7f535cd17700 1 -- --> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f535819d650 con 0x7f53581006b0 2023-07-14T19:30:12.385 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:12.378+0000 7f535cd17700 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f535819d7e0 con 0x7f53580feb30 2023-07-14T19:30:12.385 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:12.378+0000 7f5355d9b700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f53580ffd20 0x7f5358195ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:30:12.386 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:12.378+0000 7f5355d9b700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f53580ffd20 0x7f5358195ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:46722/0 (socket says 172.21.15.98:46722) 2023-07-14T19:30:12.386 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:12.378+0000 7f5355d9b700 1 -- 172.21.15.98:0/1622493165 learned_addr learned my addr 172.21.15.98:0/1622493165 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-07-14T19:30:12.386 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:12.378+0000 7f535659c700 1 --2- 172.21.15.98:0/1622493165 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f53580feb30 0x7f5358195990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:30:12.386 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:12.379+0000 7f5356d9d700 1 --2- 172.21.15.98:0/1622493165 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f53581006b0 0x7f535819cf40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:30:12.387 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:12.379+0000 7f5355d9b700 1 -- 172.21.15.98:0/1622493165 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f53580feb30 msgr2=0x7f5358195990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:30:12.387 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:12.379+0000 7f5355d9b700 1 --2- 172.21.15.98:0/1622493165 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f53580feb30 0x7f5358195990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:30:12.387 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:12.379+0000 7f5355d9b700 1 -- 172.21.15.98:0/1622493165 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f53581006b0 msgr2=0x7f535819cf40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:30:12.387 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:12.379+0000 7f5355d9b700 1 --2- 172.21.15.98:0/1622493165 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f53581006b0 0x7f535819cf40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:30:12.388 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:12.379+0000 7f5355d9b700 1 -- 172.21.15.98:0/1622493165 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f53400095f0 con 0x7f53580ffd20 2023-07-14T19:30:12.388 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:12.379+0000 7f535659c700 1 --2- 172.21.15.98:0/1622493165 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f53580feb30 0x7f5358195990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-07-14T19:30:12.388 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:12.379+0000 7f5356d9d700 1 --2- 172.21.15.98:0/1622493165 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f53581006b0 0x7f535819cf40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-07-14T19:30:12.388 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:12.379+0000 7f5355d9b700 1 --2- 172.21.15.98:0/1622493165 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f53580ffd20 0x7f5358195ed0 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7f5340005770 tx=0x7f5340009ee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:30:12.389 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:12.379+0000 7f534f7fe700 1 -- 172.21.15.98:0/1622493165 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f534001d070 con 0x7f53580ffd20 2023-07-14T19:30:12.389 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:12.379+0000 7f534f7fe700 1 -- 172.21.15.98:0/1622493165 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f53400041a0 con 0x7f53580ffd20 2023-07-14T19:30:12.389 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:12.379+0000 7f535cd17700 1 -- 172.21.15.98:0/1622493165 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f535819da60 con 0x7f53580ffd20 2023-07-14T19:30:12.390 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:12.379+0000 7f534f7fe700 1 -- 172.21.15.98:0/1622493165 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f534000f460 con 0x7f53580ffd20 2023-07-14T19:30:12.390 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:12.379+0000 7f535cd17700 1 -- 172.21.15.98:0/1622493165 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f535819de80 con 0x7f53580ffd20 2023-07-14T19:30:12.390 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:12.380+0000 7f535cd17700 1 -- 172.21.15.98:0/1622493165 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f535810a300 con 0x7f53580ffd20 2023-07-14T19:30:12.390 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:12.381+0000 7f534f7fe700 1 -- 172.21.15.98:0/1622493165 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 17) v1 ==== 135766+0+0 (secure 0 0 0) 0x7f5340003ce0 con 0x7f53580ffd20 2023-07-14T19:30:12.392 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:12.381+0000 7f534f7fe700 1 --2- 172.21.15.98:0/1622493165 >> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] conn(0x7f53440a11e0 0x7f53440a3690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:30:12.392 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:12.381+0000 7f535659c700 1 --2- 172.21.15.98:0/1622493165 >> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] conn(0x7f53440a11e0 0x7f53440a3690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:30:12.392 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:12.382+0000 7f534f7fe700 1 -- 172.21.15.98:0/1622493165 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 8211+0+0 (secure 0 0 0) 0x7f53400ce0a0 con 0x7f53580ffd20 2023-07-14T19:30:12.393 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:12.382+0000 7f535659c700 1 --2- 172.21.15.98:0/1622493165 >> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] conn(0x7f53440a11e0 0x7f53440a3690 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f53580ffb80 tx=0x7f5348005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:30:12.393 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:12.385+0000 7f534f7fe700 1 -- 172.21.15.98:0/1622493165 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f53400977c0 con 0x7f53580ffd20 2023-07-14T19:30:12.575 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:12.574+0000 7f535cd17700 1 -- 172.21.15.98:0/1622493165 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f535819e430 con 0x7f53580ffd20 2023-07-14T19:30:12.575 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:12.575+0000 7f534f7fe700 1 -- 172.21.15.98:0/1622493165 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v85) v1 ==== 104+0+11 (secure 0 0 0) 0x7f534009ade0 con 0x7f53580ffd20 2023-07-14T19:30:12.575 INFO:teuthology.orchestra.run.smithi098.stdout:pg_num: 32 2023-07-14T19:30:12.577 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:12.577+0000 7f535cd17700 1 -- 172.21.15.98:0/1622493165 >> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] conn(0x7f53440a11e0 msgr2=0x7f53440a3690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:30:12.578 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:12.577+0000 7f535cd17700 1 --2- 172.21.15.98:0/1622493165 >> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] conn(0x7f53440a11e0 0x7f53440a3690 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f53580ffb80 tx=0x7f5348005f90 comp rx=0 tx=0).stop 2023-07-14T19:30:12.578 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:12.577+0000 7f535cd17700 1 -- 172.21.15.98:0/1622493165 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f53580ffd20 msgr2=0x7f5358195ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:30:12.578 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:12.577+0000 7f535cd17700 1 --2- 172.21.15.98:0/1622493165 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f53580ffd20 0x7f5358195ed0 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7f5340005770 tx=0x7f5340009ee0 comp rx=0 tx=0).stop 2023-07-14T19:30:12.578 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:12.577+0000 7f535cd17700 1 -- 172.21.15.98:0/1622493165 shutdown_connections 2023-07-14T19:30:12.578 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:12.577+0000 7f535cd17700 1 --2- 172.21.15.98:0/1622493165 >> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] conn(0x7f53440a11e0 0x7f53440a3690 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:30:12.578 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:12.577+0000 7f535cd17700 1 --2- 172.21.15.98:0/1622493165 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f53580feb30 0x7f5358195990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:30:12.578 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:12.577+0000 7f535cd17700 1 --2- 172.21.15.98:0/1622493165 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f53581006b0 0x7f535819cf40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:30:12.579 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:12.577+0000 7f535cd17700 1 --2- 172.21.15.98:0/1622493165 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f53580ffd20 0x7f5358195ed0 unknown :-1 s=CLOSED pgs=355 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:30:12.579 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:12.577+0000 7f535cd17700 1 -- 172.21.15.98:0/1622493165 >> 172.21.15.98:0/1622493165 conn(0x7f53580fa190 msgr2=0x7f5358068a10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:30:12.579 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:12.577+0000 7f535cd17700 1 -- 172.21.15.98:0/1622493165 shutdown_connections 2023-07-14T19:30:12.579 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:12.577+0000 7f535cd17700 1 -- 172.21.15.98:0/1622493165 wait complete. 2023-07-14T19:30:12.589 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-07-14T19:30:12.714 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:12.714+0000 7ff1a4dc7700 1 -- 172.21.15.98:0/1373502038 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff1a00feb10 msgr2=0x7ff1a00fef20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:30:12.715 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:12.714+0000 7ff1a4dc7700 1 --2- 172.21.15.98:0/1373502038 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff1a00feb10 0x7ff1a00fef20 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7ff19800b0a0 tx=0x7ff19800b3b0 comp rx=0 tx=0).stop 2023-07-14T19:30:12.715 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:12.714+0000 7ff1a4dc7700 1 -- 172.21.15.98:0/1373502038 shutdown_connections 2023-07-14T19:30:12.715 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:12.714+0000 7ff1a4dc7700 1 --2- 172.21.15.98:0/1373502038 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7ff1a0100690 0x7ff1a0106f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:30:12.715 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:12.714+0000 7ff1a4dc7700 1 --2- 172.21.15.98:0/1373502038 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff1a00feb10 0x7ff1a00fef20 unknown :-1 s=CLOSED pgs=356 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:30:12.715 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:12.714+0000 7ff1a4dc7700 1 --2- 172.21.15.98:0/1373502038 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7ff1a00ffd00 0x7ff1a0100150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:30:12.715 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:12.714+0000 7ff1a4dc7700 1 -- 172.21.15.98:0/1373502038 >> 172.21.15.98:0/1373502038 conn(0x7ff1a00fa130 msgr2=0x7ff1a00fc560 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:30:12.716 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:12.714+0000 7ff1a4dc7700 1 -- 172.21.15.98:0/1373502038 shutdown_connections 2023-07-14T19:30:12.716 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:12.714+0000 7ff1a4dc7700 1 -- 172.21.15.98:0/1373502038 wait complete. 2023-07-14T19:30:12.717 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:12.714+0000 7ff1a4dc7700 1 Processor -- start 2023-07-14T19:30:12.718 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:12.714+0000 7ff1a4dc7700 1 -- start start 2023-07-14T19:30:12.718 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:12.715+0000 7ff1a4dc7700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff1a00feb10 0x7ff1a0068310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:30:12.718 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:12.715+0000 7ff1a4dc7700 1 --2- >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7ff1a00ffd00 0x7ff1a019cc00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:30:12.718 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:12.715+0000 7ff1a4dc7700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7ff1a0100690 0x7ff1a019f150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:30:12.718 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:12.715+0000 7ff1a4dc7700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff1a019f800 con 0x7ff1a00feb10 2023-07-14T19:30:12.718 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:12.715+0000 7ff1a4dc7700 1 -- --> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff1a019f970 con 0x7ff1a00ffd00 2023-07-14T19:30:12.719 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:12.715+0000 7ff1a4dc7700 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff1a019fb30 con 0x7ff1a0100690 2023-07-14T19:30:12.719 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:12.715+0000 7ff19e59c700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff1a00feb10 0x7ff1a0068310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:30:12.719 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:12.715+0000 7ff19e59c700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff1a00feb10 0x7ff1a0068310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:46734/0 (socket says 172.21.15.98:46734) 2023-07-14T19:30:12.719 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:12.715+0000 7ff19e59c700 1 -- 172.21.15.98:0/3584767653 learned_addr learned my addr 172.21.15.98:0/3584767653 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-07-14T19:30:12.719 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:12.715+0000 7ff19dd9b700 1 --2- 172.21.15.98:0/3584767653 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7ff1a00ffd00 0x7ff1a019cc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:30:12.719 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:12.715+0000 7ff19ed9d700 1 --2- 172.21.15.98:0/3584767653 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7ff1a0100690 0x7ff1a019f150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:30:12.719 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:12.715+0000 7ff19e59c700 1 -- 172.21.15.98:0/3584767653 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7ff1a0100690 msgr2=0x7ff1a019f150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:30:12.720 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:12.715+0000 7ff19e59c700 1 --2- 172.21.15.98:0/3584767653 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7ff1a0100690 0x7ff1a019f150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:30:12.720 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:12.715+0000 7ff19e59c700 1 -- 172.21.15.98:0/3584767653 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7ff1a00ffd00 msgr2=0x7ff1a019cc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:30:12.720 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:12.715+0000 7ff19e59c700 1 --2- 172.21.15.98:0/3584767653 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7ff1a00ffd00 0x7ff1a019cc00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:30:12.720 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:12.715+0000 7ff19e59c700 1 -- 172.21.15.98:0/3584767653 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff198009d00 con 0x7ff1a00feb10 2023-07-14T19:30:12.720 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:12.715+0000 7ff19e59c700 1 --2- 172.21.15.98:0/3584767653 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff1a00feb10 0x7ff1a0068310 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7ff19800b070 tx=0x7ff198004d00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:30:12.720 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:12.716+0000 7ff1977fe700 1 -- 172.21.15.98:0/3584767653 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff19800e040 con 0x7ff1a00feb10 2023-07-14T19:30:12.720 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:12.716+0000 7ff1977fe700 1 -- 172.21.15.98:0/3584767653 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7ff198009440 con 0x7ff1a00feb10 2023-07-14T19:30:12.721 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:12.716+0000 7ff1a4dc7700 1 -- 172.21.15.98:0/3584767653 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff1a019fdb0 con 0x7ff1a00feb10 2023-07-14T19:30:12.721 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:12.716+0000 7ff1977fe700 1 -- 172.21.15.98:0/3584767653 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff198004020 con 0x7ff1a00feb10 2023-07-14T19:30:12.721 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:12.716+0000 7ff1a4dc7700 1 -- 172.21.15.98:0/3584767653 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff1a01a01d0 con 0x7ff1a00feb10 2023-07-14T19:30:12.721 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:12.717+0000 7ff1a4dc7700 1 -- 172.21.15.98:0/3584767653 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff1a01060d0 con 0x7ff1a00feb10 2023-07-14T19:30:12.722 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:12.718+0000 7ff1977fe700 1 -- 172.21.15.98:0/3584767653 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 17) v1 ==== 135766+0+0 (secure 0 0 0) 0x7ff19801d690 con 0x7ff1a00feb10 2023-07-14T19:30:12.722 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:12.718+0000 7ff1977fe700 1 --2- 172.21.15.98:0/3584767653 >> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] conn(0x7ff18c0a1260 0x7ff18c0a3710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:30:12.722 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:12.718+0000 7ff19dd9b700 1 --2- 172.21.15.98:0/3584767653 >> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] conn(0x7ff18c0a1260 0x7ff18c0a3710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:30:12.722 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:12.718+0000 7ff1977fe700 1 -- 172.21.15.98:0/3584767653 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 8211+0+0 (secure 0 0 0) 0x7ff1980ce160 con 0x7ff1a00feb10 2023-07-14T19:30:12.722 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:12.719+0000 7ff19dd9b700 1 --2- 172.21.15.98:0/3584767653 >> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] conn(0x7ff18c0a1260 0x7ff18c0a3710 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7ff188005950 tx=0x7ff18800b410 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:30:12.723 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:12.723+0000 7ff1977fe700 1 -- 172.21.15.98:0/3584767653 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7ff1980977f0 con 0x7ff1a00feb10 2023-07-14T19:30:12.913 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:12.913+0000 7ff1a4dc7700 1 -- 172.21.15.98:0/3584767653 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7ff1a010a3a0 con 0x7ff1a00feb10 2023-07-14T19:30:12.914 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:12.913+0000 7ff1977fe700 1 -- 172.21.15.98:0/3584767653 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v85) v1 ==== 100+0+11 (secure 0 0 0) 0x7ff19809ae10 con 0x7ff1a00feb10 2023-07-14T19:30:12.915 INFO:teuthology.orchestra.run.smithi098.stdout:pg_num: 64 2023-07-14T19:30:12.917 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:12.915+0000 7ff1a4dc7700 1 -- 172.21.15.98:0/3584767653 >> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] conn(0x7ff18c0a1260 msgr2=0x7ff18c0a3710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:30:12.917 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:12.915+0000 7ff1a4dc7700 1 --2- 172.21.15.98:0/3584767653 >> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] conn(0x7ff18c0a1260 0x7ff18c0a3710 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7ff188005950 tx=0x7ff18800b410 comp rx=0 tx=0).stop 2023-07-14T19:30:12.917 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:12.915+0000 7ff1a4dc7700 1 -- 172.21.15.98:0/3584767653 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff1a00feb10 msgr2=0x7ff1a0068310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:30:12.918 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:12.915+0000 7ff1a4dc7700 1 --2- 172.21.15.98:0/3584767653 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff1a00feb10 0x7ff1a0068310 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7ff19800b070 tx=0x7ff198004d00 comp rx=0 tx=0).stop 2023-07-14T19:30:12.918 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:12.915+0000 7ff1a4dc7700 1 -- 172.21.15.98:0/3584767653 shutdown_connections 2023-07-14T19:30:12.918 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:12.915+0000 7ff1a4dc7700 1 --2- 172.21.15.98:0/3584767653 >> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] conn(0x7ff18c0a1260 0x7ff18c0a3710 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:30:12.918 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:12.915+0000 7ff1a4dc7700 1 --2- 172.21.15.98:0/3584767653 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7ff1a0100690 0x7ff1a019f150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:30:12.919 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:12.915+0000 7ff1a4dc7700 1 --2- 172.21.15.98:0/3584767653 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff1a00feb10 0x7ff1a0068310 unknown :-1 s=CLOSED pgs=357 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:30:12.919 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:12.915+0000 7ff1a4dc7700 1 --2- 172.21.15.98:0/3584767653 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7ff1a00ffd00 0x7ff1a019cc00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:30:12.919 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:12.915+0000 7ff1a4dc7700 1 -- 172.21.15.98:0/3584767653 >> 172.21.15.98:0/3584767653 conn(0x7ff1a00fa130 msgr2=0x7ff1a0103210 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:30:12.919 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:12.916+0000 7ff1a4dc7700 1 -- 172.21.15.98:0/3584767653 shutdown_connections 2023-07-14T19:30:12.919 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:12.916+0000 7ff1a4dc7700 1 -- 172.21.15.98:0/3584767653 wait complete. 2023-07-14T19:30:12.927 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:30:12 smithi098 ceph-mon[127486]: pgmap v290: 97 pgs: 97 active+clean; 634 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 5 op/s; 15 B/s, 0 objects/s recovering 2023-07-14T19:30:12.927 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:30:12 smithi098 ceph-mon[127486]: from='client.? 172.21.15.98:0/756378664' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-07-14T19:30:12.927 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:30:12 smithi098 ceph-mon[127486]: from='client.? 172.21.15.98:0/3734407787' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2023-07-14T19:30:12.927 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:30:12 smithi098 ceph-mon[127486]: from='client.? 172.21.15.98:0/1622493165' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2023-07-14T19:30:12.928 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-07-14T19:30:13.051 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:13.050+0000 7f577b2b8700 1 -- 172.21.15.98:0/2970117015 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f57740ffd10 msgr2=0x7f5774100160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:30:13.052 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:13.050+0000 7f577b2b8700 1 --2- 172.21.15.98:0/2970117015 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f57740ffd10 0x7f5774100160 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7f5764009990 tx=0x7f5764009ca0 comp rx=0 tx=0).stop 2023-07-14T19:30:13.052 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:13.050+0000 7f577b2b8700 1 -- 172.21.15.98:0/2970117015 shutdown_connections 2023-07-14T19:30:13.052 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:13.050+0000 7f577b2b8700 1 --2- 172.21.15.98:0/2970117015 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f57741006a0 0x7f5774106f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:30:13.052 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:13.050+0000 7f577b2b8700 1 --2- 172.21.15.98:0/2970117015 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f57740feb20 0x7f57740fef30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:30:13.053 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:13.050+0000 7f577b2b8700 1 --2- 172.21.15.98:0/2970117015 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f57740ffd10 0x7f5774100160 unknown :-1 s=CLOSED pgs=358 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:30:13.053 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:13.050+0000 7f577b2b8700 1 -- 172.21.15.98:0/2970117015 >> 172.21.15.98:0/2970117015 conn(0x7f57740fa100 msgr2=0x7f57740fc570 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:30:13.053 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:13.050+0000 7f577b2b8700 1 -- 172.21.15.98:0/2970117015 shutdown_connections 2023-07-14T19:30:13.053 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:13.050+0000 7f577b2b8700 1 -- 172.21.15.98:0/2970117015 wait complete. 2023-07-14T19:30:13.053 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:13.050+0000 7f577b2b8700 1 Processor -- start 2023-07-14T19:30:13.053 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:13.050+0000 7f577b2b8700 1 -- start start 2023-07-14T19:30:13.053 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:13.050+0000 7f577b2b8700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f57740feb20 0x7f5774069360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:30:13.054 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:13.050+0000 7f577b2b8700 1 --2- >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f57740ffd10 0x7f577419a620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:30:13.054 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:13.050+0000 7f577b2b8700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f57741006a0 0x7f577419cc70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:30:13.054 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:13.050+0000 7f577b2b8700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f577419d440 con 0x7f57740feb20 2023-07-14T19:30:13.054 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:13.050+0000 7f577b2b8700 1 -- --> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f577419d5b0 con 0x7f57740ffd10 2023-07-14T19:30:13.054 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:13.050+0000 7f577b2b8700 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f577419d770 con 0x7f57741006a0 2023-07-14T19:30:13.054 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:13.051+0000 7f5779054700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f57740feb20 0x7f5774069360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:30:13.054 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:13.051+0000 7f5779054700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f57740feb20 0x7f5774069360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:46748/0 (socket says 172.21.15.98:46748) 2023-07-14T19:30:13.054 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:13.051+0000 7f5779054700 1 -- 172.21.15.98:0/2454697739 learned_addr learned my addr 172.21.15.98:0/2454697739 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-07-14T19:30:13.055 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:13.051+0000 7f5779855700 1 --2- 172.21.15.98:0/2454697739 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f57741006a0 0x7f577419cc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:30:13.055 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:13.051+0000 7f5779054700 1 -- 172.21.15.98:0/2454697739 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f57741006a0 msgr2=0x7f577419cc70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:30:13.055 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:13.051+0000 7f5779054700 1 --2- 172.21.15.98:0/2454697739 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f57741006a0 0x7f577419cc70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:30:13.055 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:13.051+0000 7f5778853700 1 --2- 172.21.15.98:0/2454697739 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f57740ffd10 0x7f577419a620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:30:13.055 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:13.051+0000 7f5779054700 1 -- 172.21.15.98:0/2454697739 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f57740ffd10 msgr2=0x7f577419a620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:30:13.056 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:13.051+0000 7f5779054700 1 --2- 172.21.15.98:0/2454697739 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f57740ffd10 0x7f577419a620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:30:13.056 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:13.051+0000 7f5779054700 1 -- 172.21.15.98:0/2454697739 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f57640095f0 con 0x7f57740feb20 2023-07-14T19:30:13.056 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:13.051+0000 7f5779054700 1 --2- 172.21.15.98:0/2454697739 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f57740feb20 0x7f5774069360 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7f577000d980 tx=0x7f577000dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:30:13.056 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:13.051+0000 7f576a7fc700 1 -- 172.21.15.98:0/2454697739 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f577000b950 con 0x7f57740feb20 2023-07-14T19:30:13.056 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:13.051+0000 7f576a7fc700 1 -- 172.21.15.98:0/2454697739 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f5770007620 con 0x7f57740feb20 2023-07-14T19:30:13.056 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:13.051+0000 7f576a7fc700 1 -- 172.21.15.98:0/2454697739 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f577000f3c0 con 0x7f57740feb20 2023-07-14T19:30:13.056 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:13.051+0000 7f577b2b8700 1 -- 172.21.15.98:0/2454697739 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f577419da50 con 0x7f57740feb20 2023-07-14T19:30:13.057 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:13.051+0000 7f577b2b8700 1 -- 172.21.15.98:0/2454697739 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f577419df10 con 0x7f57740feb20 2023-07-14T19:30:13.058 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:13.052+0000 7f577b2b8700 1 -- 172.21.15.98:0/2454697739 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5774106070 con 0x7f57740feb20 2023-07-14T19:30:13.058 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:13.053+0000 7f576a7fc700 1 -- 172.21.15.98:0/2454697739 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 17) v1 ==== 135766+0+0 (secure 0 0 0) 0x7f5770010640 con 0x7f57740feb20 2023-07-14T19:30:13.058 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:13.054+0000 7f576a7fc700 1 --2- 172.21.15.98:0/2454697739 >> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] conn(0x7f57600a12a0 0x7f57600a3750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:30:13.058 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:13.054+0000 7f5778853700 1 --2- 172.21.15.98:0/2454697739 >> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] conn(0x7f57600a12a0 0x7f57600a3750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:30:13.058 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:13.054+0000 7f576a7fc700 1 -- 172.21.15.98:0/2454697739 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 8211+0+0 (secure 0 0 0) 0x7f57700cd1c0 con 0x7f57740feb20 2023-07-14T19:30:13.058 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:13.054+0000 7f5778853700 1 --2- 172.21.15.98:0/2454697739 >> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] conn(0x7f57600a12a0 0x7f57600a3750 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f576400b5c0 tx=0x7f5764005fb0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:30:13.058 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:13.057+0000 7f576a7fc700 1 -- 172.21.15.98:0/2454697739 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f5770096760 con 0x7f57740feb20 2023-07-14T19:30:13.191 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:30:12 smithi101 ceph-mon[128807]: pgmap v290: 97 pgs: 97 active+clean; 634 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 5 op/s; 15 B/s, 0 objects/s recovering 2023-07-14T19:30:13.191 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:30:12 smithi101 ceph-mon[128807]: from='client.? 172.21.15.98:0/756378664' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-07-14T19:30:13.192 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:30:12 smithi101 ceph-mon[128807]: from='client.? 172.21.15.98:0/3734407787' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2023-07-14T19:30:13.192 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:30:12 smithi101 ceph-mon[128807]: from='client.? 172.21.15.98:0/1622493165' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2023-07-14T19:30:13.220 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:30:12 smithi139 ceph-mon[129293]: pgmap v290: 97 pgs: 97 active+clean; 634 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 5 op/s; 15 B/s, 0 objects/s recovering 2023-07-14T19:30:13.220 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:30:12 smithi139 ceph-mon[129293]: from='client.? 172.21.15.98:0/756378664' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-07-14T19:30:13.221 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:30:12 smithi139 ceph-mon[129293]: from='client.? 172.21.15.98:0/3734407787' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2023-07-14T19:30:13.221 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:30:12 smithi139 ceph-mon[129293]: from='client.? 172.21.15.98:0/1622493165' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2023-07-14T19:30:13.289 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:13.288+0000 7f577b2b8700 1 -- 172.21.15.98:0/2454697739 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f577402c8e0 con 0x7f57740feb20 2023-07-14T19:30:13.290 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:13.289+0000 7f576a7fc700 1 -- 172.21.15.98:0/2454697739 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 22 v22) v1 ==== 94+0+10826 (secure 0 0 0) 0x7f5770016020 con 0x7f57740feb20 2023-07-14T19:30:13.290 INFO:teuthology.orchestra.run.smithi098.stdout: 2023-07-14T19:30:13.290 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":22,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14565,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6835/586280675","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6834","nonce":586280675},{"type":"v1","addr":"172.21.15.98:6835","nonce":586280675}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":4},{"gid":14577,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6837/487577257","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6836","nonce":487577257},{"type":"v1","addr":"172.21.15.98:6837","nonce":487577257}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":5},{"gid":14634,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6839/4085731209","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6838","nonce":4085731209},{"type":"v1","addr":"172.21.15.98:6839","nonce":4085731209}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":11},{"gid":14646,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6841/1679318259","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6840","nonce":1679318259},{"type":"v1","addr":"172.21.15.98:6841","nonce":1679318259}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":12},{"gid":24421,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.139:6833/3505477820","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6832","nonce":3505477820},{"type":"v1","addr":"172.21.15.139:6833","nonce":3505477820}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":24437,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.101:6833/3474460171","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.101:6832","nonce":3474460171},{"type":"v1","addr":"172.21.15.101:6833","nonce":3474460171}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":22,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2023-07-14T19:29:01.033156+0000","modified":"2023-07-14T19:29:06.291916+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24521,"mds_1":24493,"mds_2":24479},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24455":{"gid":24455,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.101:6835/1583264648","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.101:6834","nonce":1583264648},{"type":"v1","addr":"172.21.15.101:6835","nonce":1583264648}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24460":{"gid":24460,"name":"f","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.139:6835/2315281724","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6834","nonce":2315281724},{"type":"v1","addr":"172.21.15.139:6835","nonce":2315281724}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24469":{"gid":24469,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.139:6837/3597660552","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6836","nonce":3597660552},{"type":"v1","addr":"172.21.15.139:6837","nonce":3597660552}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24479":{"gid":24479,"name":"h","rank":2,"incarnation":21,"state":"up:active","state_seq":6,"addr":"172.21.15.101:6837/485741528","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.101:6836","nonce":485741528},{"type":"v1","addr":"172.21.15.101:6837","nonce":485741528}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24493":{"gid":24493,"name":"i","rank":1,"incarnation":19,"state":"up:active","state_seq":5,"addr":"172.21.15.139:6839/1258854143","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6838","nonce":1258854143},{"type":"v1","addr":"172.21.15.139:6839","nonce":1258854143}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24521":{"gid":24521,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.101:6839/48393381","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.101:6838","nonce":48393381},{"type":"v1","addr":"172.21.15.101:6839","nonce":48393381}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-07-14T19:30:13.292 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:13.291+0000 7f577b2b8700 1 -- 172.21.15.98:0/2454697739 >> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] conn(0x7f57600a12a0 msgr2=0x7f57600a3750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:30:13.292 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:13.291+0000 7f577b2b8700 1 --2- 172.21.15.98:0/2454697739 >> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] conn(0x7f57600a12a0 0x7f57600a3750 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f576400b5c0 tx=0x7f5764005fb0 comp rx=0 tx=0).stop 2023-07-14T19:30:13.292 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:13.291+0000 7f577b2b8700 1 -- 172.21.15.98:0/2454697739 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f57740feb20 msgr2=0x7f5774069360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:30:13.293 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:13.291+0000 7f577b2b8700 1 --2- 172.21.15.98:0/2454697739 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f57740feb20 0x7f5774069360 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7f577000d980 tx=0x7f577000dc90 comp rx=0 tx=0).stop 2023-07-14T19:30:13.293 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:13.291+0000 7f577b2b8700 1 -- 172.21.15.98:0/2454697739 shutdown_connections 2023-07-14T19:30:13.293 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:13.291+0000 7f577b2b8700 1 --2- 172.21.15.98:0/2454697739 >> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] conn(0x7f57600a12a0 0x7f57600a3750 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:30:13.293 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:13.291+0000 7f577b2b8700 1 --2- 172.21.15.98:0/2454697739 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f57741006a0 0x7f577419cc70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:30:13.293 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:13.291+0000 7f577b2b8700 1 --2- 172.21.15.98:0/2454697739 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f57740feb20 0x7f5774069360 unknown :-1 s=CLOSED pgs=359 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:30:13.293 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:13.291+0000 7f577b2b8700 1 --2- 172.21.15.98:0/2454697739 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f57740ffd10 0x7f577419a620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:30:13.293 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:13.291+0000 7f577b2b8700 1 -- 172.21.15.98:0/2454697739 >> 172.21.15.98:0/2454697739 conn(0x7f57740fa100 msgr2=0x7f57741031b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:30:13.294 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:13.291+0000 7f577b2b8700 1 -- 172.21.15.98:0/2454697739 shutdown_connections 2023-07-14T19:30:13.294 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:13.291+0000 7f577b2b8700 1 -- 172.21.15.98:0/2454697739 wait complete. 2023-07-14T19:30:13.294 INFO:teuthology.orchestra.run.smithi098.stderr:dumped fsmap epoch 22 2023-07-14T19:30:13.304 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-07-14T19:30:13.428 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:13.426+0000 7f8d82c30700 1 -- 172.21.15.98:0/2940608176 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8d7c0fed80 msgr2=0x7f8d7c0ff1d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:30:13.429 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:13.426+0000 7f8d82c30700 1 --2- 172.21.15.98:0/2940608176 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8d7c0fed80 0x7f8d7c0ff1d0 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7f8d68009990 tx=0x7f8d68009ca0 comp rx=0 tx=0).stop 2023-07-14T19:30:13.429 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:13.426+0000 7f8d82c30700 1 -- 172.21.15.98:0/2940608176 shutdown_connections 2023-07-14T19:30:13.429 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:13.426+0000 7f8d82c30700 1 --2- 172.21.15.98:0/2940608176 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f8d7c0ff710 0x7f8d7c106fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:30:13.429 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:13.426+0000 7f8d82c30700 1 --2- 172.21.15.98:0/2940608176 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f8d7c1000e0 0x7f8d7c1004f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:30:13.429 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:13.426+0000 7f8d82c30700 1 --2- 172.21.15.98:0/2940608176 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8d7c0fed80 0x7f8d7c0ff1d0 unknown :-1 s=CLOSED pgs=360 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:30:13.430 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:13.426+0000 7f8d82c30700 1 -- 172.21.15.98:0/2940608176 >> 172.21.15.98:0/2940608176 conn(0x7f8d7c0f98a0 msgr2=0x7f8d7c0fbcd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:30:13.430 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:13.426+0000 7f8d82c30700 1 -- 172.21.15.98:0/2940608176 shutdown_connections 2023-07-14T19:30:13.430 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:13.426+0000 7f8d82c30700 1 -- 172.21.15.98:0/2940608176 wait complete. 2023-07-14T19:30:13.430 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:13.427+0000 7f8d82c30700 1 Processor -- start 2023-07-14T19:30:13.430 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:13.427+0000 7f8d82c30700 1 -- start start 2023-07-14T19:30:13.430 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:13.427+0000 7f8d82c30700 1 --2- >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f8d7c0fed80 0x7f8d7c102b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:30:13.430 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:13.427+0000 7f8d82c30700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f8d7c0ff710 0x7f8d7c101170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:30:13.430 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:13.427+0000 7f8d82c30700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8d7c1000e0 0x7f8d7c1016b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:30:13.431 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:13.427+0000 7f8d82c30700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8d7c101e50 con 0x7f8d7c1000e0 2023-07-14T19:30:13.431 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:13.427+0000 7f8d82c30700 1 -- --> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8d7c101fc0 con 0x7f8d7c0fed80 2023-07-14T19:30:13.431 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:13.427+0000 7f8d82c30700 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8d7c19f960 con 0x7f8d7c0ff710 2023-07-14T19:30:13.431 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:13.427+0000 7f8d811cd700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8d7c1000e0 0x7f8d7c1016b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:30:13.431 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:13.427+0000 7f8d809cc700 1 --2- >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f8d7c0fed80 0x7f8d7c102b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:30:13.431 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:13.427+0000 7f8d811cd700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8d7c1000e0 0x7f8d7c1016b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:46766/0 (socket says 172.21.15.98:46766) 2023-07-14T19:30:13.431 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:13.427+0000 7f8d811cd700 1 -- 172.21.15.98:0/496039519 learned_addr learned my addr 172.21.15.98:0/496039519 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-07-14T19:30:13.432 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:13.427+0000 7f8d7bfff700 1 --2- 172.21.15.98:0/496039519 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f8d7c0ff710 0x7f8d7c101170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:30:13.432 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:13.427+0000 7f8d811cd700 1 -- 172.21.15.98:0/496039519 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f8d7c0ff710 msgr2=0x7f8d7c101170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:30:13.432 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:13.427+0000 7f8d811cd700 1 --2- 172.21.15.98:0/496039519 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f8d7c0ff710 0x7f8d7c101170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:30:13.432 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:13.427+0000 7f8d811cd700 1 -- 172.21.15.98:0/496039519 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f8d7c0fed80 msgr2=0x7f8d7c102b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:30:13.432 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:13.427+0000 7f8d811cd700 1 --2- 172.21.15.98:0/496039519 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f8d7c0fed80 0x7f8d7c102b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:30:13.432 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:13.427+0000 7f8d811cd700 1 -- 172.21.15.98:0/496039519 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8d680095f0 con 0x7f8d7c1000e0 2023-07-14T19:30:13.432 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:13.427+0000 7f8d811cd700 1 --2- 172.21.15.98:0/496039519 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8d7c1000e0 0x7f8d7c1016b0 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7f8d7400ada0 tx=0x7f8d7400c330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:30:13.432 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:13.428+0000 7f8d79ffb700 1 -- 172.21.15.98:0/496039519 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d7400f070 con 0x7f8d7c1000e0 2023-07-14T19:30:13.433 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:13.428+0000 7f8d79ffb700 1 -- 172.21.15.98:0/496039519 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f8d74007b30 con 0x7f8d7c1000e0 2023-07-14T19:30:13.433 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:13.428+0000 7f8d79ffb700 1 -- 172.21.15.98:0/496039519 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d740053e0 con 0x7f8d7c1000e0 2023-07-14T19:30:13.433 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:13.428+0000 7f8d82c30700 1 -- 172.21.15.98:0/496039519 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8d7c19fc40 con 0x7f8d7c1000e0 2023-07-14T19:30:13.433 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:13.428+0000 7f8d82c30700 1 -- 172.21.15.98:0/496039519 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8d7c1a0100 con 0x7f8d7c1000e0 2023-07-14T19:30:13.434 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:13.429+0000 7f8d82c30700 1 -- 172.21.15.98:0/496039519 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8d7c10a2d0 con 0x7f8d7c1000e0 2023-07-14T19:30:13.434 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:13.430+0000 7f8d79ffb700 1 -- 172.21.15.98:0/496039519 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 17) v1 ==== 135766+0+0 (secure 0 0 0) 0x7f8d7400c980 con 0x7f8d7c1000e0 2023-07-14T19:30:13.434 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:13.430+0000 7f8d79ffb700 1 --2- 172.21.15.98:0/496039519 >> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] conn(0x7f8d6c0a1220 0x7f8d6c0a36d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:30:13.434 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:13.430+0000 7f8d809cc700 1 --2- 172.21.15.98:0/496039519 >> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] conn(0x7f8d6c0a1220 0x7f8d6c0a36d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:30:13.434 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:13.430+0000 7f8d79ffb700 1 -- 172.21.15.98:0/496039519 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 8211+0+0 (secure 0 0 0) 0x7f8d740cbf90 con 0x7f8d7c1000e0 2023-07-14T19:30:13.435 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:13.431+0000 7f8d809cc700 1 --2- 172.21.15.98:0/496039519 >> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] conn(0x7f8d6c0a1220 0x7f8d6c0a36d0 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f8d7c067560 tx=0x7f8d7000b410 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:30:13.435 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:13.434+0000 7f8d79ffb700 1 -- 172.21.15.98:0/496039519 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f8d740956c0 con 0x7f8d7c1000e0 2023-07-14T19:30:13.627 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:13.627+0000 7f8d82c30700 1 -- 172.21.15.98:0/496039519 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f8d7c1a06b0 con 0x7f8d7c1000e0 2023-07-14T19:30:13.629 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:13.628+0000 7f8d79ffb700 1 -- 172.21.15.98:0/496039519 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v85) v1 ==== 74+0+22604 (secure 0 0 0) 0x7f8d74098ce0 con 0x7f8d7c1000e0 2023-07-14T19:30:13.629 INFO:teuthology.orchestra.run.smithi098.stdout: 2023-07-14T19:30:13.630 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":85,"fsid":"9adea648-227b-11ee-9b33-001a4aab830c","created":"2023-07-14T19:22:07.178854+0000","modified":"2023-07-14T19:30:03.314948+0000","last_up_change":"2023-07-14T19:28:19.646827+0000","last_in_change":"2023-07-14T19:28:02.018325+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-07-14T19:25:09.845827+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2023-07-14T19:28:58.952921+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"82","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2023-07-14T19:29:00.228272+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"83","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.690000057220459,"score_stable":1.690000057220459,"optimal_score":1,"raw_score_acting":1.690000057220459,"raw_score_stable":1.690000057220459,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"a3797e89-1754-41cf-9517-227e06ffdd72","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6802","nonce":120507228},{"type":"v1","addr":"172.21.15.98:6803","nonce":120507228}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6804","nonce":120507228},{"type":"v1","addr":"172.21.15.98:6805","nonce":120507228}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6808","nonce":120507228},{"type":"v1","addr":"172.21.15.98:6809","nonce":120507228}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6806","nonce":120507228},{"type":"v1","addr":"172.21.15.98:6807","nonce":120507228}]},"public_addr":"172.21.15.98:6803/120507228","cluster_addr":"172.21.15.98:6805/120507228","heartbeat_back_addr":"172.21.15.98:6809/120507228","heartbeat_front_addr":"172.21.15.98:6807/120507228","state":["exists","up"]},{"osd":1,"uuid":"2b001add-1aaa-4496-a403-7ef9725b80a0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.101:6800","nonce":2450584584},{"type":"v1","addr":"172.21.15.101:6801","nonce":2450584584}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.101:6802","nonce":2450584584},{"type":"v1","addr":"172.21.15.101:6803","nonce":2450584584}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.101:6806","nonce":2450584584},{"type":"v1","addr":"172.21.15.101:6807","nonce":2450584584}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.101:6804","nonce":2450584584},{"type":"v1","addr":"172.21.15.101:6805","nonce":2450584584}]},"public_addr":"172.21.15.101:6801/2450584584","cluster_addr":"172.21.15.101:6803/2450584584","heartbeat_back_addr":"172.21.15.101:6807/2450584584","heartbeat_front_addr":"172.21.15.101:6805/2450584584","state":["exists","up"]},{"osd":2,"uuid":"48e6d499-3ca9-424b-b0fb-3ba5f33b9fc9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":945038995},{"type":"v1","addr":"172.21.15.139:6801","nonce":945038995}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":945038995},{"type":"v1","addr":"172.21.15.139:6803","nonce":945038995}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":945038995},{"type":"v1","addr":"172.21.15.139:6807","nonce":945038995}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":945038995},{"type":"v1","addr":"172.21.15.139:6805","nonce":945038995}]},"public_addr":"172.21.15.139:6801/945038995","cluster_addr":"172.21.15.139:6803/945038995","heartbeat_back_addr":"172.21.15.139:6807/945038995","heartbeat_front_addr":"172.21.15.139:6805/945038995","state":["exists","up"]},{"osd":3,"uuid":"5530b265-31a5-447b-a1ea-17ebae9cfddf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6810","nonce":3095249665},{"type":"v1","addr":"172.21.15.98:6811","nonce":3095249665}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6812","nonce":3095249665},{"type":"v1","addr":"172.21.15.98:6813","nonce":3095249665}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6816","nonce":3095249665},{"type":"v1","addr":"172.21.15.98:6817","nonce":3095249665}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6814","nonce":3095249665},{"type":"v1","addr":"172.21.15.98:6815","nonce":3095249665}]},"public_addr":"172.21.15.98:6811/3095249665","cluster_addr":"172.21.15.98:6813/3095249665","heartbeat_back_addr":"172.21.15.98:6817/3095249665","heartbeat_front_addr":"172.21.15.98:6815/3095249665","state":["exists","up"]},{"osd":4,"uuid":"615d4642-a997-4f95-9e38-9fbdaa831400","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":32,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.101:6808","nonce":2844761406},{"type":"v1","addr":"172.21.15.101:6809","nonce":2844761406}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.101:6810","nonce":2844761406},{"type":"v1","addr":"172.21.15.101:6811","nonce":2844761406}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.101:6814","nonce":2844761406},{"type":"v1","addr":"172.21.15.101:6815","nonce":2844761406}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.101:6812","nonce":2844761406},{"type":"v1","addr":"172.21.15.101:6813","nonce":2844761406}]},"public_addr":"172.21.15.101:6809/2844761406","cluster_addr":"172.21.15.101:6811/2844761406","heartbeat_back_addr":"172.21.15.101:6815/2844761406","heartbeat_front_addr":"172.21.15.101:6813/2844761406","state":["exists","up"]},{"osd":5,"uuid":"415de5fa-607a-45a8-b732-28daf3be6dd3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":37,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":3532263042},{"type":"v1","addr":"172.21.15.139:6809","nonce":3532263042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6810","nonce":3532263042},{"type":"v1","addr":"172.21.15.139:6811","nonce":3532263042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6814","nonce":3532263042},{"type":"v1","addr":"172.21.15.139:6815","nonce":3532263042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":3532263042},{"type":"v1","addr":"172.21.15.139:6813","nonce":3532263042}]},"public_addr":"172.21.15.139:6809/3532263042","cluster_addr":"172.21.15.139:6811/3532263042","heartbeat_back_addr":"172.21.15.139:6815/3532263042","heartbeat_front_addr":"172.21.15.139:6813/3532263042","state":["exists","up"]},{"osd":6,"uuid":"d7f9d4ba-c309-4b75-a836-5a36384b528e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":45,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6818","nonce":658948743},{"type":"v1","addr":"172.21.15.98:6819","nonce":658948743}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6820","nonce":658948743},{"type":"v1","addr":"172.21.15.98:6821","nonce":658948743}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6824","nonce":658948743},{"type":"v1","addr":"172.21.15.98:6825","nonce":658948743}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6822","nonce":658948743},{"type":"v1","addr":"172.21.15.98:6823","nonce":658948743}]},"public_addr":"172.21.15.98:6819/658948743","cluster_addr":"172.21.15.98:6821/658948743","heartbeat_back_addr":"172.21.15.98:6825/658948743","heartbeat_front_addr":"172.21.15.98:6823/658948743","state":["exists","up"]},{"osd":7,"uuid":"eedb94d9-c5cd-4a93-8a8c-2648804b7195","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":50,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.101:6816","nonce":2881214066},{"type":"v1","addr":"172.21.15.101:6817","nonce":2881214066}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.101:6818","nonce":2881214066},{"type":"v1","addr":"172.21.15.101:6819","nonce":2881214066}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.101:6822","nonce":2881214066},{"type":"v1","addr":"172.21.15.101:6823","nonce":2881214066}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.101:6820","nonce":2881214066},{"type":"v1","addr":"172.21.15.101:6821","nonce":2881214066}]},"public_addr":"172.21.15.101:6817/2881214066","cluster_addr":"172.21.15.101:6819/2881214066","heartbeat_back_addr":"172.21.15.101:6823/2881214066","heartbeat_front_addr":"172.21.15.101:6821/2881214066","state":["exists","up"]},{"osd":8,"uuid":"6bfd77c2-5770-466e-aa6f-84b50c74ffb6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":57,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":3669514524},{"type":"v1","addr":"172.21.15.139:6817","nonce":3669514524}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":3669514524},{"type":"v1","addr":"172.21.15.139:6819","nonce":3669514524}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":3669514524},{"type":"v1","addr":"172.21.15.139:6823","nonce":3669514524}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6820","nonce":3669514524},{"type":"v1","addr":"172.21.15.139:6821","nonce":3669514524}]},"public_addr":"172.21.15.139:6817/3669514524","cluster_addr":"172.21.15.139:6819/3669514524","heartbeat_back_addr":"172.21.15.139:6823/3669514524","heartbeat_front_addr":"172.21.15.139:6821/3669514524","state":["exists","up"]},{"osd":9,"uuid":"19c5b54a-2242-4b23-b721-63be9559653d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":65,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6826","nonce":2893994080},{"type":"v1","addr":"172.21.15.98:6827","nonce":2893994080}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6828","nonce":2893994080},{"type":"v1","addr":"172.21.15.98:6829","nonce":2893994080}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6832","nonce":2893994080},{"type":"v1","addr":"172.21.15.98:6833","nonce":2893994080}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6830","nonce":2893994080},{"type":"v1","addr":"172.21.15.98:6831","nonce":2893994080}]},"public_addr":"172.21.15.98:6827/2893994080","cluster_addr":"172.21.15.98:6829/2893994080","heartbeat_back_addr":"172.21.15.98:6833/2893994080","heartbeat_front_addr":"172.21.15.98:6831/2893994080","state":["exists","up"]},{"osd":10,"uuid":"326f0209-ba9d-4b59-97cc-d61a758a8f29","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":70,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.101:6824","nonce":3545244225},{"type":"v1","addr":"172.21.15.101:6825","nonce":3545244225}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.101:6826","nonce":3545244225},{"type":"v1","addr":"172.21.15.101:6827","nonce":3545244225}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.101:6830","nonce":3545244225},{"type":"v1","addr":"172.21.15.101:6831","nonce":3545244225}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.101:6828","nonce":3545244225},{"type":"v1","addr":"172.21.15.101:6829","nonce":3545244225}]},"public_addr":"172.21.15.101:6825/3545244225","cluster_addr":"172.21.15.101:6827/3545244225","heartbeat_back_addr":"172.21.15.101:6831/3545244225","heartbeat_front_addr":"172.21.15.101:6829/3545244225","state":["exists","up"]},{"osd":11,"uuid":"c61c0320-1057-4f17-9bf1-7ad306ed7ed3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":78,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":2630023454},{"type":"v1","addr":"172.21.15.139:6825","nonce":2630023454}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6826","nonce":2630023454},{"type":"v1","addr":"172.21.15.139:6827","nonce":2630023454}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":2630023454},{"type":"v1","addr":"172.21.15.139:6831","nonce":2630023454}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":2630023454},{"type":"v1","addr":"172.21.15.139:6829","nonce":2630023454}]},"public_addr":"172.21.15.139:6825/2630023454","cluster_addr":"172.21.15.139:6827/2630023454","heartbeat_back_addr":"172.21.15.139:6831/2630023454","heartbeat_front_addr":"172.21.15.139:6829/2630023454","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T19:24:23.620028+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T19:24:45.779846+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T19:25:06.877949+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T19:25:27.994199+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T19:25:49.461139+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T19:26:10.409282+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T19:26:32.116390+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T19:26:53.536474+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T19:27:14.384839+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T19:27:35.635921+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T19:27:56.858744+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T19:28:18.342519+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.4","mappings":[{"from":2,"to":11}]},{"pgid":"2.f","mappings":[{"from":8,"to":11}]},{"pgid":"3.0","mappings":[{"from":4,"to":8}]},{"pgid":"3.1","mappings":[{"from":4,"to":9}]},{"pgid":"3.c","mappings":[{"from":3,"to":9}]},{"pgid":"3.17","mappings":[{"from":4,"to":6}]},{"pgid":"3.1e","mappings":[{"from":3,"to":9}]},{"pgid":"3.28","mappings":[{"from":5,"to":7}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.98:0/2164590253":"2023-07-15T19:23:01.520302+0000","172.21.15.98:0/2094734896":"2023-07-15T19:23:01.520302+0000","172.21.15.98:0/784745009":"2023-07-15T19:22:39.481502+0000","172.21.15.98:0/3657190818":"2023-07-15T19:22:39.481502+0000","172.21.15.98:6800/1732769087":"2023-07-15T19:23:01.520302+0000","172.21.15.98:0/3393542419":"2023-07-15T19:22:39.481502+0000","172.21.15.98:6800/749234665":"2023-07-15T19:22:39.481502+0000","172.21.15.98:6801/1732769087":"2023-07-15T19:23:01.520302+0000","172.21.15.98:6801/749234665":"2023-07-15T19:22:39.481502+0000","172.21.15.98:0/187822873":"2023-07-15T19:23:01.520302+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-07-14T19:30:13.631 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:13.631+0000 7f8d82c30700 1 -- 172.21.15.98:0/496039519 >> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] conn(0x7f8d6c0a1220 msgr2=0x7f8d6c0a36d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:30:13.631 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:13.631+0000 7f8d82c30700 1 --2- 172.21.15.98:0/496039519 >> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] conn(0x7f8d6c0a1220 0x7f8d6c0a36d0 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f8d7c067560 tx=0x7f8d7000b410 comp rx=0 tx=0).stop 2023-07-14T19:30:13.632 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:13.631+0000 7f8d82c30700 1 -- 172.21.15.98:0/496039519 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8d7c1000e0 msgr2=0x7f8d7c1016b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:30:13.632 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:13.631+0000 7f8d82c30700 1 --2- 172.21.15.98:0/496039519 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8d7c1000e0 0x7f8d7c1016b0 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7f8d7400ada0 tx=0x7f8d7400c330 comp rx=0 tx=0).stop 2023-07-14T19:30:13.632 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:13.631+0000 7f8d82c30700 1 -- 172.21.15.98:0/496039519 shutdown_connections 2023-07-14T19:30:13.632 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:13.631+0000 7f8d82c30700 1 --2- 172.21.15.98:0/496039519 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8d7c1000e0 0x7f8d7c1016b0 unknown :-1 s=CLOSED pgs=361 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:30:13.632 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:13.631+0000 7f8d82c30700 1 --2- 172.21.15.98:0/496039519 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f8d7c0fed80 0x7f8d7c102b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:30:13.632 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:13.631+0000 7f8d82c30700 1 --2- 172.21.15.98:0/496039519 >> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] conn(0x7f8d6c0a1220 0x7f8d6c0a36d0 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:30:13.633 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:13.631+0000 7f8d82c30700 1 --2- 172.21.15.98:0/496039519 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f8d7c0ff710 0x7f8d7c101170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:30:13.633 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:13.631+0000 7f8d82c30700 1 -- 172.21.15.98:0/496039519 >> 172.21.15.98:0/496039519 conn(0x7f8d7c0f98a0 msgr2=0x7f8d7c0fdd10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:30:13.633 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:13.631+0000 7f8d82c30700 1 -- 172.21.15.98:0/496039519 shutdown_connections 2023-07-14T19:30:13.633 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:13.631+0000 7f8d82c30700 1 -- 172.21.15.98:0/496039519 wait complete. 2023-07-14T19:30:13.644 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-07-14T19:25:09.845827+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2023-07-14T19:28:58.952921+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '82', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.8799999952316284, 'score_stable': 1.8799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2023-07-14T19:29:00.228272+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '83', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.690000057220459, 'score_stable': 1.690000057220459, 'optimal_score': 1, 'raw_score_acting': 1.690000057220459, 'raw_score_stable': 1.690000057220459, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-07-14T19:30:13.645 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-07-14T19:30:13.769 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:13.769+0000 7f3081cb0700 1 -- 172.21.15.98:0/1875640101 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f307c0fed40 msgr2=0x7f307c0ff1b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:30:13.770 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:13.769+0000 7f3081cb0700 1 --2- 172.21.15.98:0/1875640101 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f307c0fed40 0x7f307c0ff1b0 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7f306c009990 tx=0x7f306c009ca0 comp rx=0 tx=0).stop 2023-07-14T19:30:13.770 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:13.769+0000 7f3081cb0700 1 -- 172.21.15.98:0/1875640101 shutdown_connections 2023-07-14T19:30:13.770 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:13.769+0000 7f3081cb0700 1 --2- 172.21.15.98:0/1875640101 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f307c0ff6f0 0x7f307c106fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:30:13.771 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:13.769+0000 7f3081cb0700 1 --2- 172.21.15.98:0/1875640101 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f307c100180 0x7f307c100590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:30:13.771 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:13.769+0000 7f3081cb0700 1 --2- 172.21.15.98:0/1875640101 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f307c0fed40 0x7f307c0ff1b0 unknown :-1 s=CLOSED pgs=362 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:30:13.771 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:13.769+0000 7f3081cb0700 1 -- 172.21.15.98:0/1875640101 >> 172.21.15.98:0/1875640101 conn(0x7f307c0f98a0 msgr2=0x7f307c0fbcd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:30:13.771 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:13.769+0000 7f3081cb0700 1 -- 172.21.15.98:0/1875640101 shutdown_connections 2023-07-14T19:30:13.771 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:13.769+0000 7f3081cb0700 1 -- 172.21.15.98:0/1875640101 wait complete. 2023-07-14T19:30:13.771 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:13.769+0000 7f3081cb0700 1 Processor -- start 2023-07-14T19:30:13.774 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:13.770+0000 7f3081cb0700 1 -- start start 2023-07-14T19:30:13.774 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:13.770+0000 7f3081cb0700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f307c0fed40 0x7f307c195960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:30:13.774 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:13.770+0000 7f3081cb0700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f307c0ff6f0 0x7f307c195ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:30:13.774 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:13.770+0000 7f3081cb0700 1 --2- >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f307c100180 0x7f307c19cf10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:30:13.775 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:13.770+0000 7f3081cb0700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f307c19d4e0 con 0x7f307c0fed40 2023-07-14T19:30:13.775 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:13.770+0000 7f3081cb0700 1 -- --> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f307c19d620 con 0x7f307c100180 2023-07-14T19:30:13.775 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:13.770+0000 7f3081cb0700 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f307c19d7b0 con 0x7f307c0ff6f0 2023-07-14T19:30:13.775 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:13.770+0000 7f307b7fe700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f307c0fed40 0x7f307c195960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:30:13.775 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:13.770+0000 7f307b7fe700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f307c0fed40 0x7f307c195960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:46776/0 (socket says 172.21.15.98:46776) 2023-07-14T19:30:13.775 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:13.770+0000 7f307b7fe700 1 -- 172.21.15.98:0/1226672077 learned_addr learned my addr 172.21.15.98:0/1226672077 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-07-14T19:30:13.775 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:13.770+0000 7f307bfff700 1 --2- 172.21.15.98:0/1226672077 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f307c100180 0x7f307c19cf10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:30:13.776 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:13.770+0000 7f307affd700 1 --2- 172.21.15.98:0/1226672077 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f307c0ff6f0 0x7f307c195ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:30:13.776 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:13.770+0000 7f307b7fe700 1 -- 172.21.15.98:0/1226672077 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f307c0ff6f0 msgr2=0x7f307c195ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:30:13.776 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:13.770+0000 7f307b7fe700 1 --2- 172.21.15.98:0/1226672077 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f307c0ff6f0 0x7f307c195ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:30:13.776 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:13.770+0000 7f307b7fe700 1 -- 172.21.15.98:0/1226672077 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f307c100180 msgr2=0x7f307c19cf10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:30:13.776 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:13.770+0000 7f307b7fe700 1 --2- 172.21.15.98:0/1226672077 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f307c100180 0x7f307c19cf10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:30:13.776 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:13.770+0000 7f307b7fe700 1 -- 172.21.15.98:0/1226672077 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f306c0095f0 con 0x7f307c0fed40 2023-07-14T19:30:13.776 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:13.770+0000 7f307b7fe700 1 --2- 172.21.15.98:0/1226672077 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f307c0fed40 0x7f307c195960 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7f307000b530 tx=0x7f307000b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:30:13.777 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:13.770+0000 7f3078ff9700 1 -- 172.21.15.98:0/1226672077 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3070011830 con 0x7f307c0fed40 2023-07-14T19:30:13.777 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:13.771+0000 7f3078ff9700 1 -- 172.21.15.98:0/1226672077 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f3070011e50 con 0x7f307c0fed40 2023-07-14T19:30:13.777 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:13.771+0000 7f3081cb0700 1 -- 172.21.15.98:0/1226672077 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f307c19da90 con 0x7f307c0fed40 2023-07-14T19:30:13.777 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:13.771+0000 7f3078ff9700 1 -- 172.21.15.98:0/1226672077 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f307000f2e0 con 0x7f307c0fed40 2023-07-14T19:30:13.777 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:13.771+0000 7f3081cb0700 1 -- 172.21.15.98:0/1226672077 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f307c19dfa0 con 0x7f307c0fed40 2023-07-14T19:30:13.777 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:13.771+0000 7f3081cb0700 1 -- 172.21.15.98:0/1226672077 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f307c10a350 con 0x7f307c0fed40 2023-07-14T19:30:13.777 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:13.773+0000 7f3078ff9700 1 -- 172.21.15.98:0/1226672077 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 17) v1 ==== 135766+0+0 (secure 0 0 0) 0x7f3070010380 con 0x7f307c0fed40 2023-07-14T19:30:13.778 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:13.773+0000 7f3078ff9700 1 --2- 172.21.15.98:0/1226672077 >> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] conn(0x7f30640a11e0 0x7f30640a3690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:30:13.778 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:13.773+0000 7f3078ff9700 1 -- 172.21.15.98:0/1226672077 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 8211+0+0 (secure 0 0 0) 0x7f30700cca80 con 0x7f307c0fed40 2023-07-14T19:30:13.778 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:13.773+0000 7f307affd700 1 --2- 172.21.15.98:0/1226672077 >> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] conn(0x7f30640a11e0 0x7f30640a3690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:30:13.778 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:13.774+0000 7f307affd700 1 --2- 172.21.15.98:0/1226672077 >> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] conn(0x7f30640a11e0 0x7f30640a3690 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f306c006010 tx=0x7f306c00b540 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:30:13.780 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:13.776+0000 7f3078ff9700 1 -- 172.21.15.98:0/1226672077 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f3070096190 con 0x7f307c0fed40 2023-07-14T19:30:13.965 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:13.965+0000 7f3081cb0700 1 -- 172.21.15.98:0/1226672077 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f307c19e5b0 con 0x7f307c0fed40 2023-07-14T19:30:13.966 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:13.966+0000 7f3078ff9700 1 -- 172.21.15.98:0/1226672077 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v85) v1 ==== 93+0+10 (secure 0 0 0) 0x7f30700997b0 con 0x7f307c0fed40 2023-07-14T19:30:13.966 INFO:teuthology.orchestra.run.smithi098.stdout:pg_num: 1 2023-07-14T19:30:13.968 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:13.967+0000 7f3081cb0700 1 -- 172.21.15.98:0/1226672077 >> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] conn(0x7f30640a11e0 msgr2=0x7f30640a3690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:30:13.968 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:13.967+0000 7f3081cb0700 1 --2- 172.21.15.98:0/1226672077 >> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] conn(0x7f30640a11e0 0x7f30640a3690 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f306c006010 tx=0x7f306c00b540 comp rx=0 tx=0).stop 2023-07-14T19:30:13.968 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:13.967+0000 7f3081cb0700 1 -- 172.21.15.98:0/1226672077 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f307c0fed40 msgr2=0x7f307c195960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:30:13.968 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:13.967+0000 7f3081cb0700 1 --2- 172.21.15.98:0/1226672077 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f307c0fed40 0x7f307c195960 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7f307000b530 tx=0x7f307000b840 comp rx=0 tx=0).stop 2023-07-14T19:30:13.969 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:13.968+0000 7f3081cb0700 1 -- 172.21.15.98:0/1226672077 shutdown_connections 2023-07-14T19:30:13.969 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:13.968+0000 7f3081cb0700 1 --2- 172.21.15.98:0/1226672077 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f307c100180 0x7f307c19cf10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:30:13.969 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:13.968+0000 7f3081cb0700 1 --2- 172.21.15.98:0/1226672077 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f307c0fed40 0x7f307c195960 unknown :-1 s=CLOSED pgs=363 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:30:13.969 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:13.968+0000 7f3081cb0700 1 --2- 172.21.15.98:0/1226672077 >> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] conn(0x7f30640a11e0 0x7f30640a3690 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:30:13.969 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:13.968+0000 7f3081cb0700 1 --2- 172.21.15.98:0/1226672077 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f307c0ff6f0 0x7f307c195ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:30:13.969 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:13.968+0000 7f3081cb0700 1 -- 172.21.15.98:0/1226672077 >> 172.21.15.98:0/1226672077 conn(0x7f307c0f98a0 msgr2=0x7f307c0fdc50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:30:13.970 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:13.968+0000 7f3081cb0700 1 -- 172.21.15.98:0/1226672077 shutdown_connections 2023-07-14T19:30:13.970 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:13.968+0000 7f3081cb0700 1 -- 172.21.15.98:0/1226672077 wait complete. 2023-07-14T19:30:13.979 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:30:13 smithi098 ceph-mon[127486]: from='client.? 172.21.15.98:0/3584767653' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2023-07-14T19:30:13.980 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:30:13 smithi098 ceph-mon[127486]: from='client.? 172.21.15.98:0/2454697739' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-07-14T19:30:13.980 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:30:13 smithi098 ceph-mon[127486]: from='client.? 172.21.15.98:0/496039519' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-07-14T19:30:13.981 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-07-14T19:30:14.105 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:14.104+0000 7fc975c43700 1 -- 172.21.15.98:0/3049477930 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7fc9700fe2c0 msgr2=0x7fc9700fe6d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:30:14.105 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:14.104+0000 7fc975c43700 1 --2- 172.21.15.98:0/3049477930 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7fc9700fe2c0 0x7fc9700fe6d0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7fc96400aaa0 tx=0x7fc96400adb0 comp rx=0 tx=0).stop 2023-07-14T19:30:14.106 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:14.104+0000 7fc975c43700 1 -- 172.21.15.98:0/3049477930 shutdown_connections 2023-07-14T19:30:14.106 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:14.104+0000 7fc975c43700 1 --2- 172.21.15.98:0/3049477930 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fc9700ffe40 0x7fc970106700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:30:14.106 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:14.104+0000 7fc975c43700 1 --2- 172.21.15.98:0/3049477930 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7fc9700fe2c0 0x7fc9700fe6d0 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:30:14.106 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:14.104+0000 7fc975c43700 1 --2- 172.21.15.98:0/3049477930 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc9700ff4b0 0x7fc9700ff900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:30:14.106 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:14.104+0000 7fc975c43700 1 -- 172.21.15.98:0/3049477930 >> 172.21.15.98:0/3049477930 conn(0x7fc9700f98e0 msgr2=0x7fc9700fbd10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:30:14.106 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:14.104+0000 7fc975c43700 1 -- 172.21.15.98:0/3049477930 shutdown_connections 2023-07-14T19:30:14.107 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:14.104+0000 7fc975c43700 1 -- 172.21.15.98:0/3049477930 wait complete. 2023-07-14T19:30:14.107 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:14.105+0000 7fc975c43700 1 Processor -- start 2023-07-14T19:30:14.108 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:14.105+0000 7fc975c43700 1 -- start start 2023-07-14T19:30:14.109 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:14.105+0000 7fc975c43700 1 --2- >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7fc9700fe2c0 0x7fc970102250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:30:14.109 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:14.105+0000 7fc975c43700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fc9700ff4b0 0x7fc970100890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:30:14.109 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:14.105+0000 7fc975c43700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc9700ffe40 0x7fc970100dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:30:14.109 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:14.105+0000 7fc975c43700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc970101570 con 0x7fc9700ffe40 2023-07-14T19:30:14.109 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:14.105+0000 7fc975c43700 1 -- --> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc9701016e0 con 0x7fc9700fe2c0 2023-07-14T19:30:14.109 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:14.105+0000 7fc975c43700 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc97019fa10 con 0x7fc9700ff4b0 2023-07-14T19:30:14.110 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:14.105+0000 7fc96ffff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc9700ffe40 0x7fc970100dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:30:14.110 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:14.105+0000 7fc96ffff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc9700ffe40 0x7fc970100dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:46792/0 (socket says 172.21.15.98:46792) 2023-07-14T19:30:14.110 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:14.105+0000 7fc96ffff700 1 -- 172.21.15.98:0/576040503 learned_addr learned my addr 172.21.15.98:0/576040503 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-07-14T19:30:14.110 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:14.106+0000 7fc96effd700 1 --2- 172.21.15.98:0/576040503 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fc9700ff4b0 0x7fc970100890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:30:14.110 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:14.106+0000 7fc96f7fe700 1 --2- 172.21.15.98:0/576040503 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7fc9700fe2c0 0x7fc970102250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:30:14.110 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:14.106+0000 7fc96ffff700 1 -- 172.21.15.98:0/576040503 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fc9700ff4b0 msgr2=0x7fc970100890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:30:14.110 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:14.106+0000 7fc96ffff700 1 --2- 172.21.15.98:0/576040503 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fc9700ff4b0 0x7fc970100890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:30:14.111 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:14.106+0000 7fc96ffff700 1 -- 172.21.15.98:0/576040503 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7fc9700fe2c0 msgr2=0x7fc970102250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:30:14.111 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:14.106+0000 7fc96ffff700 1 --2- 172.21.15.98:0/576040503 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7fc9700fe2c0 0x7fc970102250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:30:14.111 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:14.106+0000 7fc96ffff700 1 -- 172.21.15.98:0/576040503 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc96400a700 con 0x7fc9700ffe40 2023-07-14T19:30:14.111 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:14.106+0000 7fc96ffff700 1 --2- 172.21.15.98:0/576040503 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc9700ffe40 0x7fc970100dd0 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7fc968007ee0 tx=0x7fc96800d3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:30:14.111 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:14.106+0000 7fc96cff9700 1 -- 172.21.15.98:0/576040503 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc968017070 con 0x7fc9700ffe40 2023-07-14T19:30:14.111 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:14.106+0000 7fc975c43700 1 -- 172.21.15.98:0/576040503 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc97019fcf0 con 0x7fc9700ffe40 2023-07-14T19:30:14.111 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:14.106+0000 7fc96cff9700 1 -- 172.21.15.98:0/576040503 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fc96800f040 con 0x7fc9700ffe40 2023-07-14T19:30:14.112 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:14.106+0000 7fc96cff9700 1 -- 172.21.15.98:0/576040503 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc968005740 con 0x7fc9700ffe40 2023-07-14T19:30:14.112 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:14.106+0000 7fc975c43700 1 -- 172.21.15.98:0/576040503 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc9701a01b0 con 0x7fc9700ffe40 2023-07-14T19:30:14.112 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:14.107+0000 7fc975c43700 1 -- 172.21.15.98:0/576040503 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc9701092c0 con 0x7fc9700ffe40 2023-07-14T19:30:14.112 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:14.108+0000 7fc96cff9700 1 -- 172.21.15.98:0/576040503 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 17) v1 ==== 135766+0+0 (secure 0 0 0) 0x7fc96800dc60 con 0x7fc9700ffe40 2023-07-14T19:30:14.113 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:14.109+0000 7fc96cff9700 1 --2- 172.21.15.98:0/576040503 >> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] conn(0x7fc9580a11e0 0x7fc9580a3690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:30:14.113 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:14.109+0000 7fc96f7fe700 1 --2- 172.21.15.98:0/576040503 >> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] conn(0x7fc9580a11e0 0x7fc9580a3690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:30:14.113 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:14.109+0000 7fc96cff9700 1 -- 172.21.15.98:0/576040503 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 8211+0+0 (secure 0 0 0) 0x7fc9680cc8e0 con 0x7fc9700ffe40 2023-07-14T19:30:14.113 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:14.109+0000 7fc96f7fe700 1 --2- 172.21.15.98:0/576040503 >> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] conn(0x7fc9580a11e0 0x7fc9580a3690 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7fc96400c6d0 tx=0x7fc964005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:30:14.113 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:14.112+0000 7fc96cff9700 1 -- 172.21.15.98:0/576040503 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7fc968095ee0 con 0x7fc9700ffe40 2023-07-14T19:30:14.191 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:30:13 smithi101 ceph-mon[128807]: from='client.? 172.21.15.98:0/3584767653' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2023-07-14T19:30:14.191 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:30:13 smithi101 ceph-mon[128807]: from='client.? 172.21.15.98:0/2454697739' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-07-14T19:30:14.192 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:30:13 smithi101 ceph-mon[128807]: from='client.? 172.21.15.98:0/496039519' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-07-14T19:30:14.220 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:30:13 smithi139 ceph-mon[129293]: from='client.? 172.21.15.98:0/3584767653' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2023-07-14T19:30:14.220 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:30:13 smithi139 ceph-mon[129293]: from='client.? 172.21.15.98:0/2454697739' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-07-14T19:30:14.220 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:30:13 smithi139 ceph-mon[129293]: from='client.? 172.21.15.98:0/496039519' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-07-14T19:30:14.302 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:14.302+0000 7fc975c43700 1 -- 172.21.15.98:0/576040503 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fc970109480 con 0x7fc9700ffe40 2023-07-14T19:30:14.303 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:14.302+0000 7fc96cff9700 1 -- 172.21.15.98:0/576040503 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v85) v1 ==== 104+0+11 (secure 0 0 0) 0x7fc968099500 con 0x7fc9700ffe40 2023-07-14T19:30:14.303 INFO:teuthology.orchestra.run.smithi098.stdout:pg_num: 32 2023-07-14T19:30:14.305 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:14.304+0000 7fc975c43700 1 -- 172.21.15.98:0/576040503 >> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] conn(0x7fc9580a11e0 msgr2=0x7fc9580a3690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:30:14.305 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:14.304+0000 7fc975c43700 1 --2- 172.21.15.98:0/576040503 >> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] conn(0x7fc9580a11e0 0x7fc9580a3690 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7fc96400c6d0 tx=0x7fc964005fd0 comp rx=0 tx=0).stop 2023-07-14T19:30:14.305 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:14.304+0000 7fc975c43700 1 -- 172.21.15.98:0/576040503 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc9700ffe40 msgr2=0x7fc970100dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:30:14.305 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:14.304+0000 7fc975c43700 1 --2- 172.21.15.98:0/576040503 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc9700ffe40 0x7fc970100dd0 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7fc968007ee0 tx=0x7fc96800d3b0 comp rx=0 tx=0).stop 2023-07-14T19:30:14.305 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:14.304+0000 7fc975c43700 1 -- 172.21.15.98:0/576040503 shutdown_connections 2023-07-14T19:30:14.305 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:14.304+0000 7fc975c43700 1 --2- 172.21.15.98:0/576040503 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc9700ffe40 0x7fc970100dd0 unknown :-1 s=CLOSED pgs=364 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:30:14.306 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:14.304+0000 7fc975c43700 1 --2- 172.21.15.98:0/576040503 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7fc9700fe2c0 0x7fc970102250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:30:14.306 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:14.304+0000 7fc975c43700 1 --2- 172.21.15.98:0/576040503 >> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] conn(0x7fc9580a11e0 0x7fc9580a3690 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:30:14.306 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:14.304+0000 7fc975c43700 1 --2- 172.21.15.98:0/576040503 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fc9700ff4b0 0x7fc970100890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:30:14.306 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:14.304+0000 7fc975c43700 1 -- 172.21.15.98:0/576040503 >> 172.21.15.98:0/576040503 conn(0x7fc9700f98e0 msgr2=0x7fc9700fbb00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:30:14.306 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:14.304+0000 7fc975c43700 1 -- 172.21.15.98:0/576040503 shutdown_connections 2023-07-14T19:30:14.306 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:14.304+0000 7fc975c43700 1 -- 172.21.15.98:0/576040503 wait complete. 2023-07-14T19:30:14.317 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-07-14T19:30:14.440 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:14.440+0000 7f0c2f6f1700 1 -- 172.21.15.98:0/1062015169 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0c28100180 msgr2=0x7f0c28100590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:30:14.441 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:14.440+0000 7f0c2f6f1700 1 --2- 172.21.15.98:0/1062015169 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0c28100180 0x7f0c28100590 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7f0c24009990 tx=0x7f0c24009ca0 comp rx=0 tx=0).stop 2023-07-14T19:30:14.441 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:14.440+0000 7f0c2f6f1700 1 -- 172.21.15.98:0/1062015169 shutdown_connections 2023-07-14T19:30:14.441 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:14.440+0000 7f0c2f6f1700 1 --2- 172.21.15.98:0/1062015169 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f0c280ff6f0 0x7f0c28106fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:30:14.441 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:14.440+0000 7f0c2f6f1700 1 --2- 172.21.15.98:0/1062015169 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0c28100180 0x7f0c28100590 unknown :-1 s=CLOSED pgs=365 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:30:14.441 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:14.440+0000 7f0c2f6f1700 1 --2- 172.21.15.98:0/1062015169 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f0c280fed40 0x7f0c280ff1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:30:14.441 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:14.440+0000 7f0c2f6f1700 1 -- 172.21.15.98:0/1062015169 >> 172.21.15.98:0/1062015169 conn(0x7f0c280f98a0 msgr2=0x7f0c280fbcd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:30:14.442 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:14.440+0000 7f0c2f6f1700 1 -- 172.21.15.98:0/1062015169 shutdown_connections 2023-07-14T19:30:14.442 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:14.440+0000 7f0c2f6f1700 1 -- 172.21.15.98:0/1062015169 wait complete. 2023-07-14T19:30:14.443 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:14.440+0000 7f0c2f6f1700 1 Processor -- start 2023-07-14T19:30:14.444 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:14.441+0000 7f0c2f6f1700 1 -- start start 2023-07-14T19:30:14.444 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:14.441+0000 7f0c2f6f1700 1 --2- >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f0c280fed40 0x7f0c281959a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:30:14.444 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:14.441+0000 7f0c2f6f1700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f0c280ff6f0 0x7f0c28195ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:30:14.444 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:14.441+0000 7f0c2f6f1700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0c28100180 0x7f0c2819cf50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:30:14.444 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:14.441+0000 7f0c2f6f1700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0c2819d520 con 0x7f0c28100180 2023-07-14T19:30:14.445 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:14.441+0000 7f0c2f6f1700 1 -- --> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0c2819d660 con 0x7f0c280fed40 2023-07-14T19:30:14.445 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:14.441+0000 7f0c2f6f1700 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0c2819d7f0 con 0x7f0c280ff6f0 2023-07-14T19:30:14.445 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:14.441+0000 7f0c2dc8e700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0c28100180 0x7f0c2819cf50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:30:14.445 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:14.441+0000 7f0c2dc8e700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0c28100180 0x7f0c2819cf50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:46804/0 (socket says 172.21.15.98:46804) 2023-07-14T19:30:14.445 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:14.441+0000 7f0c2dc8e700 1 -- 172.21.15.98:0/842252631 learned_addr learned my addr 172.21.15.98:0/842252631 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-07-14T19:30:14.445 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:14.441+0000 7f0c2d48d700 1 --2- 172.21.15.98:0/842252631 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f0c280fed40 0x7f0c281959a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:30:14.445 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:14.441+0000 7f0c2cc8c700 1 --2- 172.21.15.98:0/842252631 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f0c280ff6f0 0x7f0c28195ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:30:14.445 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:14.441+0000 7f0c2dc8e700 1 -- 172.21.15.98:0/842252631 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f0c280ff6f0 msgr2=0x7f0c28195ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:30:14.446 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:14.441+0000 7f0c2dc8e700 1 --2- 172.21.15.98:0/842252631 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f0c280ff6f0 0x7f0c28195ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:30:14.446 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:14.441+0000 7f0c2dc8e700 1 -- 172.21.15.98:0/842252631 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f0c280fed40 msgr2=0x7f0c281959a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:30:14.446 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:14.441+0000 7f0c2dc8e700 1 --2- 172.21.15.98:0/842252631 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f0c280fed40 0x7f0c281959a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:30:14.446 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:14.441+0000 7f0c2dc8e700 1 -- 172.21.15.98:0/842252631 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0c240095f0 con 0x7f0c28100180 2023-07-14T19:30:14.446 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:14.441+0000 7f0c2dc8e700 1 --2- 172.21.15.98:0/842252631 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0c28100180 0x7f0c2819cf50 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x7f0c2000ada0 tx=0x7f0c2000c330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:30:14.446 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:14.441+0000 7f0c1e7fc700 1 -- 172.21.15.98:0/842252631 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c2000f070 con 0x7f0c28100180 2023-07-14T19:30:14.446 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:14.441+0000 7f0c1e7fc700 1 -- 172.21.15.98:0/842252631 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f0c20007b30 con 0x7f0c28100180 2023-07-14T19:30:14.447 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:14.441+0000 7f0c1e7fc700 1 -- 172.21.15.98:0/842252631 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c200053e0 con 0x7f0c28100180 2023-07-14T19:30:14.447 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:14.441+0000 7f0c2f6f1700 1 -- 172.21.15.98:0/842252631 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0c2819dad0 con 0x7f0c28100180 2023-07-14T19:30:14.447 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:14.441+0000 7f0c2f6f1700 1 -- 172.21.15.98:0/842252631 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0c2819dfe0 con 0x7f0c28100180 2023-07-14T19:30:14.447 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:14.442+0000 7f0c2f6f1700 1 -- 172.21.15.98:0/842252631 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0c2810a2d0 con 0x7f0c28100180 2023-07-14T19:30:14.447 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:14.443+0000 7f0c1e7fc700 1 -- 172.21.15.98:0/842252631 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 17) v1 ==== 135766+0+0 (secure 0 0 0) 0x7f0c2000c980 con 0x7f0c28100180 2023-07-14T19:30:14.448 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:14.444+0000 7f0c1e7fc700 1 --2- 172.21.15.98:0/842252631 >> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] conn(0x7f0c140a1260 0x7f0c140a3710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:30:14.448 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:14.444+0000 7f0c2d48d700 1 --2- 172.21.15.98:0/842252631 >> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] conn(0x7f0c140a1260 0x7f0c140a3710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:30:14.448 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:14.444+0000 7f0c1e7fc700 1 -- 172.21.15.98:0/842252631 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 8211+0+0 (secure 0 0 0) 0x7f0c200cc040 con 0x7f0c28100180 2023-07-14T19:30:14.448 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:14.444+0000 7f0c2d48d700 1 --2- 172.21.15.98:0/842252631 >> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] conn(0x7f0c140a1260 0x7f0c140a3710 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f0c2400b5c0 tx=0x7f0c24005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:30:14.449 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:14.448+0000 7f0c1e7fc700 1 -- 172.21.15.98:0/842252631 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f0c20095670 con 0x7f0c28100180 2023-07-14T19:30:14.635 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:14.635+0000 7f0c2f6f1700 1 -- 172.21.15.98:0/842252631 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f0c2819e5f0 con 0x7f0c28100180 2023-07-14T19:30:14.636 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:14.635+0000 7f0c1e7fc700 1 -- 172.21.15.98:0/842252631 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v85) v1 ==== 100+0+11 (secure 0 0 0) 0x7f0c20098c90 con 0x7f0c28100180 2023-07-14T19:30:14.636 INFO:teuthology.orchestra.run.smithi098.stdout:pg_num: 64 2023-07-14T19:30:14.638 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:14.638+0000 7f0c2f6f1700 1 -- 172.21.15.98:0/842252631 >> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] conn(0x7f0c140a1260 msgr2=0x7f0c140a3710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:30:14.638 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:14.638+0000 7f0c2f6f1700 1 --2- 172.21.15.98:0/842252631 >> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] conn(0x7f0c140a1260 0x7f0c140a3710 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f0c2400b5c0 tx=0x7f0c24005fd0 comp rx=0 tx=0).stop 2023-07-14T19:30:14.638 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:14.638+0000 7f0c2f6f1700 1 -- 172.21.15.98:0/842252631 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0c28100180 msgr2=0x7f0c2819cf50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:30:14.638 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:14.638+0000 7f0c2f6f1700 1 --2- 172.21.15.98:0/842252631 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0c28100180 0x7f0c2819cf50 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x7f0c2000ada0 tx=0x7f0c2000c330 comp rx=0 tx=0).stop 2023-07-14T19:30:14.639 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:14.638+0000 7f0c2f6f1700 1 -- 172.21.15.98:0/842252631 shutdown_connections 2023-07-14T19:30:14.639 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:14.638+0000 7f0c2f6f1700 1 --2- 172.21.15.98:0/842252631 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0c28100180 0x7f0c2819cf50 unknown :-1 s=CLOSED pgs=366 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:30:14.639 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:14.638+0000 7f0c2f6f1700 1 --2- 172.21.15.98:0/842252631 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f0c280fed40 0x7f0c281959a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:30:14.639 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:14.638+0000 7f0c2f6f1700 1 --2- 172.21.15.98:0/842252631 >> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] conn(0x7f0c140a1260 0x7f0c140a3710 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:30:14.639 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:14.638+0000 7f0c2f6f1700 1 --2- 172.21.15.98:0/842252631 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f0c280ff6f0 0x7f0c28195ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:30:14.639 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:14.638+0000 7f0c2f6f1700 1 -- 172.21.15.98:0/842252631 >> 172.21.15.98:0/842252631 conn(0x7f0c280f98a0 msgr2=0x7f0c280fdc70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:30:14.639 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:14.638+0000 7f0c2f6f1700 1 -- 172.21.15.98:0/842252631 shutdown_connections 2023-07-14T19:30:14.640 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:14.638+0000 7f0c2f6f1700 1 -- 172.21.15.98:0/842252631 wait complete. 2023-07-14T19:30:14.651 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-07-14T19:30:14.773 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:14.772+0000 7f508636c700 1 -- 172.21.15.98:0/832852830 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f50800fe6f0 msgr2=0x7f50801082b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:30:14.773 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:14.772+0000 7f508636c700 1 --2- 172.21.15.98:0/832852830 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f50800fe6f0 0x7f50801082b0 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x7f507800b0a0 tx=0x7f507800b3b0 comp rx=0 tx=0).stop 2023-07-14T19:30:14.773 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:14.772+0000 7f508636c700 1 -- 172.21.15.98:0/832852830 shutdown_connections 2023-07-14T19:30:14.773 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:14.772+0000 7f508636c700 1 --2- 172.21.15.98:0/832852830 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f50801012d0 0x7f50801016e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:30:14.774 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:14.772+0000 7f508636c700 1 --2- 172.21.15.98:0/832852830 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f50800fe6f0 0x7f50801082b0 unknown :-1 s=CLOSED pgs=367 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:30:14.774 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:14.772+0000 7f508636c700 1 --2- 172.21.15.98:0/832852830 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f50800fdd40 0x7f50800fe1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:30:14.774 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:14.772+0000 7f508636c700 1 -- 172.21.15.98:0/832852830 >> 172.21.15.98:0/832852830 conn(0x7f50800f9990 msgr2=0x7f50800fbda0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:30:14.774 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:14.772+0000 7f508636c700 1 -- 172.21.15.98:0/832852830 shutdown_connections 2023-07-14T19:30:14.774 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:14.772+0000 7f508636c700 1 -- 172.21.15.98:0/832852830 wait complete. 2023-07-14T19:30:14.774 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:14.773+0000 7f508636c700 1 Processor -- start 2023-07-14T19:30:14.775 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:14.773+0000 7f508636c700 1 -- start start 2023-07-14T19:30:14.778 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:14.773+0000 7f508636c700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f50800fdd40 0x7f5080195970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:30:14.778 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:14.773+0000 7f508636c700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f50800fe6f0 0x7f5080195eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:30:14.778 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:14.773+0000 7f508636c700 1 --2- >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f50801012d0 0x7f508019cf20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:30:14.779 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:14.773+0000 7f508636c700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f508019d4f0 con 0x7f50800fe6f0 2023-07-14T19:30:14.779 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:14.773+0000 7f508636c700 1 -- --> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f508019d630 con 0x7f50801012d0 2023-07-14T19:30:14.779 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:14.773+0000 7f508636c700 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f508019d7c0 con 0x7f50800fdd40 2023-07-14T19:30:14.780 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:14.773+0000 7f507f7fe700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f50800fe6f0 0x7f5080195eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:30:14.780 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:14.773+0000 7f507f7fe700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f50800fe6f0 0x7f5080195eb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:46816/0 (socket says 172.21.15.98:46816) 2023-07-14T19:30:14.780 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:14.773+0000 7f507f7fe700 1 -- 172.21.15.98:0/852855985 learned_addr learned my addr 172.21.15.98:0/852855985 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-07-14T19:30:14.781 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:14.773+0000 7f507ffff700 1 --2- 172.21.15.98:0/852855985 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f50800fdd40 0x7f5080195970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:30:14.781 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:14.773+0000 7f507f7fe700 1 -- 172.21.15.98:0/852855985 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f50800fdd40 msgr2=0x7f5080195970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:30:14.781 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:14.773+0000 7f507f7fe700 1 --2- 172.21.15.98:0/852855985 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f50800fdd40 0x7f5080195970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:30:14.782 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:14.773+0000 7f507f7fe700 1 -- 172.21.15.98:0/852855985 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f50801012d0 msgr2=0x7f508019cf20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:30:14.782 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:14.773+0000 7f5084909700 1 --2- 172.21.15.98:0/852855985 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f50801012d0 0x7f508019cf20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:30:14.782 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:14.774+0000 7f507f7fe700 1 --2- 172.21.15.98:0/852855985 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f50801012d0 0x7f508019cf20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:30:14.782 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:14.774+0000 7f507f7fe700 1 -- 172.21.15.98:0/852855985 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5078009d00 con 0x7f50800fe6f0 2023-07-14T19:30:14.783 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:14.774+0000 7f507f7fe700 1 --2- 172.21.15.98:0/852855985 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f50800fe6f0 0x7f5080195eb0 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7f507000c990 tx=0x7f507000cca0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:30:14.783 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:14.774+0000 7f507d7fa700 1 -- 172.21.15.98:0/852855985 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5070007b00 con 0x7f50800fe6f0 2023-07-14T19:30:14.783 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:14.774+0000 7f507d7fa700 1 -- 172.21.15.98:0/852855985 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f5070007c60 con 0x7f50800fe6f0 2023-07-14T19:30:14.783 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:14.774+0000 7f507d7fa700 1 -- 172.21.15.98:0/852855985 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5070018660 con 0x7f50800fe6f0 2023-07-14T19:30:14.783 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:14.774+0000 7f508636c700 1 -- 172.21.15.98:0/852855985 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f508019daa0 con 0x7f50800fe6f0 2023-07-14T19:30:14.783 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:14.774+0000 7f508636c700 1 -- 172.21.15.98:0/852855985 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f508019df60 con 0x7f50800fe6f0 2023-07-14T19:30:14.784 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:14.775+0000 7f508636c700 1 -- 172.21.15.98:0/852855985 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f508010b4b0 con 0x7f50800fe6f0 2023-07-14T19:30:14.784 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:14.776+0000 7f507d7fa700 1 -- 172.21.15.98:0/852855985 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 17) v1 ==== 135766+0+0 (secure 0 0 0) 0x7f507000f450 con 0x7f50800fe6f0 2023-07-14T19:30:14.784 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:14.776+0000 7f507d7fa700 1 --2- 172.21.15.98:0/852855985 >> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] conn(0x7f50680a12a0 0x7f50680a3750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:30:14.784 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:14.776+0000 7f507ffff700 1 --2- 172.21.15.98:0/852855985 >> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] conn(0x7f50680a12a0 0x7f50680a3750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:30:14.784 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:14.776+0000 7f507d7fa700 1 -- 172.21.15.98:0/852855985 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 8211+0+0 (secure 0 0 0) 0x7f50700ccaa0 con 0x7f50800fe6f0 2023-07-14T19:30:14.784 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:14.777+0000 7f507ffff700 1 --2- 172.21.15.98:0/852855985 >> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] conn(0x7f50680a12a0 0x7f50680a3750 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f5080068740 tx=0x7f507400a380 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:30:14.785 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:14.781+0000 7f507d7fa700 1 -- 172.21.15.98:0/852855985 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f5070096040 con 0x7f50800fe6f0 2023-07-14T19:30:15.013 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:15.013+0000 7f508636c700 1 -- 172.21.15.98:0/852855985 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f508010b6a0 con 0x7f50800fe6f0 2023-07-14T19:30:15.014 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:15.013+0000 7f507d7fa700 1 -- 172.21.15.98:0/852855985 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 22 v22) v1 ==== 94+0+10826 (secure 0 0 0) 0x7f5070099660 con 0x7f50800fe6f0 2023-07-14T19:30:15.014 INFO:teuthology.orchestra.run.smithi098.stdout: 2023-07-14T19:30:15.014 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":22,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14565,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6835/586280675","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6834","nonce":586280675},{"type":"v1","addr":"172.21.15.98:6835","nonce":586280675}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":4},{"gid":14577,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6837/487577257","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6836","nonce":487577257},{"type":"v1","addr":"172.21.15.98:6837","nonce":487577257}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":5},{"gid":14634,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6839/4085731209","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6838","nonce":4085731209},{"type":"v1","addr":"172.21.15.98:6839","nonce":4085731209}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":11},{"gid":14646,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6841/1679318259","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6840","nonce":1679318259},{"type":"v1","addr":"172.21.15.98:6841","nonce":1679318259}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":12},{"gid":24421,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.139:6833/3505477820","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6832","nonce":3505477820},{"type":"v1","addr":"172.21.15.139:6833","nonce":3505477820}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":24437,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.101:6833/3474460171","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.101:6832","nonce":3474460171},{"type":"v1","addr":"172.21.15.101:6833","nonce":3474460171}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":22,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2023-07-14T19:29:01.033156+0000","modified":"2023-07-14T19:29:06.291916+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24521,"mds_1":24493,"mds_2":24479},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24455":{"gid":24455,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.101:6835/1583264648","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.101:6834","nonce":1583264648},{"type":"v1","addr":"172.21.15.101:6835","nonce":1583264648}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24460":{"gid":24460,"name":"f","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.139:6835/2315281724","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6834","nonce":2315281724},{"type":"v1","addr":"172.21.15.139:6835","nonce":2315281724}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24469":{"gid":24469,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.139:6837/3597660552","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6836","nonce":3597660552},{"type":"v1","addr":"172.21.15.139:6837","nonce":3597660552}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24479":{"gid":24479,"name":"h","rank":2,"incarnation":21,"state":"up:active","state_seq":6,"addr":"172.21.15.101:6837/485741528","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.101:6836","nonce":485741528},{"type":"v1","addr":"172.21.15.101:6837","nonce":485741528}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24493":{"gid":24493,"name":"i","rank":1,"incarnation":19,"state":"up:active","state_seq":5,"addr":"172.21.15.139:6839/1258854143","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6838","nonce":1258854143},{"type":"v1","addr":"172.21.15.139:6839","nonce":1258854143}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24521":{"gid":24521,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.101:6839/48393381","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.101:6838","nonce":48393381},{"type":"v1","addr":"172.21.15.101:6839","nonce":48393381}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-07-14T19:30:15.016 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:15.015+0000 7f508636c700 1 -- 172.21.15.98:0/852855985 >> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] conn(0x7f50680a12a0 msgr2=0x7f50680a3750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:30:15.016 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:15.015+0000 7f508636c700 1 --2- 172.21.15.98:0/852855985 >> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] conn(0x7f50680a12a0 0x7f50680a3750 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f5080068740 tx=0x7f507400a380 comp rx=0 tx=0).stop 2023-07-14T19:30:15.016 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:15.015+0000 7f508636c700 1 -- 172.21.15.98:0/852855985 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f50800fe6f0 msgr2=0x7f5080195eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:30:15.016 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:15.015+0000 7f508636c700 1 --2- 172.21.15.98:0/852855985 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f50800fe6f0 0x7f5080195eb0 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7f507000c990 tx=0x7f507000cca0 comp rx=0 tx=0).stop 2023-07-14T19:30:15.017 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:15.015+0000 7f508636c700 1 -- 172.21.15.98:0/852855985 shutdown_connections 2023-07-14T19:30:15.017 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:15.015+0000 7f508636c700 1 --2- 172.21.15.98:0/852855985 >> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] conn(0x7f50680a12a0 0x7f50680a3750 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:30:15.017 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:15.015+0000 7f508636c700 1 --2- 172.21.15.98:0/852855985 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f50800fdd40 0x7f5080195970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:30:15.018 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:15.015+0000 7f508636c700 1 --2- 172.21.15.98:0/852855985 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f50801012d0 0x7f508019cf20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:30:15.018 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:15.015+0000 7f508636c700 1 --2- 172.21.15.98:0/852855985 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f50800fe6f0 0x7f5080195eb0 unknown :-1 s=CLOSED pgs=368 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:30:15.018 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:15.015+0000 7f508636c700 1 -- 172.21.15.98:0/852855985 >> 172.21.15.98:0/852855985 conn(0x7f50800f9990 msgr2=0x7f50800fba70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:30:15.018 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:15.016+0000 7f508636c700 1 -- 172.21.15.98:0/852855985 shutdown_connections 2023-07-14T19:30:15.019 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:15.016+0000 7f508636c700 1 -- 172.21.15.98:0/852855985 wait complete. 2023-07-14T19:30:15.019 INFO:teuthology.orchestra.run.smithi098.stderr:dumped fsmap epoch 22 2023-07-14T19:30:15.027 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:30:14 smithi098 ceph-mon[127486]: pgmap v291: 97 pgs: 97 active+clean; 634 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 5 op/s; 14 B/s, 0 objects/s recovering 2023-07-14T19:30:15.027 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:30:14 smithi098 ceph-mon[127486]: from='client.? 172.21.15.98:0/1226672077' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2023-07-14T19:30:15.027 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:30:14 smithi098 ceph-mon[127486]: from='client.? 172.21.15.98:0/576040503' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2023-07-14T19:30:15.028 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:30:14 smithi098 ceph-mon[127486]: from='client.? 172.21.15.98:0/842252631' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2023-07-14T19:30:15.029 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-07-14T19:30:15.152 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:15.150+0000 7f3663355700 1 -- 172.21.15.98:0/3516114435 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f365c100160 msgr2=0x7f365c100570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:30:15.152 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:15.150+0000 7f3663355700 1 --2- 172.21.15.98:0/3516114435 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f365c100160 0x7f365c100570 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7f364c009990 tx=0x7f364c009ca0 comp rx=0 tx=0).stop 2023-07-14T19:30:15.153 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:15.150+0000 7f3663355700 1 -- 172.21.15.98:0/3516114435 shutdown_connections 2023-07-14T19:30:15.153 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:15.150+0000 7f3663355700 1 --2- 172.21.15.98:0/3516114435 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f365c0ff760 0x7f365c107010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:30:15.153 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:15.150+0000 7f3663355700 1 --2- 172.21.15.98:0/3516114435 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f365c100160 0x7f365c100570 unknown :-1 s=CLOSED pgs=369 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:30:15.153 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:15.150+0000 7f3663355700 1 --2- 172.21.15.98:0/3516114435 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f365c0fedb0 0x7f365c0ff220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:30:15.153 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:15.150+0000 7f3663355700 1 -- 172.21.15.98:0/3516114435 >> 172.21.15.98:0/3516114435 conn(0x7f365c0f98d0 msgr2=0x7f365c0fbd20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:30:15.153 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:15.150+0000 7f3663355700 1 -- 172.21.15.98:0/3516114435 shutdown_connections 2023-07-14T19:30:15.153 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:15.150+0000 7f3663355700 1 -- 172.21.15.98:0/3516114435 wait complete. 2023-07-14T19:30:15.154 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:15.151+0000 7f3663355700 1 Processor -- start 2023-07-14T19:30:15.154 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:15.151+0000 7f3663355700 1 -- start start 2023-07-14T19:30:15.154 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:15.151+0000 7f3663355700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f365c0fedb0 0x7f365c102b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:30:15.154 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:15.151+0000 7f3663355700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f365c0ff760 0x7f365c1011a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:30:15.154 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:15.151+0000 7f3663355700 1 --2- >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f365c100160 0x7f365c1016e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:30:15.154 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:15.151+0000 7f3663355700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f365c101e80 con 0x7f365c0fedb0 2023-07-14T19:30:15.154 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:15.151+0000 7f3663355700 1 -- --> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f365c101ff0 con 0x7f365c100160 2023-07-14T19:30:15.154 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:15.151+0000 7f3663355700 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f365c19f8f0 con 0x7f365c0ff760 2023-07-14T19:30:15.155 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:15.151+0000 7f36610f1700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f365c0fedb0 0x7f365c102b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:30:15.155 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:15.151+0000 7f36610f1700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f365c0fedb0 0x7f365c102b60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:46826/0 (socket says 172.21.15.98:46826) 2023-07-14T19:30:15.155 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:15.151+0000 7f36610f1700 1 -- 172.21.15.98:0/999693056 learned_addr learned my addr 172.21.15.98:0/999693056 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-07-14T19:30:15.155 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:15.151+0000 7f36608f0700 1 --2- 172.21.15.98:0/999693056 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f365c0ff760 0x7f365c1011a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:30:15.155 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:15.151+0000 7f36618f2700 1 --2- 172.21.15.98:0/999693056 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f365c100160 0x7f365c1016e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:30:15.156 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:15.151+0000 7f36610f1700 1 -- 172.21.15.98:0/999693056 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f365c0ff760 msgr2=0x7f365c1011a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:30:15.156 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:15.151+0000 7f36610f1700 1 --2- 172.21.15.98:0/999693056 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f365c0ff760 0x7f365c1011a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:30:15.156 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:15.151+0000 7f36610f1700 1 -- 172.21.15.98:0/999693056 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f365c100160 msgr2=0x7f365c1016e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:30:15.156 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:15.151+0000 7f36610f1700 1 --2- 172.21.15.98:0/999693056 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f365c100160 0x7f365c1016e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:30:15.156 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:15.151+0000 7f36610f1700 1 -- 172.21.15.98:0/999693056 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f364c0095f0 con 0x7f365c0fedb0 2023-07-14T19:30:15.156 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:15.151+0000 7f36618f2700 1 --2- 172.21.15.98:0/999693056 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f365c100160 0x7f365c1016e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-07-14T19:30:15.156 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:15.151+0000 7f36610f1700 1 --2- 172.21.15.98:0/999693056 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f365c0fedb0 0x7f365c102b60 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x7f364c0095c0 tx=0x7f364c00f690 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:30:15.157 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:15.152+0000 7f36527fc700 1 -- 172.21.15.98:0/999693056 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f364c01d070 con 0x7f365c0fedb0 2023-07-14T19:30:15.157 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:15.152+0000 7f36527fc700 1 -- 172.21.15.98:0/999693056 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f364c022470 con 0x7f365c0fedb0 2023-07-14T19:30:15.157 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:15.152+0000 7f3663355700 1 -- 172.21.15.98:0/999693056 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f365c19fb70 con 0x7f365c0fedb0 2023-07-14T19:30:15.157 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:15.152+0000 7f36527fc700 1 -- 172.21.15.98:0/999693056 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f364c017650 con 0x7f365c0fedb0 2023-07-14T19:30:15.157 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:15.152+0000 7f3663355700 1 -- 172.21.15.98:0/999693056 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f365c19ff90 con 0x7f365c0fedb0 2023-07-14T19:30:15.158 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:15.153+0000 7f3663355700 1 -- 172.21.15.98:0/999693056 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f365c10a320 con 0x7f365c0fedb0 2023-07-14T19:30:15.158 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:15.154+0000 7f36527fc700 1 -- 172.21.15.98:0/999693056 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 17) v1 ==== 135766+0+0 (secure 0 0 0) 0x7f364c00fad0 con 0x7f365c0fedb0 2023-07-14T19:30:15.158 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:15.154+0000 7f36527fc700 1 --2- 172.21.15.98:0/999693056 >> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] conn(0x7f36480a1220 0x7f36480a36d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:30:15.158 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:15.154+0000 7f36527fc700 1 -- 172.21.15.98:0/999693056 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 8211+0+0 (secure 0 0 0) 0x7f364c0ce470 con 0x7f365c0fedb0 2023-07-14T19:30:15.159 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:15.154+0000 7f36608f0700 1 --2- 172.21.15.98:0/999693056 >> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] conn(0x7f36480a1220 0x7f36480a36d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:30:15.159 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:15.155+0000 7f36608f0700 1 --2- 172.21.15.98:0/999693056 >> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] conn(0x7f36480a1220 0x7f36480a36d0 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f3658005950 tx=0x7f365800b410 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:30:15.159 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:15.157+0000 7f36527fc700 1 -- 172.21.15.98:0/999693056 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f364c097c20 con 0x7f365c0fedb0 2023-07-14T19:30:15.191 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:30:14 smithi101 ceph-mon[128807]: pgmap v291: 97 pgs: 97 active+clean; 634 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 5 op/s; 14 B/s, 0 objects/s recovering 2023-07-14T19:30:15.191 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:30:14 smithi101 ceph-mon[128807]: from='client.? 172.21.15.98:0/1226672077' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2023-07-14T19:30:15.191 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:30:14 smithi101 ceph-mon[128807]: from='client.? 172.21.15.98:0/576040503' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2023-07-14T19:30:15.192 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:30:14 smithi101 ceph-mon[128807]: from='client.? 172.21.15.98:0/842252631' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2023-07-14T19:30:15.220 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:30:14 smithi139 ceph-mon[129293]: pgmap v291: 97 pgs: 97 active+clean; 634 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 5 op/s; 14 B/s, 0 objects/s recovering 2023-07-14T19:30:15.220 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:30:14 smithi139 ceph-mon[129293]: from='client.? 172.21.15.98:0/1226672077' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2023-07-14T19:30:15.220 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:30:14 smithi139 ceph-mon[129293]: from='client.? 172.21.15.98:0/576040503' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2023-07-14T19:30:15.221 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:30:14 smithi139 ceph-mon[129293]: from='client.? 172.21.15.98:0/842252631' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2023-07-14T19:30:15.348 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:15.348+0000 7f3663355700 1 -- 172.21.15.98:0/999693056 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f365c1a0540 con 0x7f365c0fedb0 2023-07-14T19:30:15.350 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:15.349+0000 7f36527fc700 1 -- 172.21.15.98:0/999693056 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v85) v1 ==== 74+0+22604 (secure 0 0 0) 0x7f364c027090 con 0x7f365c0fedb0 2023-07-14T19:30:15.350 INFO:teuthology.orchestra.run.smithi098.stdout: 2023-07-14T19:30:15.351 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":85,"fsid":"9adea648-227b-11ee-9b33-001a4aab830c","created":"2023-07-14T19:22:07.178854+0000","modified":"2023-07-14T19:30:03.314948+0000","last_up_change":"2023-07-14T19:28:19.646827+0000","last_in_change":"2023-07-14T19:28:02.018325+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-07-14T19:25:09.845827+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2023-07-14T19:28:58.952921+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"82","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2023-07-14T19:29:00.228272+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"83","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.690000057220459,"score_stable":1.690000057220459,"optimal_score":1,"raw_score_acting":1.690000057220459,"raw_score_stable":1.690000057220459,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"a3797e89-1754-41cf-9517-227e06ffdd72","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6802","nonce":120507228},{"type":"v1","addr":"172.21.15.98:6803","nonce":120507228}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6804","nonce":120507228},{"type":"v1","addr":"172.21.15.98:6805","nonce":120507228}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6808","nonce":120507228},{"type":"v1","addr":"172.21.15.98:6809","nonce":120507228}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6806","nonce":120507228},{"type":"v1","addr":"172.21.15.98:6807","nonce":120507228}]},"public_addr":"172.21.15.98:6803/120507228","cluster_addr":"172.21.15.98:6805/120507228","heartbeat_back_addr":"172.21.15.98:6809/120507228","heartbeat_front_addr":"172.21.15.98:6807/120507228","state":["exists","up"]},{"osd":1,"uuid":"2b001add-1aaa-4496-a403-7ef9725b80a0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.101:6800","nonce":2450584584},{"type":"v1","addr":"172.21.15.101:6801","nonce":2450584584}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.101:6802","nonce":2450584584},{"type":"v1","addr":"172.21.15.101:6803","nonce":2450584584}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.101:6806","nonce":2450584584},{"type":"v1","addr":"172.21.15.101:6807","nonce":2450584584}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.101:6804","nonce":2450584584},{"type":"v1","addr":"172.21.15.101:6805","nonce":2450584584}]},"public_addr":"172.21.15.101:6801/2450584584","cluster_addr":"172.21.15.101:6803/2450584584","heartbeat_back_addr":"172.21.15.101:6807/2450584584","heartbeat_front_addr":"172.21.15.101:6805/2450584584","state":["exists","up"]},{"osd":2,"uuid":"48e6d499-3ca9-424b-b0fb-3ba5f33b9fc9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":945038995},{"type":"v1","addr":"172.21.15.139:6801","nonce":945038995}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":945038995},{"type":"v1","addr":"172.21.15.139:6803","nonce":945038995}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":945038995},{"type":"v1","addr":"172.21.15.139:6807","nonce":945038995}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":945038995},{"type":"v1","addr":"172.21.15.139:6805","nonce":945038995}]},"public_addr":"172.21.15.139:6801/945038995","cluster_addr":"172.21.15.139:6803/945038995","heartbeat_back_addr":"172.21.15.139:6807/945038995","heartbeat_front_addr":"172.21.15.139:6805/945038995","state":["exists","up"]},{"osd":3,"uuid":"5530b265-31a5-447b-a1ea-17ebae9cfddf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6810","nonce":3095249665},{"type":"v1","addr":"172.21.15.98:6811","nonce":3095249665}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6812","nonce":3095249665},{"type":"v1","addr":"172.21.15.98:6813","nonce":3095249665}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6816","nonce":3095249665},{"type":"v1","addr":"172.21.15.98:6817","nonce":3095249665}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6814","nonce":3095249665},{"type":"v1","addr":"172.21.15.98:6815","nonce":3095249665}]},"public_addr":"172.21.15.98:6811/3095249665","cluster_addr":"172.21.15.98:6813/3095249665","heartbeat_back_addr":"172.21.15.98:6817/3095249665","heartbeat_front_addr":"172.21.15.98:6815/3095249665","state":["exists","up"]},{"osd":4,"uuid":"615d4642-a997-4f95-9e38-9fbdaa831400","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":32,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.101:6808","nonce":2844761406},{"type":"v1","addr":"172.21.15.101:6809","nonce":2844761406}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.101:6810","nonce":2844761406},{"type":"v1","addr":"172.21.15.101:6811","nonce":2844761406}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.101:6814","nonce":2844761406},{"type":"v1","addr":"172.21.15.101:6815","nonce":2844761406}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.101:6812","nonce":2844761406},{"type":"v1","addr":"172.21.15.101:6813","nonce":2844761406}]},"public_addr":"172.21.15.101:6809/2844761406","cluster_addr":"172.21.15.101:6811/2844761406","heartbeat_back_addr":"172.21.15.101:6815/2844761406","heartbeat_front_addr":"172.21.15.101:6813/2844761406","state":["exists","up"]},{"osd":5,"uuid":"415de5fa-607a-45a8-b732-28daf3be6dd3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":37,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":3532263042},{"type":"v1","addr":"172.21.15.139:6809","nonce":3532263042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6810","nonce":3532263042},{"type":"v1","addr":"172.21.15.139:6811","nonce":3532263042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6814","nonce":3532263042},{"type":"v1","addr":"172.21.15.139:6815","nonce":3532263042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":3532263042},{"type":"v1","addr":"172.21.15.139:6813","nonce":3532263042}]},"public_addr":"172.21.15.139:6809/3532263042","cluster_addr":"172.21.15.139:6811/3532263042","heartbeat_back_addr":"172.21.15.139:6815/3532263042","heartbeat_front_addr":"172.21.15.139:6813/3532263042","state":["exists","up"]},{"osd":6,"uuid":"d7f9d4ba-c309-4b75-a836-5a36384b528e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":45,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6818","nonce":658948743},{"type":"v1","addr":"172.21.15.98:6819","nonce":658948743}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6820","nonce":658948743},{"type":"v1","addr":"172.21.15.98:6821","nonce":658948743}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6824","nonce":658948743},{"type":"v1","addr":"172.21.15.98:6825","nonce":658948743}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6822","nonce":658948743},{"type":"v1","addr":"172.21.15.98:6823","nonce":658948743}]},"public_addr":"172.21.15.98:6819/658948743","cluster_addr":"172.21.15.98:6821/658948743","heartbeat_back_addr":"172.21.15.98:6825/658948743","heartbeat_front_addr":"172.21.15.98:6823/658948743","state":["exists","up"]},{"osd":7,"uuid":"eedb94d9-c5cd-4a93-8a8c-2648804b7195","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":50,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.101:6816","nonce":2881214066},{"type":"v1","addr":"172.21.15.101:6817","nonce":2881214066}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.101:6818","nonce":2881214066},{"type":"v1","addr":"172.21.15.101:6819","nonce":2881214066}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.101:6822","nonce":2881214066},{"type":"v1","addr":"172.21.15.101:6823","nonce":2881214066}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.101:6820","nonce":2881214066},{"type":"v1","addr":"172.21.15.101:6821","nonce":2881214066}]},"public_addr":"172.21.15.101:6817/2881214066","cluster_addr":"172.21.15.101:6819/2881214066","heartbeat_back_addr":"172.21.15.101:6823/2881214066","heartbeat_front_addr":"172.21.15.101:6821/2881214066","state":["exists","up"]},{"osd":8,"uuid":"6bfd77c2-5770-466e-aa6f-84b50c74ffb6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":57,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":3669514524},{"type":"v1","addr":"172.21.15.139:6817","nonce":3669514524}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":3669514524},{"type":"v1","addr":"172.21.15.139:6819","nonce":3669514524}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":3669514524},{"type":"v1","addr":"172.21.15.139:6823","nonce":3669514524}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6820","nonce":3669514524},{"type":"v1","addr":"172.21.15.139:6821","nonce":3669514524}]},"public_addr":"172.21.15.139:6817/3669514524","cluster_addr":"172.21.15.139:6819/3669514524","heartbeat_back_addr":"172.21.15.139:6823/3669514524","heartbeat_front_addr":"172.21.15.139:6821/3669514524","state":["exists","up"]},{"osd":9,"uuid":"19c5b54a-2242-4b23-b721-63be9559653d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":65,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6826","nonce":2893994080},{"type":"v1","addr":"172.21.15.98:6827","nonce":2893994080}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6828","nonce":2893994080},{"type":"v1","addr":"172.21.15.98:6829","nonce":2893994080}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6832","nonce":2893994080},{"type":"v1","addr":"172.21.15.98:6833","nonce":2893994080}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6830","nonce":2893994080},{"type":"v1","addr":"172.21.15.98:6831","nonce":2893994080}]},"public_addr":"172.21.15.98:6827/2893994080","cluster_addr":"172.21.15.98:6829/2893994080","heartbeat_back_addr":"172.21.15.98:6833/2893994080","heartbeat_front_addr":"172.21.15.98:6831/2893994080","state":["exists","up"]},{"osd":10,"uuid":"326f0209-ba9d-4b59-97cc-d61a758a8f29","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":70,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.101:6824","nonce":3545244225},{"type":"v1","addr":"172.21.15.101:6825","nonce":3545244225}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.101:6826","nonce":3545244225},{"type":"v1","addr":"172.21.15.101:6827","nonce":3545244225}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.101:6830","nonce":3545244225},{"type":"v1","addr":"172.21.15.101:6831","nonce":3545244225}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.101:6828","nonce":3545244225},{"type":"v1","addr":"172.21.15.101:6829","nonce":3545244225}]},"public_addr":"172.21.15.101:6825/3545244225","cluster_addr":"172.21.15.101:6827/3545244225","heartbeat_back_addr":"172.21.15.101:6831/3545244225","heartbeat_front_addr":"172.21.15.101:6829/3545244225","state":["exists","up"]},{"osd":11,"uuid":"c61c0320-1057-4f17-9bf1-7ad306ed7ed3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":78,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":2630023454},{"type":"v1","addr":"172.21.15.139:6825","nonce":2630023454}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6826","nonce":2630023454},{"type":"v1","addr":"172.21.15.139:6827","nonce":2630023454}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":2630023454},{"type":"v1","addr":"172.21.15.139:6831","nonce":2630023454}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":2630023454},{"type":"v1","addr":"172.21.15.139:6829","nonce":2630023454}]},"public_addr":"172.21.15.139:6825/2630023454","cluster_addr":"172.21.15.139:6827/2630023454","heartbeat_back_addr":"172.21.15.139:6831/2630023454","heartbeat_front_addr":"172.21.15.139:6829/2630023454","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T19:24:23.620028+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T19:24:45.779846+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T19:25:06.877949+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T19:25:27.994199+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T19:25:49.461139+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T19:26:10.409282+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T19:26:32.116390+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T19:26:53.536474+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T19:27:14.384839+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T19:27:35.635921+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T19:27:56.858744+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T19:28:18.342519+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.4","mappings":[{"from":2,"to":11}]},{"pgid":"2.f","mappings":[{"from":8,"to":11}]},{"pgid":"3.0","mappings":[{"from":4,"to":8}]},{"pgid":"3.1","mappings":[{"from":4,"to":9}]},{"pgid":"3.c","mappings":[{"from":3,"to":9}]},{"pgid":"3.17","mappings":[{"from":4,"to":6}]},{"pgid":"3.1e","mappings":[{"from":3,"to":9}]},{"pgid":"3.28","mappings":[{"from":5,"to":7}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.98:0/2164590253":"2023-07-15T19:23:01.520302+0000","172.21.15.98:0/2094734896":"2023-07-15T19:23:01.520302+0000","172.21.15.98:0/784745009":"2023-07-15T19:22:39.481502+0000","172.21.15.98:0/3657190818":"2023-07-15T19:22:39.481502+0000","172.21.15.98:6800/1732769087":"2023-07-15T19:23:01.520302+0000","172.21.15.98:0/3393542419":"2023-07-15T19:22:39.481502+0000","172.21.15.98:6800/749234665":"2023-07-15T19:22:39.481502+0000","172.21.15.98:6801/1732769087":"2023-07-15T19:23:01.520302+0000","172.21.15.98:6801/749234665":"2023-07-15T19:22:39.481502+0000","172.21.15.98:0/187822873":"2023-07-15T19:23:01.520302+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-07-14T19:30:15.352 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:15.351+0000 7f3663355700 1 -- 172.21.15.98:0/999693056 >> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] conn(0x7f36480a1220 msgr2=0x7f36480a36d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:30:15.352 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:15.351+0000 7f3663355700 1 --2- 172.21.15.98:0/999693056 >> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] conn(0x7f36480a1220 0x7f36480a36d0 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f3658005950 tx=0x7f365800b410 comp rx=0 tx=0).stop 2023-07-14T19:30:15.352 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:15.351+0000 7f3663355700 1 -- 172.21.15.98:0/999693056 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f365c0fedb0 msgr2=0x7f365c102b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:30:15.353 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:15.351+0000 7f3663355700 1 --2- 172.21.15.98:0/999693056 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f365c0fedb0 0x7f365c102b60 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x7f364c0095c0 tx=0x7f364c00f690 comp rx=0 tx=0).stop 2023-07-14T19:30:15.353 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:15.352+0000 7f3663355700 1 -- 172.21.15.98:0/999693056 shutdown_connections 2023-07-14T19:30:15.353 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:15.352+0000 7f3663355700 1 --2- 172.21.15.98:0/999693056 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f365c100160 0x7f365c1016e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:30:15.353 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:15.352+0000 7f3663355700 1 --2- 172.21.15.98:0/999693056 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f365c0fedb0 0x7f365c102b60 unknown :-1 s=CLOSED pgs=370 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:30:15.353 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:15.352+0000 7f3663355700 1 --2- 172.21.15.98:0/999693056 >> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] conn(0x7f36480a1220 0x7f36480a36d0 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:30:15.353 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:15.352+0000 7f3663355700 1 --2- 172.21.15.98:0/999693056 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f365c0ff760 0x7f365c1011a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:30:15.353 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:15.352+0000 7f3663355700 1 -- 172.21.15.98:0/999693056 >> 172.21.15.98:0/999693056 conn(0x7f365c0f98d0 msgr2=0x7f365c0fdd20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:30:15.353 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:15.352+0000 7f3663355700 1 -- 172.21.15.98:0/999693056 shutdown_connections 2023-07-14T19:30:15.354 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:15.352+0000 7f3663355700 1 -- 172.21.15.98:0/999693056 wait complete. 2023-07-14T19:30:15.365 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 22, 'flags': 50, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': True, 'refuse_client_session': False, 'balance_automate': False}, 'ever_allowed_features': 32, 'explicitly_allowed_features': 32, 'created': '2023-07-14T19:29:01.033156+0000', 'modified': '2023-07-14T19:29:06.291916+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 3, 'in': [0, 1, 2], 'up': {'mds_0': 24521, 'mds_1': 24493, 'mds_2': 24479}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_24455': {'gid': 24455, 'name': 'e', 'rank': 2, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.101:6835/1583264648', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.101:6834', 'nonce': 1583264648}, {'type': 'v1', 'addr': '172.21.15.101:6835', 'nonce': 1583264648}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138322906710015, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}}, 'gid_24460': {'gid': 24460, 'name': 'f', 'rank': 1, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.139:6835/2315281724', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.139:6834', 'nonce': 2315281724}, {'type': 'v1', 'addr': '172.21.15.139:6835', 'nonce': 2315281724}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138322906710015, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}}, 'gid_24469': {'gid': 24469, 'name': 'c', 'rank': 0, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.139:6837/3597660552', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.139:6836', 'nonce': 3597660552}, {'type': 'v1', 'addr': '172.21.15.139:6837', 'nonce': 3597660552}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138322906710015, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}}, 'gid_24479': {'gid': 24479, 'name': 'h', 'rank': 2, 'incarnation': 21, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.101:6837/485741528', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.101:6836', 'nonce': 485741528}, {'type': 'v1', 'addr': '172.21.15.101:6837', 'nonce': 485741528}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138322906710015, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}}, 'gid_24493': {'gid': 24493, 'name': 'i', 'rank': 1, 'incarnation': 19, 'state': 'up:active', 'state_seq': 5, 'addr': '172.21.15.139:6839/1258854143', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.139:6838', 'nonce': 1258854143}, {'type': 'v1', 'addr': '172.21.15.139:6839', 'nonce': 1258854143}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138322906710015, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}}, 'gid_24521': {'gid': 24521, 'name': 'b', 'rank': 0, 'incarnation': 14, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.101:6839/48393381', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.101:6838', 'nonce': 48393381}, {'type': 'v1', 'addr': '172.21.15.101:6839', 'nonce': 48393381}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138322906710015, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2023-07-14T19:30:15.365 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 3/3 2023-07-14T19:30:15.365 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.h status 2023-07-14T19:30:15.487 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:15.487+0000 7f6035f65700 1 -- 172.21.15.98:0/2890277111 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f60300ffd20 msgr2=0x7f6030100170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:30:15.488 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:15.487+0000 7f6035f65700 1 --2- 172.21.15.98:0/2890277111 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f60300ffd20 0x7f6030100170 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 crypto rx=0x7f6020009990 tx=0x7f6020009ca0 comp rx=0 tx=0).stop 2023-07-14T19:30:15.488 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:15.487+0000 7f6035f65700 1 -- 172.21.15.98:0/2890277111 shutdown_connections 2023-07-14T19:30:15.488 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:15.487+0000 7f6035f65700 1 --2- 172.21.15.98:0/2890277111 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f60301006b0 0x7f6030106f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:30:15.488 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:15.487+0000 7f6035f65700 1 --2- 172.21.15.98:0/2890277111 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f60300feb30 0x7f60300fef40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:30:15.488 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:15.487+0000 7f6035f65700 1 --2- 172.21.15.98:0/2890277111 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f60300ffd20 0x7f6030100170 unknown :-1 s=CLOSED pgs=371 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:30:15.489 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:15.487+0000 7f6035f65700 1 -- 172.21.15.98:0/2890277111 >> 172.21.15.98:0/2890277111 conn(0x7f60300fa150 msgr2=0x7f60300fc580 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:30:15.489 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:15.487+0000 7f6035f65700 1 -- 172.21.15.98:0/2890277111 shutdown_connections 2023-07-14T19:30:15.489 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:15.487+0000 7f6035f65700 1 -- 172.21.15.98:0/2890277111 wait complete. 2023-07-14T19:30:15.489 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:15.487+0000 7f6035f65700 1 Processor -- start 2023-07-14T19:30:15.491 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:15.488+0000 7f6035f65700 1 -- start start 2023-07-14T19:30:15.491 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:15.488+0000 7f6035f65700 1 --2- >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f60300feb30 0x7f60301947c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:30:15.491 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:15.488+0000 7f6035f65700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f60300ffd20 0x7f6030194d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:30:15.491 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:15.488+0000 7f6035f65700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f60301006b0 0x7f603019bd70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:30:15.491 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:15.488+0000 7f6035f65700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f603019c340 con 0x7f60301006b0 2023-07-14T19:30:15.491 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:15.488+0000 7f6035f65700 1 -- --> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f603019c480 con 0x7f60300feb30 2023-07-14T19:30:15.492 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:15.488+0000 7f6035f65700 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f603019c610 con 0x7f60300ffd20 2023-07-14T19:30:15.492 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:15.488+0000 7f602ffff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f60301006b0 0x7f603019bd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:30:15.492 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:15.488+0000 7f602ffff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f60301006b0 0x7f603019bd70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:46844/0 (socket says 172.21.15.98:46844) 2023-07-14T19:30:15.492 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:15.488+0000 7f602ffff700 1 -- 172.21.15.98:0/1981388592 learned_addr learned my addr 172.21.15.98:0/1981388592 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-07-14T19:30:15.492 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:15.488+0000 7f602f7fe700 1 --2- 172.21.15.98:0/1981388592 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f60300feb30 0x7f60301947c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:30:15.492 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:15.488+0000 7f602effd700 1 --2- 172.21.15.98:0/1981388592 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f60300ffd20 0x7f6030194d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:30:15.492 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:15.488+0000 7f602ffff700 1 -- 172.21.15.98:0/1981388592 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f60300ffd20 msgr2=0x7f6030194d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:30:15.493 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:15.488+0000 7f602ffff700 1 --2- 172.21.15.98:0/1981388592 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f60300ffd20 0x7f6030194d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:30:15.493 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:15.488+0000 7f602ffff700 1 -- 172.21.15.98:0/1981388592 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f60300feb30 msgr2=0x7f60301947c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:30:15.493 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:15.488+0000 7f602ffff700 1 --2- 172.21.15.98:0/1981388592 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f60300feb30 0x7f60301947c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:30:15.493 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:15.488+0000 7f602ffff700 1 -- 172.21.15.98:0/1981388592 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f60200095f0 con 0x7f60301006b0 2023-07-14T19:30:15.493 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:15.488+0000 7f602ffff700 1 --2- 172.21.15.98:0/1981388592 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f60301006b0 0x7f603019bd70 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 crypto rx=0x7f6028007ee0 tx=0x7f602800d3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:30:15.493 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:15.488+0000 7f602cff9700 1 -- 172.21.15.98:0/1981388592 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6028017070 con 0x7f60301006b0 2023-07-14T19:30:15.493 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:15.488+0000 7f602cff9700 1 -- 172.21.15.98:0/1981388592 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f602800f040 con 0x7f60301006b0 2023-07-14T19:30:15.494 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:15.488+0000 7f6035f65700 1 -- 172.21.15.98:0/1981388592 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f603019c8f0 con 0x7f60301006b0 2023-07-14T19:30:15.494 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:15.488+0000 7f602cff9700 1 -- 172.21.15.98:0/1981388592 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6028005740 con 0x7f60301006b0 2023-07-14T19:30:15.494 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:15.489+0000 7f6035f65700 1 -- 172.21.15.98:0/1981388592 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f603019cdb0 con 0x7f60301006b0 2023-07-14T19:30:15.494 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:15.490+0000 7f602cff9700 1 -- 172.21.15.98:0/1981388592 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 17) v1 ==== 135766+0+0 (secure 0 0 0) 0x7f602800dc60 con 0x7f60301006b0 2023-07-14T19:30:15.499 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:15.491+0000 7f602cff9700 1 --2- 172.21.15.98:0/1981388592 >> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] conn(0x7f60180a11e0 0x7f60180a3690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:30:15.499 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:15.491+0000 7f602f7fe700 1 --2- 172.21.15.98:0/1981388592 >> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] conn(0x7f60180a11e0 0x7f60180a3690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:30:15.499 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:15.491+0000 7f602cff9700 1 -- 172.21.15.98:0/1981388592 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 8211+0+0 (secure 0 0 0) 0x7f60280cc2d0 con 0x7f60301006b0 2023-07-14T19:30:15.499 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:15.491+0000 7f602f7fe700 1 --2- 172.21.15.98:0/1981388592 >> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] conn(0x7f60180a11e0 0x7f60180a3690 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f60300ffb80 tx=0x7f602400a3b0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:30:15.499 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:15.492+0000 7f60420fdb80 1 Processor -- start 2023-07-14T19:30:15.499 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:15.492+0000 7f60420fdb80 1 -- start start 2023-07-14T19:30:15.499 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:15.492+0000 7f60420fdb80 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55796896a0d0 0x55796896a4e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:30:15.500 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:15.492+0000 7f60420fdb80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55796896aa20 0x55796896b0c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:30:15.500 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:15.492+0000 7f60420fdb80 1 --2- >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x55796896b650 0x55796896bac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:30:15.500 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:15.492+0000 7f60420fdb80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x55796886cc60 con 0x55796896aa20 2023-07-14T19:30:15.500 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:15.492+0000 7f60420fdb80 1 -- --> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] -- mon_getmap magic: 0 v1 -- 0x5579688bf9f0 con 0x55796896b650 2023-07-14T19:30:15.500 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:15.492+0000 7f60420fdb80 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 v1 -- 0x5579688b8e80 con 0x55796896a0d0 2023-07-14T19:30:15.500 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:15.492+0000 7f602effd700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55796896aa20 0x55796896b0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:30:15.500 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:15.492+0000 7f602effd700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55796896aa20 0x55796896b0c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:46850/0 (socket says 172.21.15.98:46850) 2023-07-14T19:30:15.501 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:15.492+0000 7f602effd700 1 -- 172.21.15.98:0/4240659066 learned_addr learned my addr 172.21.15.98:0/4240659066 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-07-14T19:30:15.501 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:15.492+0000 7f602f7fe700 1 --2- 172.21.15.98:0/4240659066 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55796896a0d0 0x55796896a4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:30:15.501 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:15.492+0000 7f602effd700 1 -- 172.21.15.98:0/4240659066 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55796896a0d0 msgr2=0x55796896a4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-14T19:30:15.501 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:15.492+0000 7f602effd700 1 --2- 172.21.15.98:0/4240659066 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55796896a0d0 0x55796896a4e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:30:15.501 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:15.492+0000 7f602ffff700 1 --2- 172.21.15.98:0/4240659066 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x55796896b650 0x55796896bac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:30:15.501 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:15.492+0000 7f602effd700 1 -- 172.21.15.98:0/4240659066 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x55796896b650 msgr2=0x55796896bac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-14T19:30:15.501 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:15.492+0000 7f602effd700 1 --2- 172.21.15.98:0/4240659066 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x55796896b650 0x55796896bac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:30:15.501 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:15.493+0000 7f602effd700 1 -- 172.21.15.98:0/4240659066 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5579686e6d90 con 0x55796896aa20 2023-07-14T19:30:15.502 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:15.493+0000 7f602f7fe700 1 --2- 172.21.15.98:0/4240659066 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55796896a0d0 0x55796896a4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-07-14T19:30:15.502 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:15.493+0000 7f602effd700 1 --2- 172.21.15.98:0/4240659066 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55796896aa20 0x55796896b0c0 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7f6020006010 tx=0x7f6020009f60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=47c2479a28cdd0b8 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:30:15.502 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:15.493+0000 7f6015ffb700 1 -- 172.21.15.98:0/4240659066 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6020005470 con 0x55796896aa20 2023-07-14T19:30:15.502 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:15.493+0000 7f6015ffb700 1 -- 172.21.15.98:0/4240659066 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f602001e040 con 0x55796896aa20 2023-07-14T19:30:15.502 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:15.493+0000 7f6015ffb700 1 -- 172.21.15.98:0/4240659066 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6020004900 con 0x55796896aa20 2023-07-14T19:30:15.502 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:15.493+0000 7f60420fdb80 1 -- 172.21.15.98:0/4240659066 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55796896aa20 msgr2=0x55796896b0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:30:15.502 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:15.493+0000 7f60420fdb80 1 --2- 172.21.15.98:0/4240659066 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55796896aa20 0x55796896b0c0 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7f6020006010 tx=0x7f6020009f60 comp rx=0 tx=0).stop 2023-07-14T19:30:15.503 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:15.493+0000 7f60420fdb80 1 -- 172.21.15.98:0/4240659066 shutdown_connections 2023-07-14T19:30:15.503 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:15.493+0000 7f60420fdb80 1 --2- 172.21.15.98:0/4240659066 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55796896a0d0 0x55796896a4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:30:15.503 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:15.493+0000 7f60420fdb80 1 --2- 172.21.15.98:0/4240659066 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x55796896b650 0x55796896bac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:30:15.503 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:15.493+0000 7f60420fdb80 1 --2- 172.21.15.98:0/4240659066 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55796896aa20 0x55796896b0c0 unknown :-1 s=CLOSED pgs=373 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:30:15.503 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:15.493+0000 7f60420fdb80 1 -- 172.21.15.98:0/4240659066 >> 172.21.15.98:0/4240659066 conn(0x55796895d630 msgr2=0x557968962980 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:30:15.503 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:15.494+0000 7f60420fdb80 1 -- 172.21.15.98:0/4240659066 shutdown_connections 2023-07-14T19:30:15.503 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:15.494+0000 7f60420fdb80 1 -- 172.21.15.98:0/4240659066 wait complete. 2023-07-14T19:30:15.504 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:15.494+0000 7f60420fdb80 1 -- start start 2023-07-14T19:30:15.504 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:15.494+0000 7f60420fdb80 1 Processor -- start 2023-07-14T19:30:15.504 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:15.494+0000 7f60420fdb80 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55796896a0d0 0x557968973ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:30:15.504 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:15.494+0000 7f60420fdb80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55796896aa20 0x557968974010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:30:15.505 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:15.494+0000 7f60420fdb80 1 --2- >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x55796896b650 0x55796897b050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:30:15.505 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:15.494+0000 7f60420fdb80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x557968858a70 con 0x55796896aa20 2023-07-14T19:30:15.505 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:15.495+0000 7f60420fdb80 1 -- --> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] -- mon_getmap magic: 0 v1 -- 0x557968858770 con 0x55796896b650 2023-07-14T19:30:15.505 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:15.495+0000 7f60420fdb80 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 v1 -- 0x55796873c8d0 con 0x55796896a0d0 2023-07-14T19:30:15.505 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:15.495+0000 7f60420fdb80 10 client.0 _ll_register_callbacks cb 0x557968856710 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-07-14T19:30:15.505 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:15.495+0000 7f602effd700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55796896aa20 0x557968974010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:30:15.506 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:15.495+0000 7f602effd700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55796896aa20 0x557968974010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:46856/0 (socket says 172.21.15.98:46856) 2023-07-14T19:30:15.506 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:15.495+0000 7f602effd700 1 -- 172.21.15.98:0/13922002 learned_addr learned my addr 172.21.15.98:0/13922002 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-07-14T19:30:15.506 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:15.495+0000 7f602f7fe700 1 --2- 172.21.15.98:0/13922002 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55796896a0d0 0x557968973ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:30:15.506 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:15.495+0000 7f602ffff700 1 --2- 172.21.15.98:0/13922002 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x55796896b650 0x55796897b050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:30:15.506 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:15.495+0000 7f602effd700 1 -- 172.21.15.98:0/13922002 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55796896a0d0 msgr2=0x557968973ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-14T19:30:15.506 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:15.495+0000 7f602effd700 1 --2- 172.21.15.98:0/13922002 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55796896a0d0 0x557968973ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:30:15.507 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:15.495+0000 7f602effd700 1 -- 172.21.15.98:0/13922002 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x55796896b650 msgr2=0x55796897b050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-14T19:30:15.507 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:15.495+0000 7f602effd700 1 --2- 172.21.15.98:0/13922002 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x55796896b650 0x55796897b050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:30:15.507 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:15.495+0000 7f602effd700 1 -- 172.21.15.98:0/13922002 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f60301078a0 con 0x55796896aa20 2023-07-14T19:30:15.507 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:15.495+0000 7f602f7fe700 1 --2- 172.21.15.98:0/13922002 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55796896a0d0 0x557968973ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-07-14T19:30:15.507 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:15.495+0000 7f602effd700 1 --2- 172.21.15.98:0/13922002 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55796896aa20 0x557968974010 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7f602000b5c0 tx=0x7f602000bad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5908bd413562ea4 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:30:15.507 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:15.495+0000 7f600effd700 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2023-07-14T19:30:15.507 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:15.495+0000 7f600effd700 1 -- 172.21.15.98:0/13922002 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f60200056a0 con 0x55796896aa20 2023-07-14T19:30:15.508 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:15.495+0000 7f60420fdb80 1 -- 172.21.15.98:0/13922002 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x5579688124a0 con 0x55796896aa20 2023-07-14T19:30:15.508 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:15.495+0000 7f600effd700 1 -- 172.21.15.98:0/13922002 <== mon.0 v2:172.21.15.98:3300/0 2 ==== osd_map(85..85 src has 1..85) v4 ==== 8211+0+0 (secure 0 0 0) 0x7f6020004a90 con 0x55796896aa20 2023-07-14T19:30:15.508 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:15.495+0000 7f600effd700 1 -- 172.21.15.98:0/13922002 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=22) v2 ==== 24+0+0 (secure 0 0 0) 0x7f6020004c50 con 0x55796896aa20 2023-07-14T19:30:15.508 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:15.495+0000 7f60420fdb80 10 client.15222 fetch_fsmap learned FSMap version 22 2023-07-14T19:30:15.508 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:15.495+0000 7f60420fdb80 1 -- 172.21.15.98:0/13922002 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=22,monmap=4+}) v3 -- 0x5579686cfc10 con 0x55796896aa20 2023-07-14T19:30:15.508 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:15.495+0000 7f600effd700 1 -- 172.21.15.98:0/13922002 <== mon.0 v2:172.21.15.98:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x5579688124a0 con 0x55796896aa20 2023-07-14T19:30:15.509 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:15.496+0000 7f600effd700 1 -- 172.21.15.98:0/13922002 <== mon.0 v2:172.21.15.98:3300/0 5 ==== fsmap(e 22) v1 ==== 7462+0+0 (secure 0 0 0) 0x7f602000f460 con 0x55796896aa20 2023-07-14T19:30:15.509 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:15.496+0000 7f60420fdb80 10 client.15222 fetch_fsmap finished waiting for FSMap version 22 2023-07-14T19:30:15.509 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:15.496+0000 7f60420fdb80 10 client.15222 resolve_mds: resolved name 'h' to daemon mds.h 2023-07-14T19:30:15.509 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:15.496+0000 7f60420fdb80 20 client.15222 populate_metadata read hostname 'smithi098' 2023-07-14T19:30:15.509 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:15.496+0000 7f60420fdb80 1 --2- 172.21.15.98:0/13922002 >> [v2:172.21.15.101:6836/485741528,v1:172.21.15.101:6837/485741528] conn(0x55796897e840 0x55796897df80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:30:15.510 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:15.496+0000 7f60420fdb80 4 client.15222 mds_command: new command op to 24479 tid=0[{"prefix": "get_command_descriptions"}] 2023-07-14T19:30:15.510 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:15.496+0000 7f60420fdb80 1 -- 172.21.15.98:0/13922002 --> [v2:172.21.15.101:6836/485741528,v1:172.21.15.101:6837/485741528] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x557968936d50 con 0x55796897e840 2023-07-14T19:30:15.510 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:15.496+0000 7f602f7fe700 1 --2- 172.21.15.98:0/13922002 >> [v2:172.21.15.101:6836/485741528,v1:172.21.15.101:6837/485741528] conn(0x55796897e840 0x55796897df80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:30:15.510 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:15.497+0000 7f600effd700 10 client.15222 ms_handle_connect on v2:172.21.15.101:6836/485741528 2023-07-14T19:30:15.510 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:15.497+0000 7f602f7fe700 1 --2- 172.21.15.98:0/13922002 >> [v2:172.21.15.101:6836/485741528,v1:172.21.15.101:6837/485741528] conn(0x55796897e840 0x55796897df80 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=34f516c4397fdbec server_cookie=434b68b60a62b23c in_seq=0 out_seq=0 2023-07-14T19:30:15.511 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:15.497+0000 7f600effd700 1 -- 172.21.15.98:0/13922002 <== mds.2 v2:172.21.15.101:6836/485741528 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18017 (crc 0 0 0) 0x557968936d50 con 0x55796897e840 2023-07-14T19:30:15.511 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:15.497+0000 7f600effd700 10 client.15222 handle_command_reply: tid=0 2023-07-14T19:30:15.511 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:15.497+0000 7f60420fdb80 1 client.15222 shutdown 2023-07-14T19:30:15.511 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:15.497+0000 7f60420fdb80 1 -- 172.21.15.98:0/13922002 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55796896aa20 msgr2=0x557968974010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:30:15.512 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:15.497+0000 7f60420fdb80 1 --2- 172.21.15.98:0/13922002 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55796896aa20 0x557968974010 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7f602000b5c0 tx=0x7f602000bad0 comp rx=0 tx=0).stop 2023-07-14T19:30:15.512 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:15.497+0000 7f60420fdb80 1 -- 172.21.15.98:0/13922002 shutdown_connections 2023-07-14T19:30:15.512 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:15.497+0000 7f60420fdb80 1 --2- 172.21.15.98:0/13922002 >> [v2:172.21.15.101:6836/485741528,v1:172.21.15.101:6837/485741528] conn(0x55796897e840 0x55796897df80 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:30:15.512 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:15.497+0000 7f60420fdb80 1 --2- 172.21.15.98:0/13922002 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55796896a0d0 0x557968973ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:30:15.513 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:15.497+0000 7f60420fdb80 1 --2- 172.21.15.98:0/13922002 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x55796896b650 0x55796897b050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:30:15.513 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:15.497+0000 7f600effd700 0 client.15222 ms_handle_reset on v2:172.21.15.101:6836/485741528 2023-07-14T19:30:15.513 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:15.497+0000 7f60420fdb80 1 --2- 172.21.15.98:0/13922002 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55796896aa20 0x557968974010 unknown :-1 s=CLOSED pgs=374 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:30:15.513 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:15.497+0000 7f60420fdb80 1 -- 172.21.15.98:0/13922002 >> 172.21.15.98:0/13922002 conn(0x55796895d630 msgr2=0x557968961600 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:30:15.513 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:15.497+0000 7f60420fdb80 1 -- 172.21.15.98:0/13922002 shutdown_connections 2023-07-14T19:30:15.514 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:15.497+0000 7f60420fdb80 1 -- 172.21.15.98:0/13922002 wait complete. 2023-07-14T19:30:15.514 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:15.498+0000 7f60420fdb80 20 client.15222 trim_cache size 0 max 16384 2023-07-14T19:30:15.521 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:15.515+0000 7f60420fdb80 1 Processor -- start 2023-07-14T19:30:15.521 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:15.515+0000 7f60420fdb80 1 -- start start 2023-07-14T19:30:15.521 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:15.515+0000 7f60420fdb80 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55796896a0d0 0x55796895eb00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:30:15.522 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:15.515+0000 7f60420fdb80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55796896aa20 0x55796897b010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:30:15.522 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:15.515+0000 7f60420fdb80 1 --2- >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x55796896b650 0x5579689727e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:30:15.522 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:15.515+0000 7f60420fdb80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x5579688124a0 con 0x55796896aa20 2023-07-14T19:30:15.522 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:15.515+0000 7f60420fdb80 1 -- --> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] -- mon_getmap magic: 0 v1 -- 0x557968974cc0 con 0x55796896b650 2023-07-14T19:30:15.523 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:15.515+0000 7f60420fdb80 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 v1 -- 0x557968862470 con 0x55796896a0d0 2023-07-14T19:30:15.523 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:15.515+0000 7f602effd700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55796896aa20 0x55796897b010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:30:15.523 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:15.515+0000 7f602effd700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55796896aa20 0x55796897b010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:46864/0 (socket says 172.21.15.98:46864) 2023-07-14T19:30:15.523 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:15.515+0000 7f602effd700 1 -- 172.21.15.98:0/61811646 learned_addr learned my addr 172.21.15.98:0/61811646 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-07-14T19:30:15.523 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:15.515+0000 7f602effd700 1 -- 172.21.15.98:0/61811646 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55796896a0d0 msgr2=0x55796895eb00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-14T19:30:15.524 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:15.515+0000 7f602effd700 1 --2- 172.21.15.98:0/61811646 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55796896a0d0 0x55796895eb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:30:15.524 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:15.515+0000 7f602effd700 1 -- 172.21.15.98:0/61811646 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x55796896b650 msgr2=0x5579689727e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-14T19:30:15.524 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:15.515+0000 7f602effd700 1 --2- 172.21.15.98:0/61811646 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x55796896b650 0x5579689727e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:30:15.524 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:15.515+0000 7f602effd700 1 -- 172.21.15.98:0/61811646 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55796870d200 con 0x55796896aa20 2023-07-14T19:30:15.525 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:15.516+0000 7f602effd700 1 --2- 172.21.15.98:0/61811646 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55796896aa20 0x55796897b010 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x7f6020006010 tx=0x7f6020009f80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b6331dde531245e6 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:30:15.525 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:15.516+0000 7f60167fc700 1 -- 172.21.15.98:0/61811646 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f60200056a0 con 0x55796896aa20 2023-07-14T19:30:15.525 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:15.516+0000 7f60167fc700 1 -- 172.21.15.98:0/61811646 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f602000fab0 con 0x55796896aa20 2023-07-14T19:30:15.525 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:15.516+0000 7f60167fc700 1 -- 172.21.15.98:0/61811646 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f602002b7a0 con 0x55796896aa20 2023-07-14T19:30:15.525 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:15.516+0000 7f60420fdb80 1 -- 172.21.15.98:0/61811646 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55796896aa20 msgr2=0x55796897b010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:30:15.526 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:15.516+0000 7f60420fdb80 1 --2- 172.21.15.98:0/61811646 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55796896aa20 0x55796897b010 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x7f6020006010 tx=0x7f6020009f80 comp rx=0 tx=0).stop 2023-07-14T19:30:15.526 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:15.516+0000 7f60420fdb80 1 -- 172.21.15.98:0/61811646 shutdown_connections 2023-07-14T19:30:15.526 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:15.516+0000 7f60420fdb80 1 --2- 172.21.15.98:0/61811646 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55796896a0d0 0x55796895eb00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:30:15.526 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:15.516+0000 7f60420fdb80 1 --2- 172.21.15.98:0/61811646 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x55796896b650 0x5579689727e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:30:15.527 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:15.516+0000 7f60420fdb80 1 --2- 172.21.15.98:0/61811646 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55796896aa20 0x55796897b010 unknown :-1 s=CLOSED pgs=375 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:30:15.527 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:15.516+0000 7f60420fdb80 1 -- 172.21.15.98:0/61811646 >> 172.21.15.98:0/61811646 conn(0x55796895d630 msgr2=0x557968962950 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:30:15.527 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:15.516+0000 7f60420fdb80 1 -- 172.21.15.98:0/61811646 shutdown_connections 2023-07-14T19:30:15.527 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:15.516+0000 7f60420fdb80 1 -- 172.21.15.98:0/61811646 wait complete. 2023-07-14T19:30:15.527 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:15.517+0000 7f60420fdb80 1 -- start start 2023-07-14T19:30:15.528 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:15.517+0000 7f60420fdb80 1 Processor -- start 2023-07-14T19:30:15.528 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:15.517+0000 7f60420fdb80 1 --2- >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x55796896a0d0 0x5579689761e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:30:15.528 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:15.517+0000 7f60420fdb80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55796896aa20 0x557968976720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:30:15.528 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:15.517+0000 7f60420fdb80 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55796896b650 0x5579689795d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:30:15.528 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:15.517+0000 7f60420fdb80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x557968979b10 con 0x55796896aa20 2023-07-14T19:30:15.529 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:15.517+0000 7f60420fdb80 1 -- --> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] -- mon_getmap magic: 0 v1 -- 0x557968979c80 con 0x55796896a0d0 2023-07-14T19:30:15.529 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:15.517+0000 7f60420fdb80 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 v1 -- 0x557968979e40 con 0x55796896b650 2023-07-14T19:30:15.529 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:15.517+0000 7f60420fdb80 10 client.0 _ll_register_callbacks cb 0x7f6020009810 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-07-14T19:30:15.529 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:15.517+0000 7f602effd700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55796896aa20 0x557968976720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:30:15.529 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:15.517+0000 7f602effd700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55796896aa20 0x557968976720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:46870/0 (socket says 172.21.15.98:46870) 2023-07-14T19:30:15.529 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:15.517+0000 7f602effd700 1 -- 172.21.15.98:0/3332135676 learned_addr learned my addr 172.21.15.98:0/3332135676 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-07-14T19:30:15.530 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:15.517+0000 7f602ffff700 1 --2- 172.21.15.98:0/3332135676 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55796896b650 0x5579689795d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:30:15.530 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:15.517+0000 7f602f7fe700 1 --2- 172.21.15.98:0/3332135676 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x55796896a0d0 0x5579689761e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:30:15.530 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:15.518+0000 7f602effd700 1 -- 172.21.15.98:0/3332135676 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55796896b650 msgr2=0x5579689795d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-14T19:30:15.530 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:15.518+0000 7f602effd700 1 --2- 172.21.15.98:0/3332135676 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55796896b650 0x5579689795d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:30:15.530 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:15.518+0000 7f602effd700 1 -- 172.21.15.98:0/3332135676 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x55796896a0d0 msgr2=0x5579689761e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-14T19:30:15.530 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:15.518+0000 7f602effd700 1 --2- 172.21.15.98:0/3332135676 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x55796896a0d0 0x5579689761e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:30:15.530 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:15.518+0000 7f602effd700 1 -- 172.21.15.98:0/3332135676 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5579686cfc10 con 0x55796896aa20 2023-07-14T19:30:15.531 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:15.518+0000 7f602ffff700 1 --2- 172.21.15.98:0/3332135676 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55796896b650 0x5579689795d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-07-14T19:30:15.531 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:15.518+0000 7f602f7fe700 1 --2- 172.21.15.98:0/3332135676 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x55796896a0d0 0x5579689761e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-07-14T19:30:15.531 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:15.518+0000 7f602effd700 1 --2- 172.21.15.98:0/3332135676 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55796896aa20 0x557968976720 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 crypto rx=0x7f6020000c00 tx=0x7f6020005a00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=40962a4c830ea876 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:30:15.531 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:15.518+0000 7f600effd700 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2023-07-14T19:30:15.531 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:15.518+0000 7f600effd700 1 -- 172.21.15.98:0/3332135676 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6020004a90 con 0x55796896aa20 2023-07-14T19:30:15.531 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:15.518+0000 7f60420fdb80 1 -- 172.21.15.98:0/3332135676 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f602001e040 con 0x55796896aa20 2023-07-14T19:30:15.531 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:15.518+0000 7f600effd700 1 -- 172.21.15.98:0/3332135676 <== mon.0 v2:172.21.15.98:3300/0 2 ==== osd_map(85..85 src has 1..85) v4 ==== 8211+0+0 (secure 0 0 0) 0x7f6020000e40 con 0x55796896aa20 2023-07-14T19:30:15.531 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:15.518+0000 7f600effd700 1 -- 172.21.15.98:0/3332135676 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=22) v2 ==== 24+0+0 (secure 0 0 0) 0x7f6020004bf0 con 0x55796896aa20 2023-07-14T19:30:15.532 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:15.518+0000 7f60420fdb80 10 client.15228 fetch_fsmap learned FSMap version 22 2023-07-14T19:30:15.532 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:15.518+0000 7f60420fdb80 1 -- 172.21.15.98:0/3332135676 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=22,monmap=4+}) v3 -- 0x5579686e5260 con 0x55796896aa20 2023-07-14T19:30:15.532 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:15.518+0000 7f600effd700 1 -- 172.21.15.98:0/3332135676 <== mon.0 v2:172.21.15.98:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f602001e040 con 0x55796896aa20 2023-07-14T19:30:15.532 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:15.518+0000 7f600effd700 1 -- 172.21.15.98:0/3332135676 <== mon.0 v2:172.21.15.98:3300/0 5 ==== fsmap(e 22) v1 ==== 7462+0+0 (secure 0 0 0) 0x7f6020053070 con 0x55796896aa20 2023-07-14T19:30:15.532 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:15.518+0000 7f60420fdb80 10 client.15228 fetch_fsmap finished waiting for FSMap version 22 2023-07-14T19:30:15.532 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:15.518+0000 7f60420fdb80 10 client.15228 resolve_mds: resolved name 'h' to daemon mds.h 2023-07-14T19:30:15.532 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:15.518+0000 7f60420fdb80 20 client.15228 populate_metadata read hostname 'smithi098' 2023-07-14T19:30:15.533 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:15.518+0000 7f60420fdb80 1 --2- 172.21.15.98:0/3332135676 >> [v2:172.21.15.101:6836/485741528,v1:172.21.15.101:6837/485741528] conn(0x5579689d7620 0x5579689d44f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:30:15.533 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:15.518+0000 7f60420fdb80 4 client.15228 mds_command: new command op to 24479 tid=0[{"prefix": "status"}] 2023-07-14T19:30:15.533 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:15.518+0000 7f60420fdb80 1 -- 172.21.15.98:0/3332135676 --> [v2:172.21.15.101:6836/485741528,v1:172.21.15.101:6837/485741528] -- command(tid 0: {"prefix": "status"}) v1 -- 0x5579688b1e60 con 0x5579689d7620 2023-07-14T19:30:15.533 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:15.519+0000 7f602f7fe700 1 --2- 172.21.15.98:0/3332135676 >> [v2:172.21.15.101:6836/485741528,v1:172.21.15.101:6837/485741528] conn(0x5579689d7620 0x5579689d44f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:30:15.533 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:15.519+0000 7f600effd700 10 client.15228 ms_handle_connect on v2:172.21.15.101:6836/485741528 2023-07-14T19:30:15.533 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:15.519+0000 7f602f7fe700 1 --2- 172.21.15.98:0/3332135676 >> [v2:172.21.15.101:6836/485741528,v1:172.21.15.101:6837/485741528] conn(0x5579689d7620 0x5579689d44f0 crc :-1 s=READY pgs=19 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=e3f4e67036baf0a server_cookie=3a8833149d84491 in_seq=0 out_seq=0 2023-07-14T19:30:15.533 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:15.519+0000 7f600effd700 1 -- 172.21.15.98:0/3332135676 <== mds.2 v2:172.21.15.101:6836/485741528 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+332 (crc 0 0 0) 0x5579688b1e60 con 0x5579689d7620 2023-07-14T19:30:15.534 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:15.519+0000 7f600effd700 10 client.15228 handle_command_reply: tid=0 2023-07-14T19:30:15.534 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:15.519+0000 7f60420fdb80 1 client.15228 shutdown 2023-07-14T19:30:15.534 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:15.520+0000 7f60420fdb80 1 -- 172.21.15.98:0/3332135676 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55796896aa20 msgr2=0x557968976720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:30:15.534 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:15.520+0000 7f60420fdb80 1 --2- 172.21.15.98:0/3332135676 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55796896aa20 0x557968976720 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 crypto rx=0x7f6020000c00 tx=0x7f6020005a00 comp rx=0 tx=0).stop 2023-07-14T19:30:15.534 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:15.520+0000 7f60420fdb80 1 -- 172.21.15.98:0/3332135676 shutdown_connections 2023-07-14T19:30:15.534 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:15.520+0000 7f60420fdb80 1 --2- 172.21.15.98:0/3332135676 >> [v2:172.21.15.101:6836/485741528,v1:172.21.15.101:6837/485741528] conn(0x5579689d7620 0x5579689d44f0 crc :-1 s=READY pgs=19 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:30:15.534 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:15.520+0000 7f60420fdb80 1 --2- 172.21.15.98:0/3332135676 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55796896b650 0x5579689795d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:30:15.535 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:15.520+0000 7f600effd700 0 client.15228 ms_handle_reset on v2:172.21.15.101:6836/485741528 2023-07-14T19:30:15.535 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:15.520+0000 7f60420fdb80 1 --2- 172.21.15.98:0/3332135676 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x55796896a0d0 0x5579689761e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:30:15.535 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:15.520+0000 7f60420fdb80 1 --2- 172.21.15.98:0/3332135676 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55796896aa20 0x557968976720 unknown :-1 s=CLOSED pgs=376 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:30:15.535 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:15.520+0000 7f60420fdb80 1 -- 172.21.15.98:0/3332135676 >> 172.21.15.98:0/3332135676 conn(0x55796895d630 msgr2=0x557968971980 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:30:15.535 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:15.520+0000 7f60420fdb80 1 -- 172.21.15.98:0/3332135676 shutdown_connections 2023-07-14T19:30:15.535 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:15.520+0000 7f60420fdb80 1 -- 172.21.15.98:0/3332135676 wait complete. 2023-07-14T19:30:15.535 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:15.520+0000 7f60420fdb80 20 client.15228 trim_cache size 0 max 16384 2023-07-14T19:30:15.536 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2023-07-14T19:30:15.536 INFO:teuthology.orchestra.run.smithi098.stdout: "cluster_fsid": "9adea648-227b-11ee-9b33-001a4aab830c", 2023-07-14T19:30:15.536 INFO:teuthology.orchestra.run.smithi098.stdout: "whoami": 2, 2023-07-14T19:30:15.536 INFO:teuthology.orchestra.run.smithi098.stdout: "id": 24479, 2023-07-14T19:30:15.536 INFO:teuthology.orchestra.run.smithi098.stdout: "want_state": "up:active", 2023-07-14T19:30:15.537 INFO:teuthology.orchestra.run.smithi098.stdout: "state": "up:active", 2023-07-14T19:30:15.537 INFO:teuthology.orchestra.run.smithi098.stdout: "fs_name": "cephfs", 2023-07-14T19:30:15.537 INFO:teuthology.orchestra.run.smithi098.stdout: "rank_uptime": 69.849795380000003, 2023-07-14T19:30:15.537 INFO:teuthology.orchestra.run.smithi098.stdout: "mdsmap_epoch": 22, 2023-07-14T19:30:15.537 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap_epoch": 85, 2023-07-14T19:30:15.537 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap_epoch_barrier": 83, 2023-07-14T19:30:15.537 INFO:teuthology.orchestra.run.smithi098.stdout: "uptime": 88.830969030999995 2023-07-14T19:30:15.537 INFO:teuthology.orchestra.run.smithi098.stdout:} 2023-07-14T19:30:15.538 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:15.520+0000 7f6035f65700 1 -- 172.21.15.98:0/1981388592 >> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] conn(0x7f60180a11e0 msgr2=0x7f60180a3690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:30:15.539 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:15.520+0000 7f6035f65700 1 --2- 172.21.15.98:0/1981388592 >> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] conn(0x7f60180a11e0 0x7f60180a3690 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f60300ffb80 tx=0x7f602400a3b0 comp rx=0 tx=0).stop 2023-07-14T19:30:15.539 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:15.520+0000 7f6035f65700 1 -- 172.21.15.98:0/1981388592 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f60301006b0 msgr2=0x7f603019bd70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:30:15.539 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:15.520+0000 7f6035f65700 1 --2- 172.21.15.98:0/1981388592 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f60301006b0 0x7f603019bd70 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 crypto rx=0x7f6028007ee0 tx=0x7f602800d3b0 comp rx=0 tx=0).stop 2023-07-14T19:30:15.539 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:15.520+0000 7f6035f65700 1 -- 172.21.15.98:0/1981388592 shutdown_connections 2023-07-14T19:30:15.539 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:15.520+0000 7f6035f65700 1 --2- 172.21.15.98:0/1981388592 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f60301006b0 0x7f603019bd70 unknown :-1 s=CLOSED pgs=372 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:30:15.539 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:15.520+0000 7f6035f65700 1 --2- 172.21.15.98:0/1981388592 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f60300feb30 0x7f60301947c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:30:15.540 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:15.520+0000 7f6035f65700 1 --2- 172.21.15.98:0/1981388592 >> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] conn(0x7f60180a11e0 0x7f60180a3690 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:30:15.540 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:15.520+0000 7f6035f65700 1 --2- 172.21.15.98:0/1981388592 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f60300ffd20 0x7f6030194d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:30:15.540 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:15.520+0000 7f6035f65700 1 -- 172.21.15.98:0/1981388592 >> 172.21.15.98:0/1981388592 conn(0x7f60300fa150 msgr2=0x7f60300fc370 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:30:15.540 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:15.520+0000 7f6035f65700 1 -- 172.21.15.98:0/1981388592 shutdown_connections 2023-07-14T19:30:15.540 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:15.520+0000 7f6035f65700 1 -- 172.21.15.98:0/1981388592 wait complete. 2023-07-14T19:30:15.540 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.i status 2023-07-14T19:30:15.662 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:15.662+0000 7f49cbf95700 1 -- 172.21.15.98:0/2831621489 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f49c4103070 msgr2=0x7f49c4105450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:30:15.663 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:15.662+0000 7f49cbf95700 1 --2- 172.21.15.98:0/2831621489 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f49c4103070 0x7f49c4105450 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 crypto rx=0x7f49b4009990 tx=0x7f49b4009ca0 comp rx=0 tx=0).stop 2023-07-14T19:30:15.663 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:15.662+0000 7f49cbf95700 1 -- 172.21.15.98:0/2831621489 shutdown_connections 2023-07-14T19:30:15.663 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:15.662+0000 7f49cbf95700 1 --2- 172.21.15.98:0/2831621489 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f49c4105990 0x7f49c4107d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:30:15.663 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:15.662+0000 7f49cbf95700 1 --2- 172.21.15.98:0/2831621489 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f49c4100750 0x7f49c4102b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:30:15.663 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:15.662+0000 7f49cbf95700 1 --2- 172.21.15.98:0/2831621489 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f49c4103070 0x7f49c4105450 unknown :-1 s=CLOSED pgs=377 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:30:15.663 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:15.662+0000 7f49cbf95700 1 -- 172.21.15.98:0/2831621489 >> 172.21.15.98:0/2831621489 conn(0x7f49c40fa150 msgr2=0x7f49c40fc580 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:30:15.664 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:15.662+0000 7f49cbf95700 1 -- 172.21.15.98:0/2831621489 shutdown_connections 2023-07-14T19:30:15.664 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:15.662+0000 7f49cbf95700 1 -- 172.21.15.98:0/2831621489 wait complete. 2023-07-14T19:30:15.664 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:15.662+0000 7f49cbf95700 1 Processor -- start 2023-07-14T19:30:15.664 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:15.662+0000 7f49cbf95700 1 -- start start 2023-07-14T19:30:15.665 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:15.662+0000 7f49cbf95700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f49c4100750 0x7f49c4197bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:30:15.666 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:15.662+0000 7f49cbf95700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f49c4103070 0x7f49c4198100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:30:15.666 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:15.662+0000 7f49cbf95700 1 --2- >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f49c4105990 0x7f49c419f170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:30:15.666 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:15.662+0000 7f49cbf95700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f49c419f740 con 0x7f49c4100750 2023-07-14T19:30:15.666 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:15.662+0000 7f49cbf95700 1 -- --> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f49c419f880 con 0x7f49c4105990 2023-07-14T19:30:15.666 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:15.662+0000 7f49cbf95700 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f49c419fa10 con 0x7f49c4103070 2023-07-14T19:30:15.666 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:15.662+0000 7f49c9d31700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f49c4100750 0x7f49c4197bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:30:15.667 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:15.663+0000 7f49c9d31700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f49c4100750 0x7f49c4197bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:46882/0 (socket says 172.21.15.98:46882) 2023-07-14T19:30:15.667 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:15.663+0000 7f49c9d31700 1 -- 172.21.15.98:0/1921696566 learned_addr learned my addr 172.21.15.98:0/1921696566 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-07-14T19:30:15.667 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:15.663+0000 7f49ca532700 1 --2- 172.21.15.98:0/1921696566 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f49c4105990 0x7f49c419f170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:30:15.667 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:15.663+0000 7f49c9530700 1 --2- 172.21.15.98:0/1921696566 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f49c4103070 0x7f49c4198100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:30:15.667 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:15.663+0000 7f49c9d31700 1 -- 172.21.15.98:0/1921696566 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f49c4103070 msgr2=0x7f49c4198100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:30:15.667 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:15.663+0000 7f49c9d31700 1 --2- 172.21.15.98:0/1921696566 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f49c4103070 0x7f49c4198100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:30:15.667 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:15.663+0000 7f49c9d31700 1 -- 172.21.15.98:0/1921696566 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f49c4105990 msgr2=0x7f49c419f170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:30:15.668 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:15.663+0000 7f49c9d31700 1 --2- 172.21.15.98:0/1921696566 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f49c4105990 0x7f49c419f170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:30:15.668 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:15.663+0000 7f49c9d31700 1 -- 172.21.15.98:0/1921696566 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f49b40095f0 con 0x7f49c4100750 2023-07-14T19:30:15.668 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:15.663+0000 7f49c9530700 1 --2- 172.21.15.98:0/1921696566 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f49c4103070 0x7f49c4198100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-07-14T19:30:15.668 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:15.663+0000 7f49c9d31700 1 --2- 172.21.15.98:0/1921696566 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f49c4100750 0x7f49c4197bc0 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7f49c000d950 tx=0x7f49c000dc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:30:15.668 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:15.663+0000 7f49baffd700 1 -- 172.21.15.98:0/1921696566 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f49c0004220 con 0x7f49c4100750 2023-07-14T19:30:15.668 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:15.663+0000 7f49baffd700 1 -- 172.21.15.98:0/1921696566 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f49c0007620 con 0x7f49c4100750 2023-07-14T19:30:15.668 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:15.663+0000 7f49cbf95700 1 -- 172.21.15.98:0/1921696566 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f49c419fcf0 con 0x7f49c4100750 2023-07-14T19:30:15.668 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:15.663+0000 7f49baffd700 1 -- 172.21.15.98:0/1921696566 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f49c0005690 con 0x7f49c4100750 2023-07-14T19:30:15.669 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:15.663+0000 7f49cbf95700 1 -- 172.21.15.98:0/1921696566 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f49c41a01b0 con 0x7f49c4100750 2023-07-14T19:30:15.669 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:15.665+0000 7f49baffd700 1 -- 172.21.15.98:0/1921696566 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 17) v1 ==== 135766+0+0 (secure 0 0 0) 0x7f49c0004750 con 0x7f49c4100750 2023-07-14T19:30:15.673 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:15.666+0000 7f49baffd700 1 --2- 172.21.15.98:0/1921696566 >> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] conn(0x7f49b00a11e0 0x7f49b00a3690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:30:15.674 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:15.666+0000 7f49c9530700 1 --2- 172.21.15.98:0/1921696566 >> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] conn(0x7f49b00a11e0 0x7f49b00a3690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:30:15.674 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:15.666+0000 7f49baffd700 1 -- 172.21.15.98:0/1921696566 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 8211+0+0 (secure 0 0 0) 0x7f49c00cc900 con 0x7f49c4100750 2023-07-14T19:30:15.675 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:15.667+0000 7f49c9530700 1 --2- 172.21.15.98:0/1921696566 >> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] conn(0x7f49b00a11e0 0x7f49b00a3690 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f49b4006010 tx=0x7f49b400b540 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:30:15.675 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:15.667+0000 7f49d812db80 1 Processor -- start 2023-07-14T19:30:15.675 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:15.667+0000 7f49d812db80 1 -- start start 2023-07-14T19:30:15.675 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:15.667+0000 7f49d812db80 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55cffd20ff10 0x55cffd210320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:30:15.675 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:15.667+0000 7f49d812db80 1 --2- >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x55cffd210860 0x55cffd210e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:30:15.675 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:15.667+0000 7f49d812db80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55cffd211460 0x55cffd2118d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:30:15.675 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:15.667+0000 7f49d812db80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x55cffd114c60 con 0x55cffd211460 2023-07-14T19:30:15.676 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:15.667+0000 7f49d812db80 1 -- --> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] -- mon_getmap magic: 0 v1 -- 0x55cffd1679f0 con 0x55cffd210860 2023-07-14T19:30:15.676 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:15.667+0000 7f49d812db80 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 v1 -- 0x55cffd160e80 con 0x55cffd20ff10 2023-07-14T19:30:15.676 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:15.668+0000 7f49ca532700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55cffd211460 0x55cffd2118d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:30:15.676 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:15.668+0000 7f49ca532700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55cffd211460 0x55cffd2118d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:46894/0 (socket says 172.21.15.98:46894) 2023-07-14T19:30:15.676 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:15.668+0000 7f49ca532700 1 -- 172.21.15.98:0/140604210 learned_addr learned my addr 172.21.15.98:0/140604210 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-07-14T19:30:15.676 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:15.668+0000 7f49c9d31700 1 --2- 172.21.15.98:0/140604210 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55cffd20ff10 0x55cffd210320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:30:15.676 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:15.668+0000 7f49ca532700 1 -- 172.21.15.98:0/140604210 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55cffd20ff10 msgr2=0x55cffd210320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-14T19:30:15.677 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:15.668+0000 7f49c9530700 1 --2- 172.21.15.98:0/140604210 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x55cffd210860 0x55cffd210e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:30:15.677 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:15.668+0000 7f49ca532700 1 --2- 172.21.15.98:0/140604210 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55cffd20ff10 0x55cffd210320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:30:15.677 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:15.668+0000 7f49ca532700 1 -- 172.21.15.98:0/140604210 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x55cffd210860 msgr2=0x55cffd210e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-14T19:30:15.677 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:15.668+0000 7f49ca532700 1 --2- 172.21.15.98:0/140604210 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x55cffd210860 0x55cffd210e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:30:15.677 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:15.668+0000 7f49ca532700 1 -- 172.21.15.98:0/140604210 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55cffcf8ed90 con 0x55cffd211460 2023-07-14T19:30:15.677 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:15.668+0000 7f49c9d31700 1 --2- 172.21.15.98:0/140604210 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55cffd20ff10 0x55cffd210320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-07-14T19:30:15.677 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:15.668+0000 7f49ca532700 1 --2- 172.21.15.98:0/140604210 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55cffd211460 0x55cffd2118d0 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 crypto rx=0x7f49bc00b560 tx=0x7f49bc008830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fb395f4c7a5d8571 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:30:15.677 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:15.668+0000 7f49affff700 1 -- 172.21.15.98:0/140604210 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f49bc019070 con 0x55cffd211460 2023-07-14T19:30:15.678 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:15.668+0000 7f49affff700 1 -- 172.21.15.98:0/140604210 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f49bc011040 con 0x55cffd211460 2023-07-14T19:30:15.678 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:15.668+0000 7f49affff700 1 -- 172.21.15.98:0/140604210 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f49bc010050 con 0x55cffd211460 2023-07-14T19:30:15.678 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:15.669+0000 7f49d812db80 1 -- 172.21.15.98:0/140604210 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55cffd211460 msgr2=0x55cffd2118d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:30:15.678 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:15.669+0000 7f49d812db80 1 --2- 172.21.15.98:0/140604210 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55cffd211460 0x55cffd2118d0 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 crypto rx=0x7f49bc00b560 tx=0x7f49bc008830 comp rx=0 tx=0).stop 2023-07-14T19:30:15.678 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:15.669+0000 7f49d812db80 1 -- 172.21.15.98:0/140604210 shutdown_connections 2023-07-14T19:30:15.678 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:15.669+0000 7f49d812db80 1 --2- 172.21.15.98:0/140604210 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55cffd20ff10 0x55cffd210320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:30:15.678 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:15.669+0000 7f49d812db80 1 --2- 172.21.15.98:0/140604210 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55cffd211460 0x55cffd2118d0 unknown :-1 s=CLOSED pgs=379 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:30:15.679 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:15.669+0000 7f49d812db80 1 --2- 172.21.15.98:0/140604210 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x55cffd210860 0x55cffd210e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:30:15.679 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:15.669+0000 7f49d812db80 1 -- 172.21.15.98:0/140604210 >> 172.21.15.98:0/140604210 conn(0x55cffd205630 msgr2=0x55cffd20a980 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:30:15.679 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:15.669+0000 7f49d812db80 1 -- 172.21.15.98:0/140604210 shutdown_connections 2023-07-14T19:30:15.679 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:15.669+0000 7f49d812db80 1 -- 172.21.15.98:0/140604210 wait complete. 2023-07-14T19:30:15.679 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:15.669+0000 7f49d812db80 1 -- start start 2023-07-14T19:30:15.679 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:15.669+0000 7f49d812db80 1 Processor -- start 2023-07-14T19:30:15.679 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:15.670+0000 7f49d812db80 1 --2- >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x55cffd20ff10 0x55cffd219880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:30:15.679 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:15.670+0000 7f49d812db80 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55cffd210860 0x55cffd219dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:30:15.680 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:15.670+0000 7f49d812db80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55cffd211460 0x55cffd220e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:30:15.680 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:15.670+0000 7f49d812db80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x55cffd100a70 con 0x55cffd211460 2023-07-14T19:30:15.680 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:15.670+0000 7f49d812db80 1 -- --> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] -- mon_getmap magic: 0 v1 -- 0x55cffd100770 con 0x55cffd20ff10 2023-07-14T19:30:15.680 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:15.670+0000 7f49d812db80 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 v1 -- 0x55cffcfe48d0 con 0x55cffd210860 2023-07-14T19:30:15.680 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:15.670+0000 7f49d812db80 10 client.0 _ll_register_callbacks cb 0x55cffd0fe710 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-07-14T19:30:15.680 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:15.670+0000 7f49ca532700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55cffd211460 0x55cffd220e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:30:15.680 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:15.670+0000 7f49ca532700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55cffd211460 0x55cffd220e00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:46900/0 (socket says 172.21.15.98:46900) 2023-07-14T19:30:15.681 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:15.670+0000 7f49ca532700 1 -- 172.21.15.98:0/1906573510 learned_addr learned my addr 172.21.15.98:0/1906573510 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-07-14T19:30:15.681 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:15.670+0000 7f49c9d31700 1 --2- 172.21.15.98:0/1906573510 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x55cffd20ff10 0x55cffd219880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:30:15.681 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:15.670+0000 7f49c9530700 1 --2- 172.21.15.98:0/1906573510 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55cffd210860 0x55cffd219dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:30:15.681 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:15.670+0000 7f49ca532700 1 -- 172.21.15.98:0/1906573510 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55cffd210860 msgr2=0x55cffd219dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-14T19:30:15.681 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:15.670+0000 7f49ca532700 1 --2- 172.21.15.98:0/1906573510 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55cffd210860 0x55cffd219dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:30:15.681 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:15.670+0000 7f49ca532700 1 -- 172.21.15.98:0/1906573510 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x55cffd20ff10 msgr2=0x55cffd219880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-14T19:30:15.681 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:15.670+0000 7f49ca532700 1 --2- 172.21.15.98:0/1906573510 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x55cffd20ff10 0x55cffd219880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:30:15.681 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:15.670+0000 7f49ca532700 1 -- 172.21.15.98:0/1906573510 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f49bc008e20 con 0x55cffd211460 2023-07-14T19:30:15.682 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:15.670+0000 7f49ca532700 1 --2- 172.21.15.98:0/1906573510 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55cffd211460 0x55cffd220e00 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7f49bc008b90 tx=0x7f49bc00b3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8aec4cfc93b5d088 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:30:15.682 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:15.670+0000 7f49ad7fa700 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2023-07-14T19:30:15.682 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:15.670+0000 7f49ad7fa700 1 -- 172.21.15.98:0/1906573510 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f49bc019070 con 0x55cffd211460 2023-07-14T19:30:15.682 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:15.670+0000 7f49ad7fa700 1 -- 172.21.15.98:0/1906573510 <== mon.0 v2:172.21.15.98:3300/0 2 ==== osd_map(85..85 src has 1..85) v4 ==== 8211+0+0 (secure 0 0 0) 0x7f49bc004cd0 con 0x55cffd211460 2023-07-14T19:30:15.682 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:15.670+0000 7f49d812db80 1 -- 172.21.15.98:0/1906573510 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55cffd0ba4a0 con 0x55cffd211460 2023-07-14T19:30:15.682 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:15.671+0000 7f49ad7fa700 1 -- 172.21.15.98:0/1906573510 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=22) v2 ==== 24+0+0 (secure 0 0 0) 0x7f49bc0041a0 con 0x55cffd211460 2023-07-14T19:30:15.682 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:15.671+0000 7f49d812db80 10 client.15240 fetch_fsmap learned FSMap version 22 2023-07-14T19:30:15.683 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:15.671+0000 7f49d812db80 1 -- 172.21.15.98:0/1906573510 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=22,monmap=4+}) v3 -- 0x55cffcf77c10 con 0x55cffd211460 2023-07-14T19:30:15.683 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:15.671+0000 7f49ad7fa700 1 -- 172.21.15.98:0/1906573510 <== mon.0 v2:172.21.15.98:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x55cffd0ba4a0 con 0x55cffd211460 2023-07-14T19:30:15.683 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:15.671+0000 7f49ad7fa700 1 -- 172.21.15.98:0/1906573510 <== mon.0 v2:172.21.15.98:3300/0 5 ==== fsmap(e 22) v1 ==== 7462+0+0 (secure 0 0 0) 0x7f49bc009400 con 0x55cffd211460 2023-07-14T19:30:15.683 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:15.671+0000 7f49d812db80 10 client.15240 fetch_fsmap finished waiting for FSMap version 22 2023-07-14T19:30:15.683 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:15.671+0000 7f49d812db80 10 client.15240 resolve_mds: resolved name 'i' to daemon mds.i 2023-07-14T19:30:15.683 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:15.671+0000 7f49d812db80 20 client.15240 populate_metadata read hostname 'smithi098' 2023-07-14T19:30:15.683 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:15.671+0000 7f49d812db80 1 --2- 172.21.15.98:0/1906573510 >> [v2:172.21.15.139:6838/1258854143,v1:172.21.15.139:6839/1258854143] conn(0x55cffd224aa0 0x55cffd223eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:30:15.683 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:15.671+0000 7f49d812db80 4 client.15240 mds_command: new command op to 24493 tid=0[{"prefix": "get_command_descriptions"}] 2023-07-14T19:30:15.684 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:15.671+0000 7f49d812db80 1 -- 172.21.15.98:0/1906573510 --> [v2:172.21.15.139:6838/1258854143,v1:172.21.15.139:6839/1258854143] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55cffd1ded50 con 0x55cffd224aa0 2023-07-14T19:30:15.684 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:15.671+0000 7f49c9d31700 1 --2- 172.21.15.98:0/1906573510 >> [v2:172.21.15.139:6838/1258854143,v1:172.21.15.139:6839/1258854143] conn(0x55cffd224aa0 0x55cffd223eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:30:15.684 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:15.672+0000 7f49c9d31700 1 --2- 172.21.15.98:0/1906573510 >> [v2:172.21.15.139:6838/1258854143,v1:172.21.15.139:6839/1258854143] conn(0x55cffd224aa0 0x55cffd223eb0 crc :-1 s=READY pgs=19 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=e1cecbce182be1 server_cookie=4d9a371ad8c81501 in_seq=0 out_seq=0 2023-07-14T19:30:15.684 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:15.672+0000 7f49ad7fa700 10 client.15240 ms_handle_connect on v2:172.21.15.139:6838/1258854143 2023-07-14T19:30:15.684 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:15.672+0000 7f49ad7fa700 1 -- 172.21.15.98:0/1906573510 <== mds.1 v2:172.21.15.139:6838/1258854143 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18017 (crc 0 0 0) 0x55cffd1ded50 con 0x55cffd224aa0 2023-07-14T19:30:15.684 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:15.672+0000 7f49ad7fa700 10 client.15240 handle_command_reply: tid=0 2023-07-14T19:30:15.684 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:15.672+0000 7f49d812db80 1 client.15240 shutdown 2023-07-14T19:30:15.685 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:15.673+0000 7f49d812db80 1 -- 172.21.15.98:0/1906573510 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55cffd211460 msgr2=0x55cffd220e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:30:15.685 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:15.673+0000 7f49d812db80 1 --2- 172.21.15.98:0/1906573510 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55cffd211460 0x55cffd220e00 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7f49bc008b90 tx=0x7f49bc00b3b0 comp rx=0 tx=0).stop 2023-07-14T19:30:15.685 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:15.673+0000 7f49d812db80 1 -- 172.21.15.98:0/1906573510 shutdown_connections 2023-07-14T19:30:15.685 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:15.673+0000 7f49d812db80 1 --2- 172.21.15.98:0/1906573510 >> [v2:172.21.15.139:6838/1258854143,v1:172.21.15.139:6839/1258854143] conn(0x55cffd224aa0 0x55cffd223eb0 crc :-1 s=READY pgs=19 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:30:15.685 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:15.673+0000 7f49d812db80 1 --2- 172.21.15.98:0/1906573510 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55cffd211460 0x55cffd220e00 unknown :-1 s=CLOSED pgs=380 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:30:15.685 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:15.673+0000 7f49d812db80 1 --2- 172.21.15.98:0/1906573510 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x55cffd20ff10 0x55cffd219880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:30:15.685 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:15.673+0000 7f49d812db80 1 --2- 172.21.15.98:0/1906573510 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55cffd210860 0x55cffd219dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:30:15.685 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:15.673+0000 7f49ad7fa700 0 client.15240 ms_handle_reset on v2:172.21.15.139:6838/1258854143 2023-07-14T19:30:15.686 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:15.673+0000 7f49d812db80 1 -- 172.21.15.98:0/1906573510 >> 172.21.15.98:0/1906573510 conn(0x55cffd205630 msgr2=0x55cffd209600 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:30:15.686 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:15.673+0000 7f49d812db80 1 -- 172.21.15.98:0/1906573510 shutdown_connections 2023-07-14T19:30:15.686 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:15.673+0000 7f49d812db80 1 -- 172.21.15.98:0/1906573510 wait complete. 2023-07-14T19:30:15.686 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:15.673+0000 7f49d812db80 20 client.15240 trim_cache size 0 max 16384 2023-07-14T19:30:15.692 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:15.691+0000 7f49d812db80 1 Processor -- start 2023-07-14T19:30:15.692 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:15.691+0000 7f49d812db80 1 -- start start 2023-07-14T19:30:15.692 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:15.691+0000 7f49d812db80 1 --2- >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x55cffd20ff10 0x55cffd217840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:30:15.693 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:15.691+0000 7f49d812db80 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55cffd210860 0x55cffd220d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:30:15.693 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:15.691+0000 7f49d812db80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55cffd211460 0x55cffd223eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:30:15.693 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:15.691+0000 7f49d812db80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x55cffd12f790 con 0x55cffd211460 2023-07-14T19:30:15.693 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:15.691+0000 7f49d812db80 1 -- --> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] -- mon_getmap magic: 0 v1 -- 0x55cffcff0de0 con 0x55cffd20ff10 2023-07-14T19:30:15.693 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:15.691+0000 7f49d812db80 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 v1 -- 0x55cffd11f480 con 0x55cffd210860 2023-07-14T19:30:15.693 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:15.691+0000 7f49ca532700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55cffd211460 0x55cffd223eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:30:15.693 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:15.691+0000 7f49ca532700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55cffd211460 0x55cffd223eb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:46908/0 (socket says 172.21.15.98:46908) 2023-07-14T19:30:15.694 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:15.691+0000 7f49ca532700 1 -- 172.21.15.98:0/2100208032 learned_addr learned my addr 172.21.15.98:0/2100208032 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-07-14T19:30:15.694 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:15.691+0000 7f49c9d31700 1 --2- 172.21.15.98:0/2100208032 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x55cffd20ff10 0x55cffd217840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:30:15.694 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:15.691+0000 7f49c9530700 1 --2- 172.21.15.98:0/2100208032 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55cffd210860 0x55cffd220d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:30:15.694 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:15.691+0000 7f49ca532700 1 -- 172.21.15.98:0/2100208032 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55cffd210860 msgr2=0x55cffd220d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-14T19:30:15.694 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:15.691+0000 7f49ca532700 1 --2- 172.21.15.98:0/2100208032 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55cffd210860 0x55cffd220d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:30:15.694 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:15.691+0000 7f49ca532700 1 -- 172.21.15.98:0/2100208032 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x55cffd20ff10 msgr2=0x55cffd217840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-14T19:30:15.694 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:15.691+0000 7f49ca532700 1 --2- 172.21.15.98:0/2100208032 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x55cffd20ff10 0x55cffd217840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:30:15.694 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:15.691+0000 7f49ca532700 1 -- 172.21.15.98:0/2100208032 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55cffcfb5200 con 0x55cffd211460 2023-07-14T19:30:15.695 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:15.691+0000 7f49c9d31700 1 --2- 172.21.15.98:0/2100208032 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x55cffd20ff10 0x55cffd217840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-07-14T19:30:15.695 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:15.691+0000 7f49ca532700 1 --2- 172.21.15.98:0/2100208032 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55cffd211460 0x55cffd223eb0 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0x7f49bc008ae0 tx=0x7f49bc008fb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6478bf53d4d0b7f0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:30:15.695 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:15.692+0000 7f49b8ff9700 1 -- 172.21.15.98:0/2100208032 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f49bc010070 con 0x55cffd211460 2023-07-14T19:30:15.695 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:15.692+0000 7f49b8ff9700 1 -- 172.21.15.98:0/2100208032 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f49bc01a040 con 0x55cffd211460 2023-07-14T19:30:15.695 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:15.692+0000 7f49b8ff9700 1 -- 172.21.15.98:0/2100208032 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f49bc019040 con 0x55cffd211460 2023-07-14T19:30:15.698 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:15.692+0000 7f49d812db80 1 -- 172.21.15.98:0/2100208032 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55cffd211460 msgr2=0x55cffd223eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:30:15.699 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:15.692+0000 7f49d812db80 1 --2- 172.21.15.98:0/2100208032 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55cffd211460 0x55cffd223eb0 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0x7f49bc008ae0 tx=0x7f49bc008fb0 comp rx=0 tx=0).stop 2023-07-14T19:30:15.699 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:15.692+0000 7f49d812db80 1 -- 172.21.15.98:0/2100208032 shutdown_connections 2023-07-14T19:30:15.699 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:15.692+0000 7f49d812db80 1 --2- 172.21.15.98:0/2100208032 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55cffd211460 0x55cffd223eb0 unknown :-1 s=CLOSED pgs=381 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:30:15.699 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:15.692+0000 7f49d812db80 1 --2- 172.21.15.98:0/2100208032 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x55cffd20ff10 0x55cffd217840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:30:15.699 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:15.692+0000 7f49d812db80 1 --2- 172.21.15.98:0/2100208032 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55cffd210860 0x55cffd220d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:30:15.699 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:15.692+0000 7f49d812db80 1 -- 172.21.15.98:0/2100208032 >> 172.21.15.98:0/2100208032 conn(0x55cffd205630 msgr2=0x55cffd20a0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:30:15.700 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:15.692+0000 7f49d812db80 1 -- 172.21.15.98:0/2100208032 shutdown_connections 2023-07-14T19:30:15.700 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:15.692+0000 7f49d812db80 1 -- 172.21.15.98:0/2100208032 wait complete. 2023-07-14T19:30:15.700 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:15.693+0000 7f49d812db80 1 -- start start 2023-07-14T19:30:15.700 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:15.693+0000 7f49d812db80 1 Processor -- start 2023-07-14T19:30:15.700 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:15.693+0000 7f49d812db80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55cffd20ff10 0x55cffd21bf90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:30:15.700 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:15.693+0000 7f49d812db80 1 --2- >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x55cffd210860 0x55cffd21c4d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:30:15.700 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:15.693+0000 7f49d812db80 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55cffd211460 0x55cffd21f4a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:30:15.700 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:15.693+0000 7f49d812db80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x55cffd21f9e0 con 0x55cffd20ff10 2023-07-14T19:30:15.701 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:15.693+0000 7f49d812db80 1 -- --> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] -- mon_getmap magic: 0 v1 -- 0x55cffd21fb50 con 0x55cffd210860 2023-07-14T19:30:15.701 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:15.693+0000 7f49d812db80 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 v1 -- 0x55cffd21fd10 con 0x55cffd211460 2023-07-14T19:30:15.701 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:15.693+0000 7f49d812db80 10 client.0 _ll_register_callbacks cb 0x7f49bc00b4b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-07-14T19:30:15.701 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:15.693+0000 7f49c9d31700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55cffd20ff10 0x55cffd21bf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:30:15.701 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:15.693+0000 7f49c9d31700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55cffd20ff10 0x55cffd21bf90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:46912/0 (socket says 172.21.15.98:46912) 2023-07-14T19:30:15.701 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:15.693+0000 7f49c9d31700 1 -- 172.21.15.98:0/300440339 learned_addr learned my addr 172.21.15.98:0/300440339 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-07-14T19:30:15.701 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:15.693+0000 7f49ca532700 1 --2- 172.21.15.98:0/300440339 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55cffd211460 0x55cffd21f4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:30:15.702 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:15.693+0000 7f49c9530700 1 --2- 172.21.15.98:0/300440339 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x55cffd210860 0x55cffd21c4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:30:15.702 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:15.693+0000 7f49c9d31700 1 -- 172.21.15.98:0/300440339 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55cffd211460 msgr2=0x55cffd21f4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-14T19:30:15.702 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:15.693+0000 7f49c9d31700 1 --2- 172.21.15.98:0/300440339 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55cffd211460 0x55cffd21f4a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:30:15.702 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:15.693+0000 7f49c9d31700 1 -- 172.21.15.98:0/300440339 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x55cffd210860 msgr2=0x55cffd21c4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-14T19:30:15.702 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:15.693+0000 7f49c9d31700 1 --2- 172.21.15.98:0/300440339 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x55cffd210860 0x55cffd21c4d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:30:15.702 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:15.693+0000 7f49c9d31700 1 -- 172.21.15.98:0/300440339 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55cffcf77c10 con 0x55cffd20ff10 2023-07-14T19:30:15.702 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:15.693+0000 7f49c9530700 1 --2- 172.21.15.98:0/300440339 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x55cffd210860 0x55cffd21c4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-07-14T19:30:15.702 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:15.693+0000 7f49c9d31700 1 --2- 172.21.15.98:0/300440339 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55cffd20ff10 0x55cffd21bf90 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x7f49c000b540 tx=0x7f49c0095f80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a37cb8af1491a479 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:30:15.703 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:15.693+0000 7f49ad7fa700 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2023-07-14T19:30:15.703 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:15.694+0000 7f49ad7fa700 1 -- 172.21.15.98:0/300440339 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f49c00a2010 con 0x55cffd20ff10 2023-07-14T19:30:15.703 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:15.694+0000 7f49ad7fa700 1 -- 172.21.15.98:0/300440339 <== mon.0 v2:172.21.15.98:3300/0 2 ==== osd_map(85..85 src has 1..85) v4 ==== 8211+0+0 (secure 0 0 0) 0x7f49c00ca510 con 0x55cffd20ff10 2023-07-14T19:30:15.703 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:15.694+0000 7f49d812db80 1 -- 172.21.15.98:0/300440339 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f49bc011040 con 0x55cffd20ff10 2023-07-14T19:30:15.703 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:15.694+0000 7f49ad7fa700 1 -- 172.21.15.98:0/300440339 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=22) v2 ==== 24+0+0 (secure 0 0 0) 0x7f49c00b2240 con 0x55cffd20ff10 2023-07-14T19:30:15.703 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:15.694+0000 7f49d812db80 10 client.15246 fetch_fsmap learned FSMap version 22 2023-07-14T19:30:15.703 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:15.694+0000 7f49d812db80 1 -- 172.21.15.98:0/300440339 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=22,monmap=4+}) v3 -- 0x55cffcf8d260 con 0x55cffd20ff10 2023-07-14T19:30:15.703 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:15.694+0000 7f49ad7fa700 1 -- 172.21.15.98:0/300440339 <== mon.0 v2:172.21.15.98:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f49c0065030 con 0x55cffd20ff10 2023-07-14T19:30:15.704 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:15.694+0000 7f49ad7fa700 1 -- 172.21.15.98:0/300440339 <== mon.0 v2:172.21.15.98:3300/0 5 ==== fsmap(e 22) v1 ==== 7462+0+0 (secure 0 0 0) 0x7f49c00a1b80 con 0x55cffd20ff10 2023-07-14T19:30:15.704 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:15.694+0000 7f49d812db80 10 client.15246 fetch_fsmap finished waiting for FSMap version 22 2023-07-14T19:30:15.704 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:15.694+0000 7f49d812db80 10 client.15246 resolve_mds: resolved name 'i' to daemon mds.i 2023-07-14T19:30:15.704 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:15.694+0000 7f49d812db80 20 client.15246 populate_metadata read hostname 'smithi098' 2023-07-14T19:30:15.704 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:15.694+0000 7f49d812db80 1 --2- 172.21.15.98:0/300440339 >> [v2:172.21.15.139:6838/1258854143,v1:172.21.15.139:6839/1258854143] conn(0x55cffd27dd00 0x55cffd27a710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:30:15.704 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:15.694+0000 7f49d812db80 4 client.15246 mds_command: new command op to 24493 tid=0[{"prefix": "status"}] 2023-07-14T19:30:15.704 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:15.694+0000 7f49d812db80 1 -- 172.21.15.98:0/300440339 --> [v2:172.21.15.139:6838/1258854143,v1:172.21.15.139:6839/1258854143] -- command(tid 0: {"prefix": "status"}) v1 -- 0x55cffd159e60 con 0x55cffd27dd00 2023-07-14T19:30:15.705 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:15.694+0000 7f49c9530700 1 --2- 172.21.15.98:0/300440339 >> [v2:172.21.15.139:6838/1258854143,v1:172.21.15.139:6839/1258854143] conn(0x55cffd27dd00 0x55cffd27a710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:30:15.705 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:15.695+0000 7f49c9530700 1 --2- 172.21.15.98:0/300440339 >> [v2:172.21.15.139:6838/1258854143,v1:172.21.15.139:6839/1258854143] conn(0x55cffd27dd00 0x55cffd27a710 crc :-1 s=READY pgs=20 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=b4879d3594a89855 server_cookie=e51e3829b17d35a6 in_seq=0 out_seq=0 2023-07-14T19:30:15.705 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:15.695+0000 7f49ad7fa700 10 client.15246 ms_handle_connect on v2:172.21.15.139:6838/1258854143 2023-07-14T19:30:15.705 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:15.695+0000 7f49ad7fa700 1 -- 172.21.15.98:0/300440339 <== mds.1 v2:172.21.15.139:6838/1258854143 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+326 (crc 0 0 0) 0x55cffd159e60 con 0x55cffd27dd00 2023-07-14T19:30:15.705 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:15.695+0000 7f49ad7fa700 10 client.15246 handle_command_reply: tid=0 2023-07-14T19:30:15.705 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:15.695+0000 7f49d812db80 1 client.15246 shutdown 2023-07-14T19:30:15.705 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:15.695+0000 7f49d812db80 1 -- 172.21.15.98:0/300440339 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55cffd20ff10 msgr2=0x55cffd21bf90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:30:15.705 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:15.695+0000 7f49d812db80 1 --2- 172.21.15.98:0/300440339 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55cffd20ff10 0x55cffd21bf90 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x7f49c000b540 tx=0x7f49c0095f80 comp rx=0 tx=0).stop 2023-07-14T19:30:15.706 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:15.695+0000 7f49d812db80 1 -- 172.21.15.98:0/300440339 shutdown_connections 2023-07-14T19:30:15.706 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:15.695+0000 7f49d812db80 1 --2- 172.21.15.98:0/300440339 >> [v2:172.21.15.139:6838/1258854143,v1:172.21.15.139:6839/1258854143] conn(0x55cffd27dd00 0x55cffd27a710 crc :-1 s=READY pgs=20 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:30:15.706 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:15.695+0000 7f49d812db80 1 --2- 172.21.15.98:0/300440339 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x55cffd211460 0x55cffd21f4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:30:15.706 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:15.695+0000 7f49d812db80 1 --2- 172.21.15.98:0/300440339 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55cffd20ff10 0x55cffd21bf90 unknown :-1 s=CLOSED pgs=382 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:30:15.706 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:15.695+0000 7f49ad7fa700 0 client.15246 ms_handle_reset on v2:172.21.15.139:6838/1258854143 2023-07-14T19:30:15.706 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:15.695+0000 7f49d812db80 1 --2- 172.21.15.98:0/300440339 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x55cffd210860 0x55cffd21c4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:30:15.706 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:15.695+0000 7f49d812db80 1 -- 172.21.15.98:0/300440339 >> 172.21.15.98:0/300440339 conn(0x55cffd205630 msgr2=0x55cffd217840 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:30:15.707 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:15.695+0000 7f49d812db80 1 -- 172.21.15.98:0/300440339 shutdown_connections 2023-07-14T19:30:15.707 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:15.695+0000 7f49d812db80 1 -- 172.21.15.98:0/300440339 wait complete. 2023-07-14T19:30:15.707 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:15.695+0000 7f49d812db80 20 client.15246 trim_cache size 0 max 16384 2023-07-14T19:30:15.707 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2023-07-14T19:30:15.707 INFO:teuthology.orchestra.run.smithi098.stdout: "cluster_fsid": "9adea648-227b-11ee-9b33-001a4aab830c", 2023-07-14T19:30:15.708 INFO:teuthology.orchestra.run.smithi098.stdout: "whoami": 1, 2023-07-14T19:30:15.708 INFO:teuthology.orchestra.run.smithi098.stdout: "id": 24493, 2023-07-14T19:30:15.708 INFO:teuthology.orchestra.run.smithi098.stdout: "want_state": "up:active", 2023-07-14T19:30:15.708 INFO:teuthology.orchestra.run.smithi098.stdout: "state": "up:active", 2023-07-14T19:30:15.708 INFO:teuthology.orchestra.run.smithi098.stdout: "fs_name": "cephfs", 2023-07-14T19:30:15.708 INFO:teuthology.orchestra.run.smithi098.stdout: "rank_uptime": 71.035944891, 2023-07-14T19:30:15.708 INFO:teuthology.orchestra.run.smithi098.stdout: "mdsmap_epoch": 22, 2023-07-14T19:30:15.708 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap_epoch": 85, 2023-07-14T19:30:15.709 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap_epoch_barrier": 83, 2023-07-14T19:30:15.709 INFO:teuthology.orchestra.run.smithi098.stdout: "uptime": 86.960554408999997 2023-07-14T19:30:15.709 INFO:teuthology.orchestra.run.smithi098.stdout:} 2023-07-14T19:30:15.710 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:15.696+0000 7f49cbf95700 1 -- 172.21.15.98:0/1921696566 >> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] conn(0x7f49b00a11e0 msgr2=0x7f49b00a3690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:30:15.710 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:15.696+0000 7f49cbf95700 1 --2- 172.21.15.98:0/1921696566 >> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] conn(0x7f49b00a11e0 0x7f49b00a3690 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f49b4006010 tx=0x7f49b400b540 comp rx=0 tx=0).stop 2023-07-14T19:30:15.710 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:15.696+0000 7f49cbf95700 1 -- 172.21.15.98:0/1921696566 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f49c4100750 msgr2=0x7f49c4197bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:30:15.710 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:15.696+0000 7f49cbf95700 1 --2- 172.21.15.98:0/1921696566 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f49c4100750 0x7f49c4197bc0 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7f49c000d950 tx=0x7f49c000dc60 comp rx=0 tx=0).stop 2023-07-14T19:30:15.711 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:15.696+0000 7f49cbf95700 1 -- 172.21.15.98:0/1921696566 shutdown_connections 2023-07-14T19:30:15.711 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:15.696+0000 7f49cbf95700 1 --2- 172.21.15.98:0/1921696566 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f49c4105990 0x7f49c419f170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:30:15.711 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:15.696+0000 7f49cbf95700 1 --2- 172.21.15.98:0/1921696566 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f49c4100750 0x7f49c4197bc0 unknown :-1 s=CLOSED pgs=378 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:30:15.711 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:15.696+0000 7f49cbf95700 1 --2- 172.21.15.98:0/1921696566 >> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] conn(0x7f49b00a11e0 0x7f49b00a3690 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:30:15.711 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:15.696+0000 7f49cbf95700 1 --2- 172.21.15.98:0/1921696566 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f49c4103070 0x7f49c4198100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:30:15.711 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:15.696+0000 7f49cbf95700 1 -- 172.21.15.98:0/1921696566 >> 172.21.15.98:0/1921696566 conn(0x7f49c40fa150 msgr2=0x7f49c40fedd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:30:15.711 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:15.696+0000 7f49cbf95700 1 -- 172.21.15.98:0/1921696566 shutdown_connections 2023-07-14T19:30:15.711 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:15.696+0000 7f49cbf95700 1 -- 172.21.15.98:0/1921696566 wait complete. 2023-07-14T19:30:15.712 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.b status 2023-07-14T19:30:15.836 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:15.836+0000 7f1422bcc700 1 -- 172.21.15.98:0/2350597454 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f141c100690 msgr2=0x7f141c106f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:30:15.837 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:15.836+0000 7f1422bcc700 1 --2- 172.21.15.98:0/2350597454 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f141c100690 0x7f141c106f50 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0x7f141400b0a0 tx=0x7f141400b3b0 comp rx=0 tx=0).stop 2023-07-14T19:30:15.837 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:15.836+0000 7f1422bcc700 1 -- 172.21.15.98:0/2350597454 shutdown_connections 2023-07-14T19:30:15.837 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:15.836+0000 7f1422bcc700 1 --2- 172.21.15.98:0/2350597454 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f141c0feb10 0x7f141c0fef20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:30:15.837 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:15.836+0000 7f1422bcc700 1 --2- 172.21.15.98:0/2350597454 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f141c100690 0x7f141c106f50 unknown :-1 s=CLOSED pgs=383 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:30:15.837 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:15.836+0000 7f1422bcc700 1 --2- 172.21.15.98:0/2350597454 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f141c0ffd00 0x7f141c100150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:30:15.837 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:15.836+0000 7f1422bcc700 1 -- 172.21.15.98:0/2350597454 >> 172.21.15.98:0/2350597454 conn(0x7f141c0fa130 msgr2=0x7f141c0fc560 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:30:15.837 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:15.836+0000 7f1422bcc700 1 -- 172.21.15.98:0/2350597454 shutdown_connections 2023-07-14T19:30:15.838 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:15.836+0000 7f1422bcc700 1 -- 172.21.15.98:0/2350597454 wait complete. 2023-07-14T19:30:15.838 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:15.836+0000 7f1422bcc700 1 Processor -- start 2023-07-14T19:30:15.838 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:15.836+0000 7f1422bcc700 1 -- start start 2023-07-14T19:30:15.839 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:15.836+0000 7f1422bcc700 1 --2- >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f141c0feb10 0x7f141c102aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:30:15.840 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:15.836+0000 7f1422bcc700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f141c0ffd00 0x7f141c1010e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:30:15.840 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:15.836+0000 7f1422bcc700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f141c100690 0x7f141c101620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:30:15.840 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:15.836+0000 7f1422bcc700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f141c101dc0 con 0x7f141c100690 2023-07-14T19:30:15.840 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:15.836+0000 7f1422bcc700 1 -- --> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f141c101f30 con 0x7f141c0feb10 2023-07-14T19:30:15.840 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:15.836+0000 7f1422bcc700 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f141c19fb00 con 0x7f141c0ffd00 2023-07-14T19:30:15.840 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:15.837+0000 7f1421169700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f141c100690 0x7f141c101620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:30:15.840 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:15.837+0000 7f1421169700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f141c100690 0x7f141c101620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:46928/0 (socket says 172.21.15.98:46928) 2023-07-14T19:30:15.841 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:15.837+0000 7f1421169700 1 -- 172.21.15.98:0/2136988652 learned_addr learned my addr 172.21.15.98:0/2136988652 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-07-14T19:30:15.841 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:15.837+0000 7f1420968700 1 --2- 172.21.15.98:0/2136988652 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f141c0feb10 0x7f141c102aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:30:15.841 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:15.837+0000 7f141bfff700 1 --2- 172.21.15.98:0/2136988652 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f141c0ffd00 0x7f141c1010e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:30:15.841 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:15.837+0000 7f1421169700 1 -- 172.21.15.98:0/2136988652 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f141c0ffd00 msgr2=0x7f141c1010e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:30:15.841 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:15.837+0000 7f1421169700 1 --2- 172.21.15.98:0/2136988652 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f141c0ffd00 0x7f141c1010e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:30:15.841 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:15.837+0000 7f1421169700 1 -- 172.21.15.98:0/2136988652 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f141c0feb10 msgr2=0x7f141c102aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:30:15.841 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:15.837+0000 7f1421169700 1 --2- 172.21.15.98:0/2136988652 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f141c0feb10 0x7f141c102aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:30:15.841 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:15.837+0000 7f1421169700 1 -- 172.21.15.98:0/2136988652 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1414009d00 con 0x7f141c100690 2023-07-14T19:30:15.842 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:15.837+0000 7f1421169700 1 --2- 172.21.15.98:0/2136988652 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f141c100690 0x7f141c101620 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7f141400d010 tx=0x7f1414004c80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:30:15.842 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:15.837+0000 7f1419ffb700 1 -- 172.21.15.98:0/2136988652 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f141400e040 con 0x7f141c100690 2023-07-14T19:30:15.842 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:15.837+0000 7f1419ffb700 1 -- 172.21.15.98:0/2136988652 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f1414004e30 con 0x7f141c100690 2023-07-14T19:30:15.842 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:15.837+0000 7f1422bcc700 1 -- 172.21.15.98:0/2136988652 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f141c19fd80 con 0x7f141c100690 2023-07-14T19:30:15.842 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:15.837+0000 7f1419ffb700 1 -- 172.21.15.98:0/2136988652 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1414004020 con 0x7f141c100690 2023-07-14T19:30:15.842 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:15.837+0000 7f1422bcc700 1 -- 172.21.15.98:0/2136988652 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f141c1a01b0 con 0x7f141c100690 2023-07-14T19:30:15.842 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:15.839+0000 7f1419ffb700 1 -- 172.21.15.98:0/2136988652 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 17) v1 ==== 135766+0+0 (secure 0 0 0) 0x7f141401d690 con 0x7f141c100690 2023-07-14T19:30:15.847 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:15.840+0000 7f1419ffb700 1 --2- 172.21.15.98:0/2136988652 >> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] conn(0x7f140c0a11e0 0x7f140c0a3690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:30:15.847 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:15.840+0000 7f1419ffb700 1 -- 172.21.15.98:0/2136988652 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 8211+0+0 (secure 0 0 0) 0x7f14140ce4b0 con 0x7f141c100690 2023-07-14T19:30:15.847 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:15.840+0000 7f1420968700 1 --2- 172.21.15.98:0/2136988652 >> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] conn(0x7f140c0a11e0 0x7f140c0a3690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:30:15.847 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:15.840+0000 7f1420968700 1 --2- 172.21.15.98:0/2136988652 >> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] conn(0x7f140c0a11e0 0x7f140c0a3690 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f141c0ffb60 tx=0x7f141000b410 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:30:15.848 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:15.841+0000 7f142ed64b80 1 Processor -- start 2023-07-14T19:30:15.848 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:15.841+0000 7f142ed64b80 1 -- start start 2023-07-14T19:30:15.848 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:15.841+0000 7f142ed64b80 1 --2- >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x558f5f8f8f10 0x558f5f8f9320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:30:15.848 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:15.841+0000 7f142ed64b80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558f5f8f9860 0x558f5f8f9e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:30:15.848 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:15.841+0000 7f142ed64b80 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x558f5f8fa460 0x558f5f8fa8d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:30:15.848 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:15.841+0000 7f142ed64b80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x558f5f7fdc60 con 0x558f5f8f9860 2023-07-14T19:30:15.848 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:15.841+0000 7f142ed64b80 1 -- --> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] -- mon_getmap magic: 0 v1 -- 0x558f5f8509f0 con 0x558f5f8f8f10 2023-07-14T19:30:15.848 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:15.841+0000 7f142ed64b80 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 v1 -- 0x558f5f849e80 con 0x558f5f8fa460 2023-07-14T19:30:15.849 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:15.841+0000 7f141bfff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558f5f8f9860 0x558f5f8f9e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:30:15.849 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:15.841+0000 7f141bfff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558f5f8f9860 0x558f5f8f9e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:46934/0 (socket says 172.21.15.98:46934) 2023-07-14T19:30:15.849 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:15.841+0000 7f141bfff700 1 -- 172.21.15.98:0/3750975343 learned_addr learned my addr 172.21.15.98:0/3750975343 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-07-14T19:30:15.849 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:15.841+0000 7f1421169700 1 --2- 172.21.15.98:0/3750975343 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x558f5f8fa460 0x558f5f8fa8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:30:15.849 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:15.841+0000 7f1420968700 1 --2- 172.21.15.98:0/3750975343 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x558f5f8f8f10 0x558f5f8f9320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:30:15.849 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:15.842+0000 7f141bfff700 1 -- 172.21.15.98:0/3750975343 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x558f5f8fa460 msgr2=0x558f5f8fa8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-14T19:30:15.849 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:15.842+0000 7f141bfff700 1 --2- 172.21.15.98:0/3750975343 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x558f5f8fa460 0x558f5f8fa8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:30:15.850 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:15.842+0000 7f141bfff700 1 -- 172.21.15.98:0/3750975343 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x558f5f8f8f10 msgr2=0x558f5f8f9320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-14T19:30:15.850 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:15.842+0000 7f141bfff700 1 --2- 172.21.15.98:0/3750975343 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x558f5f8f8f10 0x558f5f8f9320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:30:15.850 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:15.842+0000 7f141bfff700 1 -- 172.21.15.98:0/3750975343 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x558f5f677d90 con 0x558f5f8f9860 2023-07-14T19:30:15.850 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:15.842+0000 7f141bfff700 1 --2- 172.21.15.98:0/3750975343 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558f5f8f9860 0x558f5f8f9e80 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 crypto rx=0x7f140800bd70 tx=0x7f14080106b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6257b89150cf3a08 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:30:15.850 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:15.842+0000 7f1406ffd700 1 -- 172.21.15.98:0/3750975343 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1408010ec0 con 0x558f5f8f9860 2023-07-14T19:30:15.850 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:15.842+0000 7f1406ffd700 1 -- 172.21.15.98:0/3750975343 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f1408015070 con 0x558f5f8f9860 2023-07-14T19:30:15.850 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:15.842+0000 7f1406ffd700 1 -- 172.21.15.98:0/3750975343 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1408011800 con 0x558f5f8f9860 2023-07-14T19:30:15.850 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:15.842+0000 7f142ed64b80 1 -- 172.21.15.98:0/3750975343 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558f5f8f9860 msgr2=0x558f5f8f9e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:30:15.851 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:15.842+0000 7f142ed64b80 1 --2- 172.21.15.98:0/3750975343 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558f5f8f9860 0x558f5f8f9e80 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 crypto rx=0x7f140800bd70 tx=0x7f14080106b0 comp rx=0 tx=0).stop 2023-07-14T19:30:15.851 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:15.843+0000 7f142ed64b80 1 -- 172.21.15.98:0/3750975343 shutdown_connections 2023-07-14T19:30:15.851 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:15.843+0000 7f142ed64b80 1 --2- 172.21.15.98:0/3750975343 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x558f5f8fa460 0x558f5f8fa8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:30:15.851 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:15.843+0000 7f142ed64b80 1 --2- 172.21.15.98:0/3750975343 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x558f5f8f8f10 0x558f5f8f9320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:30:15.851 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:15.843+0000 7f142ed64b80 1 --2- 172.21.15.98:0/3750975343 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558f5f8f9860 0x558f5f8f9e80 unknown :-1 s=CLOSED pgs=385 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:30:15.851 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:15.843+0000 7f142ed64b80 1 -- 172.21.15.98:0/3750975343 >> 172.21.15.98:0/3750975343 conn(0x558f5f8ee630 msgr2=0x558f5f8f3980 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:30:15.851 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:15.843+0000 7f142ed64b80 1 -- 172.21.15.98:0/3750975343 shutdown_connections 2023-07-14T19:30:15.851 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:15.843+0000 7f142ed64b80 1 -- 172.21.15.98:0/3750975343 wait complete. 2023-07-14T19:30:15.852 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:15.843+0000 7f142ed64b80 1 -- start start 2023-07-14T19:30:15.852 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:15.843+0000 7f142ed64b80 1 Processor -- start 2023-07-14T19:30:15.852 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:15.843+0000 7f142ed64b80 1 --2- >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x558f5f8f8f10 0x558f5f902950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:30:15.852 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:15.843+0000 7f142ed64b80 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x558f5f8f9860 0x558f5f902e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:30:15.852 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:15.843+0000 7f142ed64b80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558f5f8fa460 0x558f5f909ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:30:15.852 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:15.843+0000 7f142ed64b80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x558f5f7e9a70 con 0x558f5f8fa460 2023-07-14T19:30:15.852 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:15.843+0000 7f142ed64b80 1 -- --> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] -- mon_getmap magic: 0 v1 -- 0x558f5f7e9770 con 0x558f5f8f8f10 2023-07-14T19:30:15.853 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:15.843+0000 7f142ed64b80 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 v1 -- 0x558f5f6cd8d0 con 0x558f5f8f9860 2023-07-14T19:30:15.853 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:15.843+0000 7f142ed64b80 10 client.0 _ll_register_callbacks cb 0x558f5f7e7710 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-07-14T19:30:15.853 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:15.843+0000 7f1421169700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558f5f8fa460 0x558f5f909ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:30:15.853 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:15.843+0000 7f1421169700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558f5f8fa460 0x558f5f909ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:46942/0 (socket says 172.21.15.98:46942) 2023-07-14T19:30:15.853 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:15.843+0000 7f1421169700 1 -- 172.21.15.98:0/2703811719 learned_addr learned my addr 172.21.15.98:0/2703811719 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-07-14T19:30:15.853 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:15.844+0000 7f141bfff700 1 --2- 172.21.15.98:0/2703811719 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x558f5f8f9860 0x558f5f902e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:30:15.853 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:15.844+0000 7f1420968700 1 --2- 172.21.15.98:0/2703811719 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x558f5f8f8f10 0x558f5f902950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:30:15.854 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:15.844+0000 7f1421169700 1 -- 172.21.15.98:0/2703811719 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x558f5f8f9860 msgr2=0x558f5f902e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-14T19:30:15.854 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:15.844+0000 7f1421169700 1 --2- 172.21.15.98:0/2703811719 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x558f5f8f9860 0x558f5f902e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:30:15.854 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:15.844+0000 7f1421169700 1 -- 172.21.15.98:0/2703811719 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x558f5f8f8f10 msgr2=0x558f5f902950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-14T19:30:15.854 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:15.844+0000 7f1421169700 1 --2- 172.21.15.98:0/2703811719 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x558f5f8f8f10 0x558f5f902950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:30:15.854 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:15.844+0000 7f1421169700 1 -- 172.21.15.98:0/2703811719 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f140800b9a0 con 0x558f5f8fa460 2023-07-14T19:30:15.854 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:15.844+0000 7f141bfff700 1 --2- 172.21.15.98:0/2703811719 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x558f5f8f9860 0x558f5f902e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-07-14T19:30:15.854 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:15.844+0000 7f1421169700 1 --2- 172.21.15.98:0/2703811719 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558f5f8fa460 0x558f5f909ed0 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7f1414097d00 tx=0x7f141409b340 comp rx=0 tx=0).ready entity=mon.0 client_cookie=65e0fb0f1382f015 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:30:15.854 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:15.844+0000 7f13fbfff700 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2023-07-14T19:30:15.855 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:15.844+0000 7f13fbfff700 1 -- 172.21.15.98:0/2703811719 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f14140bb270 con 0x558f5f8fa460 2023-07-14T19:30:15.855 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:15.844+0000 7f13fbfff700 1 -- 172.21.15.98:0/2703811719 <== mon.0 v2:172.21.15.98:3300/0 2 ==== osd_map(85..85 src has 1..85) v4 ==== 8211+0+0 (secure 0 0 0) 0x7f141409b800 con 0x558f5f8fa460 2023-07-14T19:30:15.855 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:15.844+0000 7f142ed64b80 1 -- 172.21.15.98:0/2703811719 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x558f5f7a34a0 con 0x558f5f8fa460 2023-07-14T19:30:15.855 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:15.844+0000 7f13fbfff700 1 -- 172.21.15.98:0/2703811719 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=22) v2 ==== 24+0+0 (secure 0 0 0) 0x7f141409f830 con 0x558f5f8fa460 2023-07-14T19:30:15.855 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:15.844+0000 7f142ed64b80 10 client.15258 fetch_fsmap learned FSMap version 22 2023-07-14T19:30:15.855 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:15.844+0000 7f142ed64b80 1 -- 172.21.15.98:0/2703811719 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=22,monmap=4+}) v3 -- 0x558f5f660c10 con 0x558f5f8fa460 2023-07-14T19:30:15.855 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:15.844+0000 7f13fbfff700 1 -- 172.21.15.98:0/2703811719 <== mon.0 v2:172.21.15.98:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x558f5f7a34a0 con 0x558f5f8fa460 2023-07-14T19:30:15.855 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:15.844+0000 7f13fbfff700 1 -- 172.21.15.98:0/2703811719 <== mon.0 v2:172.21.15.98:3300/0 5 ==== fsmap(e 22) v1 ==== 7462+0+0 (secure 0 0 0) 0x7f14140a3730 con 0x558f5f8fa460 2023-07-14T19:30:15.856 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:15.844+0000 7f142ed64b80 10 client.15258 fetch_fsmap finished waiting for FSMap version 22 2023-07-14T19:30:15.856 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:15.844+0000 7f142ed64b80 10 client.15258 resolve_mds: resolved name 'b' to daemon mds.b 2023-07-14T19:30:15.856 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:15.844+0000 7f142ed64b80 20 client.15258 populate_metadata read hostname 'smithi098' 2023-07-14T19:30:15.856 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:15.844+0000 7f142ed64b80 1 --2- 172.21.15.98:0/2703811719 >> [v2:172.21.15.101:6838/48393381,v1:172.21.15.101:6839/48393381] conn(0x558f5f90d940 0x558f5f90ce00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:30:15.856 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:15.844+0000 7f142ed64b80 4 client.15258 mds_command: new command op to 24521 tid=0[{"prefix": "get_command_descriptions"}] 2023-07-14T19:30:15.856 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:15.844+0000 7f142ed64b80 1 -- 172.21.15.98:0/2703811719 --> [v2:172.21.15.101:6838/48393381,v1:172.21.15.101:6839/48393381] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x558f5f8c7d50 con 0x558f5f90d940 2023-07-14T19:30:15.856 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:15.845+0000 7f1420968700 1 --2- 172.21.15.98:0/2703811719 >> [v2:172.21.15.101:6838/48393381,v1:172.21.15.101:6839/48393381] conn(0x558f5f90d940 0x558f5f90ce00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:30:15.857 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:15.845+0000 7f13fbfff700 10 client.15258 ms_handle_connect on v2:172.21.15.101:6838/48393381 2023-07-14T19:30:15.857 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:15.845+0000 7f1420968700 1 --2- 172.21.15.98:0/2703811719 >> [v2:172.21.15.101:6838/48393381,v1:172.21.15.101:6839/48393381] conn(0x558f5f90d940 0x558f5f90ce00 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6272662aff5b36a8 server_cookie=307075bb0cc16e0a in_seq=0 out_seq=0 2023-07-14T19:30:15.857 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:15.846+0000 7f13fbfff700 1 -- 172.21.15.98:0/2703811719 <== mds.0 v2:172.21.15.101:6838/48393381 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18017 (crc 0 0 0) 0x558f5f8c7d50 con 0x558f5f90d940 2023-07-14T19:30:15.857 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:15.846+0000 7f13fbfff700 10 client.15258 handle_command_reply: tid=0 2023-07-14T19:30:15.857 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:15.846+0000 7f142ed64b80 1 client.15258 shutdown 2023-07-14T19:30:15.857 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:15.846+0000 7f142ed64b80 1 -- 172.21.15.98:0/2703811719 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558f5f8fa460 msgr2=0x558f5f909ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:30:15.857 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:15.846+0000 7f142ed64b80 1 --2- 172.21.15.98:0/2703811719 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558f5f8fa460 0x558f5f909ed0 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7f1414097d00 tx=0x7f141409b340 comp rx=0 tx=0).stop 2023-07-14T19:30:15.857 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:15.846+0000 7f142ed64b80 1 -- 172.21.15.98:0/2703811719 shutdown_connections 2023-07-14T19:30:15.858 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:15.846+0000 7f142ed64b80 1 --2- 172.21.15.98:0/2703811719 >> [v2:172.21.15.101:6838/48393381,v1:172.21.15.101:6839/48393381] conn(0x558f5f90d940 0x558f5f90ce00 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:30:15.858 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:15.846+0000 7f142ed64b80 1 --2- 172.21.15.98:0/2703811719 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558f5f8fa460 0x558f5f909ed0 unknown :-1 s=CLOSED pgs=386 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:30:15.858 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:15.846+0000 7f142ed64b80 1 --2- 172.21.15.98:0/2703811719 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x558f5f8f8f10 0x558f5f902950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:30:15.858 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:15.846+0000 7f13fbfff700 0 client.15258 ms_handle_reset on v2:172.21.15.101:6838/48393381 2023-07-14T19:30:15.858 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:15.846+0000 7f142ed64b80 1 --2- 172.21.15.98:0/2703811719 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x558f5f8f9860 0x558f5f902e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:30:15.858 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:15.846+0000 7f142ed64b80 1 -- 172.21.15.98:0/2703811719 >> 172.21.15.98:0/2703811719 conn(0x558f5f8ee630 msgr2=0x558f5f8f2600 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:30:15.858 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:15.846+0000 7f142ed64b80 1 -- 172.21.15.98:0/2703811719 shutdown_connections 2023-07-14T19:30:15.859 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:15.846+0000 7f142ed64b80 1 -- 172.21.15.98:0/2703811719 wait complete. 2023-07-14T19:30:15.859 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:15.846+0000 7f142ed64b80 20 client.15258 trim_cache size 0 max 16384 2023-07-14T19:30:15.865 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:15.863+0000 7f142ed64b80 1 Processor -- start 2023-07-14T19:30:15.865 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:15.863+0000 7f142ed64b80 1 -- start start 2023-07-14T19:30:15.865 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:15.864+0000 7f142ed64b80 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x558f5f8f8f10 0x558f5f90ce00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:30:15.865 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:15.864+0000 7f142ed64b80 1 --2- >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x558f5f8f9860 0x558f5f909e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:30:15.865 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:15.864+0000 7f142ed64b80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558f5f8fa460 0x558f5f8fd380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:30:15.866 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:15.864+0000 7f142ed64b80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x558f5f8f4e60 con 0x558f5f8fa460 2023-07-14T19:30:15.866 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:15.864+0000 7f142ed64b80 1 -- --> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] -- mon_getmap magic: 0 v1 -- 0x558f5f8f06d0 con 0x558f5f8f9860 2023-07-14T19:30:15.866 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:15.864+0000 7f142ed64b80 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 v1 -- 0x558f5f7a34a0 con 0x558f5f8f8f10 2023-07-14T19:30:15.866 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:15.864+0000 7f1421169700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558f5f8fa460 0x558f5f8fd380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:30:15.866 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:15.864+0000 7f1421169700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558f5f8fa460 0x558f5f8fd380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:46950/0 (socket says 172.21.15.98:46950) 2023-07-14T19:30:15.866 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:15.864+0000 7f1421169700 1 -- 172.21.15.98:0/1622685491 learned_addr learned my addr 172.21.15.98:0/1622685491 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-07-14T19:30:15.866 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:15.864+0000 7f141bfff700 1 --2- 172.21.15.98:0/1622685491 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x558f5f8f9860 0x558f5f909e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:30:15.866 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:15.864+0000 7f1420968700 1 --2- 172.21.15.98:0/1622685491 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x558f5f8f8f10 0x558f5f90ce00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:30:15.867 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:15.864+0000 7f1421169700 1 -- 172.21.15.98:0/1622685491 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x558f5f8f8f10 msgr2=0x558f5f90ce00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-14T19:30:15.867 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:15.864+0000 7f1421169700 1 --2- 172.21.15.98:0/1622685491 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x558f5f8f8f10 0x558f5f90ce00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:30:15.867 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:15.864+0000 7f1421169700 1 -- 172.21.15.98:0/1622685491 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x558f5f8f9860 msgr2=0x558f5f909e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-14T19:30:15.867 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:15.864+0000 7f1421169700 1 --2- 172.21.15.98:0/1622685491 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x558f5f8f9860 0x558f5f909e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:30:15.867 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:15.864+0000 7f1421169700 1 -- 172.21.15.98:0/1622685491 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x558f5f69e200 con 0x558f5f8fa460 2023-07-14T19:30:15.867 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:15.864+0000 7f1420968700 1 --2- 172.21.15.98:0/1622685491 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x558f5f8f8f10 0x558f5f90ce00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-07-14T19:30:15.867 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:15.864+0000 7f141bfff700 1 --2- 172.21.15.98:0/1622685491 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x558f5f8f9860 0x558f5f909e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-07-14T19:30:15.868 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:15.864+0000 7f1421169700 1 --2- 172.21.15.98:0/1622685491 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558f5f8fa460 0x558f5f8fd380 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 crypto rx=0x7f1414009fd0 tx=0x7f1414013fb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7addadfef41f8efc server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:30:15.868 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:15.864+0000 7f14077fe700 1 -- 172.21.15.98:0/1622685491 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f141409acf0 con 0x558f5f8fa460 2023-07-14T19:30:15.868 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:15.864+0000 7f14077fe700 1 -- 172.21.15.98:0/1622685491 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f141409b800 con 0x558f5f8fa460 2023-07-14T19:30:15.868 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:15.864+0000 7f14077fe700 1 -- 172.21.15.98:0/1622685491 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f14140a3850 con 0x558f5f8fa460 2023-07-14T19:30:15.871 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:15.865+0000 7f142ed64b80 1 -- 172.21.15.98:0/1622685491 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558f5f8fa460 msgr2=0x558f5f8fd380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:30:15.871 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:15.865+0000 7f142ed64b80 1 --2- 172.21.15.98:0/1622685491 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558f5f8fa460 0x558f5f8fd380 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 crypto rx=0x7f1414009fd0 tx=0x7f1414013fb0 comp rx=0 tx=0).stop 2023-07-14T19:30:15.871 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:15.865+0000 7f142ed64b80 1 -- 172.21.15.98:0/1622685491 shutdown_connections 2023-07-14T19:30:15.871 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:15.865+0000 7f142ed64b80 1 --2- 172.21.15.98:0/1622685491 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x558f5f8f8f10 0x558f5f90ce00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:30:15.871 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:15.865+0000 7f142ed64b80 1 --2- 172.21.15.98:0/1622685491 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558f5f8fa460 0x558f5f8fd380 unknown :-1 s=CLOSED pgs=387 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:30:15.872 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:15.865+0000 7f142ed64b80 1 --2- 172.21.15.98:0/1622685491 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x558f5f8f9860 0x558f5f909e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:30:15.872 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:15.865+0000 7f142ed64b80 1 -- 172.21.15.98:0/1622685491 >> 172.21.15.98:0/1622685491 conn(0x558f5f8ee630 msgr2=0x558f5f900850 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:30:15.872 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:15.865+0000 7f142ed64b80 1 -- 172.21.15.98:0/1622685491 shutdown_connections 2023-07-14T19:30:15.872 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:15.865+0000 7f142ed64b80 1 -- 172.21.15.98:0/1622685491 wait complete. 2023-07-14T19:30:15.872 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:15.865+0000 7f142ed64b80 1 -- start start 2023-07-14T19:30:15.872 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:15.865+0000 7f142ed64b80 1 Processor -- start 2023-07-14T19:30:15.872 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:15.866+0000 7f142ed64b80 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x558f5f8f8f10 0x558f5f905100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:30:15.873 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:15.866+0000 7f142ed64b80 1 --2- >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x558f5f8f9860 0x558f5f905640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:30:15.873 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:15.866+0000 7f142ed64b80 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558f5f8fa460 0x558f5f9615a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:30:15.873 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:15.866+0000 7f142ed64b80 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x558f5f961ae0 con 0x558f5f8fa460 2023-07-14T19:30:15.873 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:15.866+0000 7f142ed64b80 1 -- --> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] -- mon_getmap magic: 0 v1 -- 0x558f5f961c20 con 0x558f5f8f9860 2023-07-14T19:30:15.873 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:15.866+0000 7f142ed64b80 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 v1 -- 0x558f5f961d90 con 0x558f5f8f8f10 2023-07-14T19:30:15.873 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:15.866+0000 7f142ed64b80 10 client.0 _ll_register_callbacks cb 0x7f1414097b40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-07-14T19:30:15.873 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:15.866+0000 7f1421169700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558f5f8fa460 0x558f5f9615a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:30:15.874 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:15.866+0000 7f1421169700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558f5f8fa460 0x558f5f9615a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:46956/0 (socket says 172.21.15.98:46956) 2023-07-14T19:30:15.874 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:15.866+0000 7f1421169700 1 -- 172.21.15.98:0/1486504220 learned_addr learned my addr 172.21.15.98:0/1486504220 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-07-14T19:30:15.874 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:15.866+0000 7f141bfff700 1 --2- 172.21.15.98:0/1486504220 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x558f5f8f9860 0x558f5f905640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:30:15.874 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:15.866+0000 7f1420968700 1 --2- 172.21.15.98:0/1486504220 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x558f5f8f8f10 0x558f5f905100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:30:15.874 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:15.866+0000 7f1421169700 1 -- 172.21.15.98:0/1486504220 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x558f5f8f8f10 msgr2=0x558f5f905100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-14T19:30:15.874 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:15.866+0000 7f1421169700 1 --2- 172.21.15.98:0/1486504220 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x558f5f8f8f10 0x558f5f905100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:30:15.874 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:15.866+0000 7f1421169700 1 -- 172.21.15.98:0/1486504220 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x558f5f8f9860 msgr2=0x558f5f905640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-14T19:30:15.874 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:15.866+0000 7f1421169700 1 --2- 172.21.15.98:0/1486504220 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x558f5f8f9860 0x558f5f905640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:30:15.875 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:15.866+0000 7f1421169700 1 -- 172.21.15.98:0/1486504220 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x558f5f660c10 con 0x558f5f8fa460 2023-07-14T19:30:15.875 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:15.866+0000 7f1421169700 1 --2- 172.21.15.98:0/1486504220 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558f5f8fa460 0x558f5f9615a0 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 crypto rx=0x7f141400b830 tx=0x7f141409b340 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5c4d4a9f3b8895e2 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:30:15.875 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:15.866+0000 7f13fbfff700 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2023-07-14T19:30:15.875 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:15.866+0000 7f13fbfff700 1 -- 172.21.15.98:0/1486504220 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f141409acf0 con 0x558f5f8fa460 2023-07-14T19:30:15.875 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:15.866+0000 7f13fbfff700 1 -- 172.21.15.98:0/1486504220 <== mon.0 v2:172.21.15.98:3300/0 2 ==== osd_map(85..85 src has 1..85) v4 ==== 8211+0+0 (secure 0 0 0) 0x7f141409f830 con 0x558f5f8fa460 2023-07-14T19:30:15.875 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:15.866+0000 7f142ed64b80 1 -- 172.21.15.98:0/1486504220 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f1408015070 con 0x558f5f8fa460 2023-07-14T19:30:15.875 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:15.866+0000 7f13fbfff700 1 -- 172.21.15.98:0/1486504220 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=22) v2 ==== 24+0+0 (secure 0 0 0) 0x7f14140a3830 con 0x558f5f8fa460 2023-07-14T19:30:15.875 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:15.867+0000 7f142ed64b80 10 client.15264 fetch_fsmap learned FSMap version 22 2023-07-14T19:30:15.876 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:15.867+0000 7f142ed64b80 1 -- 172.21.15.98:0/1486504220 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=22,monmap=4+}) v3 -- 0x558f5f676260 con 0x558f5f8fa460 2023-07-14T19:30:15.876 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:15.867+0000 7f13fbfff700 1 -- 172.21.15.98:0/1486504220 <== mon.0 v2:172.21.15.98:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f1408015070 con 0x558f5f8fa460 2023-07-14T19:30:15.876 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:15.867+0000 7f13fbfff700 1 -- 172.21.15.98:0/1486504220 <== mon.0 v2:172.21.15.98:3300/0 5 ==== fsmap(e 22) v1 ==== 7462+0+0 (secure 0 0 0) 0x7f1414097040 con 0x558f5f8fa460 2023-07-14T19:30:15.876 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:15.867+0000 7f142ed64b80 10 client.15264 fetch_fsmap finished waiting for FSMap version 22 2023-07-14T19:30:15.876 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:15.867+0000 7f142ed64b80 10 client.15264 resolve_mds: resolved name 'b' to daemon mds.b 2023-07-14T19:30:15.876 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:15.867+0000 7f142ed64b80 20 client.15264 populate_metadata read hostname 'smithi098' 2023-07-14T19:30:15.876 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:15.867+0000 7f142ed64b80 1 --2- 172.21.15.98:0/1486504220 >> [v2:172.21.15.101:6838/48393381,v1:172.21.15.101:6839/48393381] conn(0x558f5f966230 0x558f5f962f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:30:15.877 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:15.867+0000 7f142ed64b80 4 client.15264 mds_command: new command op to 24521 tid=0[{"prefix": "status"}] 2023-07-14T19:30:15.877 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:15.867+0000 7f142ed64b80 1 -- 172.21.15.98:0/1486504220 --> [v2:172.21.15.101:6838/48393381,v1:172.21.15.101:6839/48393381] -- command(tid 0: {"prefix": "status"}) v1 -- 0x558f5f842e60 con 0x558f5f966230 2023-07-14T19:30:15.877 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:15.867+0000 7f1420968700 1 --2- 172.21.15.98:0/1486504220 >> [v2:172.21.15.101:6838/48393381,v1:172.21.15.101:6839/48393381] conn(0x558f5f966230 0x558f5f962f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:30:15.877 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:15.867+0000 7f13fbfff700 10 client.15264 ms_handle_connect on v2:172.21.15.101:6838/48393381 2023-07-14T19:30:15.877 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:15.867+0000 7f1420968700 1 --2- 172.21.15.98:0/1486504220 >> [v2:172.21.15.101:6838/48393381,v1:172.21.15.101:6839/48393381] conn(0x558f5f966230 0x558f5f962f50 crc :-1 s=READY pgs=19 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f488bfaba99ff70a server_cookie=2c6bf6d99687583c in_seq=0 out_seq=0 2023-07-14T19:30:15.877 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:15.868+0000 7f13fbfff700 1 -- 172.21.15.98:0/1486504220 <== mds.0 v2:172.21.15.101:6838/48393381 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+332 (crc 0 0 0) 0x558f5f842e60 con 0x558f5f966230 2023-07-14T19:30:15.877 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:15.868+0000 7f13fbfff700 10 client.15264 handle_command_reply: tid=0 2023-07-14T19:30:15.877 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:15.868+0000 7f142ed64b80 1 client.15264 shutdown 2023-07-14T19:30:15.878 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:15.868+0000 7f142ed64b80 1 -- 172.21.15.98:0/1486504220 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558f5f8fa460 msgr2=0x558f5f9615a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:30:15.878 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:15.868+0000 7f142ed64b80 1 --2- 172.21.15.98:0/1486504220 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558f5f8fa460 0x558f5f9615a0 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 crypto rx=0x7f141400b830 tx=0x7f141409b340 comp rx=0 tx=0).stop 2023-07-14T19:30:15.878 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:15.868+0000 7f142ed64b80 1 -- 172.21.15.98:0/1486504220 shutdown_connections 2023-07-14T19:30:15.878 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:15.868+0000 7f142ed64b80 1 --2- 172.21.15.98:0/1486504220 >> [v2:172.21.15.101:6838/48393381,v1:172.21.15.101:6839/48393381] conn(0x558f5f966230 0x558f5f962f50 crc :-1 s=READY pgs=19 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:30:15.878 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:15.868+0000 7f142ed64b80 1 --2- 172.21.15.98:0/1486504220 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x558f5f8f8f10 0x558f5f905100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:30:15.878 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:15.868+0000 7f13fbfff700 0 client.15264 ms_handle_reset on v2:172.21.15.101:6838/48393381 2023-07-14T19:30:15.878 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:15.868+0000 7f142ed64b80 1 --2- 172.21.15.98:0/1486504220 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558f5f8fa460 0x558f5f9615a0 unknown :-1 s=CLOSED pgs=388 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:30:15.879 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:15.868+0000 7f142ed64b80 1 --2- 172.21.15.98:0/1486504220 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x558f5f8f9860 0x558f5f905640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:30:15.879 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:15.868+0000 7f142ed64b80 1 -- 172.21.15.98:0/1486504220 >> 172.21.15.98:0/1486504220 conn(0x558f5f8ee630 msgr2=0x558f5f90ce00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:30:15.879 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:15.868+0000 7f142ed64b80 1 -- 172.21.15.98:0/1486504220 shutdown_connections 2023-07-14T19:30:15.879 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:15.868+0000 7f142ed64b80 1 -- 172.21.15.98:0/1486504220 wait complete. 2023-07-14T19:30:15.879 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:15.868+0000 7f142ed64b80 20 client.15264 trim_cache size 0 max 16384 2023-07-14T19:30:15.879 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2023-07-14T19:30:15.880 INFO:teuthology.orchestra.run.smithi098.stdout: "cluster_fsid": "9adea648-227b-11ee-9b33-001a4aab830c", 2023-07-14T19:30:15.880 INFO:teuthology.orchestra.run.smithi098.stdout: "whoami": 0, 2023-07-14T19:30:15.880 INFO:teuthology.orchestra.run.smithi098.stdout: "id": 24521, 2023-07-14T19:30:15.880 INFO:teuthology.orchestra.run.smithi098.stdout: "want_state": "up:active", 2023-07-14T19:30:15.880 INFO:teuthology.orchestra.run.smithi098.stdout: "state": "up:active", 2023-07-14T19:30:15.880 INFO:teuthology.orchestra.run.smithi098.stdout: "fs_name": "cephfs", 2023-07-14T19:30:15.880 INFO:teuthology.orchestra.run.smithi098.stdout: "rank_uptime": 74.237420083000004, 2023-07-14T19:30:15.881 INFO:teuthology.orchestra.run.smithi098.stdout: "mdsmap_epoch": 22, 2023-07-14T19:30:15.881 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap_epoch": 85, 2023-07-14T19:30:15.881 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap_epoch_barrier": 83, 2023-07-14T19:30:15.881 INFO:teuthology.orchestra.run.smithi098.stdout: "uptime": 80.575278510999993 2023-07-14T19:30:15.881 INFO:teuthology.orchestra.run.smithi098.stdout:} 2023-07-14T19:30:15.882 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:15.868+0000 7f1422bcc700 1 -- 172.21.15.98:0/2136988652 >> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] conn(0x7f140c0a11e0 msgr2=0x7f140c0a3690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:30:15.882 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:15.868+0000 7f1422bcc700 1 --2- 172.21.15.98:0/2136988652 >> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] conn(0x7f140c0a11e0 0x7f140c0a3690 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f141c0ffb60 tx=0x7f141000b410 comp rx=0 tx=0).stop 2023-07-14T19:30:15.882 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:15.868+0000 7f1422bcc700 1 -- 172.21.15.98:0/2136988652 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f141c100690 msgr2=0x7f141c101620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:30:15.883 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:15.868+0000 7f1422bcc700 1 --2- 172.21.15.98:0/2136988652 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f141c100690 0x7f141c101620 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7f141400d010 tx=0x7f1414004c80 comp rx=0 tx=0).stop 2023-07-14T19:30:15.883 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:15.869+0000 7f1422bcc700 1 -- 172.21.15.98:0/2136988652 shutdown_connections 2023-07-14T19:30:15.883 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:15.869+0000 7f1422bcc700 1 --2- 172.21.15.98:0/2136988652 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f141c100690 0x7f141c101620 unknown :-1 s=CLOSED pgs=384 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:30:15.883 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:15.869+0000 7f1422bcc700 1 --2- 172.21.15.98:0/2136988652 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f141c0feb10 0x7f141c102aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:30:15.883 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:15.869+0000 7f1422bcc700 1 --2- 172.21.15.98:0/2136988652 >> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] conn(0x7f140c0a11e0 0x7f140c0a3690 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:30:15.883 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:15.869+0000 7f1422bcc700 1 --2- 172.21.15.98:0/2136988652 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f141c0ffd00 0x7f141c1010e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:30:15.883 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:15.869+0000 7f1422bcc700 1 -- 172.21.15.98:0/2136988652 >> 172.21.15.98:0/2136988652 conn(0x7f141c0fa130 msgr2=0x7f141c0677e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:30:15.884 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:15.869+0000 7f1422bcc700 1 -- 172.21.15.98:0/2136988652 shutdown_connections 2023-07-14T19:30:15.884 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:15.869+0000 7f1422bcc700 1 -- 172.21.15.98:0/2136988652 wait complete. 2023-07-14T19:30:15.884 INFO:teuthology.run_tasks:Running task kclient... 2023-07-14T19:30:15.908 INFO:tasks.kclient:Mounting kernel clients... 2023-07-14T19:30:15.909 INFO:tasks.kclient:config is {'client.0': {}} 2023-07-14T19:30:15.909 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-07-14T19:30:15.909 DEBUG:teuthology.orchestra.run.smithi098:> ip netns list 2023-07-14T19:30:15.926 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-07-14T19:30:15.927 DEBUG:teuthology.orchestra.run.smithi098:> sudo ip link delete ceph-brx 2023-07-14T19:30:15.989 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:30:15 smithi098 ceph-mon[127486]: from='client.? 172.21.15.98:0/852855985' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-07-14T19:30:15.989 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:30:15 smithi098 ceph-mon[127486]: from='client.? 172.21.15.98:0/999693056' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-07-14T19:30:15.994 INFO:teuthology.orchestra.run.smithi098.stderr:Cannot find device "ceph-brx" 2023-07-14T19:30:15.995 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-07-14T19:30:15.996 INFO:tasks.kclient:client.0 config is {'mntopts': ['ms_mode=crc', 'wsync'], 'syntax': 'v2', 'mount_subvol_num': 1} 2023-07-14T19:30:15.996 INFO:tasks.cephfs.mount:cephfs_mntpt = None 2023-07-14T19:30:15.997 INFO:tasks.cephfs.mount:hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2023-07-14T19:30:15.997 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81c9387d2e7576da065747ec9c16d1c7d69016ec shell --fsid 9adea648-227b-11ee-9b33-001a4aab830c -- ceph fs ls 2023-07-14T19:30:16.191 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:30:15 smithi101 ceph-mon[128807]: from='client.? 172.21.15.98:0/852855985' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-07-14T19:30:16.191 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:30:15 smithi101 ceph-mon[128807]: from='client.? 172.21.15.98:0/999693056' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-07-14T19:30:16.220 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:30:15 smithi139 ceph-mon[129293]: from='client.? 172.21.15.98:0/852855985' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-07-14T19:30:16.220 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:30:15 smithi139 ceph-mon[129293]: from='client.? 172.21.15.98:0/999693056' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-07-14T19:30:16.583 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/9adea648-227b-11ee-9b33-001a4aab830c/mon.a/config 2023-07-14T19:30:17.031 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:30:16 smithi098 ceph-mon[127486]: pgmap v292: 97 pgs: 97 active+clean; 634 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s; 12 B/s, 0 objects/s recovering 2023-07-14T19:30:17.191 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:30:16 smithi101 ceph-mon[128807]: pgmap v292: 97 pgs: 97 active+clean; 634 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s; 12 B/s, 0 objects/s recovering 2023-07-14T19:30:17.220 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:30:16 smithi139 ceph-mon[129293]: pgmap v292: 97 pgs: 97 active+clean; 634 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s; 12 B/s, 0 objects/s recovering 2023-07-14T19:30:17.650 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:17.648+0000 7faf7d24f700 1 -- 172.21.15.98:0/2233123777 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7faf78103f10 msgr2=0x7faf7810ba90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:30:17.650 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:17.648+0000 7faf7d24f700 1 --2- 172.21.15.98:0/2233123777 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7faf78103f10 0x7faf7810ba90 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 crypto rx=0x7faf7000b0d0 tx=0x7faf7000b3e0 comp rx=0 tx=0).stop 2023-07-14T19:30:17.650 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:17.648+0000 7faf7d24f700 1 -- 172.21.15.98:0/2233123777 shutdown_connections 2023-07-14T19:30:17.650 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:17.648+0000 7faf7d24f700 1 --2- 172.21.15.98:0/2233123777 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7faf780fee70 0x7faf780ff240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:30:17.651 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:17.648+0000 7faf7d24f700 1 --2- 172.21.15.98:0/2233123777 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7faf78103f10 0x7faf7810ba90 unknown :-1 s=CLOSED pgs=389 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:30:17.651 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:17.648+0000 7faf7d24f700 1 --2- 172.21.15.98:0/2233123777 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7faf780ff810 0x7faf781037e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:30:17.651 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:17.648+0000 7faf7d24f700 1 -- 172.21.15.98:0/2233123777 >> 172.21.15.98:0/2233123777 conn(0x7faf78075d20 msgr2=0x7faf78076120 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:30:17.651 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:17.648+0000 7faf7d24f700 1 -- 172.21.15.98:0/2233123777 shutdown_connections 2023-07-14T19:30:17.651 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:17.648+0000 7faf7d24f700 1 -- 172.21.15.98:0/2233123777 wait complete. 2023-07-14T19:30:17.651 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:17.649+0000 7faf7d24f700 1 Processor -- start 2023-07-14T19:30:17.651 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:17.649+0000 7faf7d24f700 1 -- start start 2023-07-14T19:30:17.651 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:17.649+0000 7faf7d24f700 1 --2- >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7faf780fee70 0x7faf78100880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:30:17.652 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:17.649+0000 7faf7d24f700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7faf780ff810 0x7faf78100dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:30:17.652 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:17.649+0000 7faf7d24f700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7faf78103f10 0x7faf78101340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:30:17.652 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:17.649+0000 7faf7d24f700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faf781a4390 con 0x7faf780ff810 2023-07-14T19:30:17.652 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:17.649+0000 7faf7d24f700 1 -- --> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faf781a4500 con 0x7faf780fee70 2023-07-14T19:30:17.652 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:17.649+0000 7faf7d24f700 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faf781a46c0 con 0x7faf78103f10 2023-07-14T19:30:17.652 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:17.650+0000 7faf6ffff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7faf780ff810 0x7faf78100dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:30:17.653 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:17.650+0000 7faf76d9d700 1 --2- >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7faf780fee70 0x7faf78100880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:30:17.653 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:17.650+0000 7faf6ffff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7faf780ff810 0x7faf78100dc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:46968/0 (socket says 172.21.15.98:46968) 2023-07-14T19:30:17.653 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:17.650+0000 7faf6ffff700 1 -- 172.21.15.98:0/1524318238 learned_addr learned my addr 172.21.15.98:0/1524318238 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-07-14T19:30:17.653 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:17.650+0000 7faf7759e700 1 --2- 172.21.15.98:0/1524318238 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7faf78103f10 0x7faf78101340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:30:17.654 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:17.650+0000 7faf6ffff700 1 -- 172.21.15.98:0/1524318238 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7faf78103f10 msgr2=0x7faf78101340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:30:17.654 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:17.650+0000 7faf6ffff700 1 --2- 172.21.15.98:0/1524318238 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7faf78103f10 0x7faf78101340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:30:17.654 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:17.650+0000 7faf6ffff700 1 -- 172.21.15.98:0/1524318238 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7faf780fee70 msgr2=0x7faf78100880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:30:17.655 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:17.650+0000 7faf6ffff700 1 --2- 172.21.15.98:0/1524318238 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7faf780fee70 0x7faf78100880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:30:17.655 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:17.650+0000 7faf6ffff700 1 -- 172.21.15.98:0/1524318238 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faf70009d00 con 0x7faf780ff810 2023-07-14T19:30:17.655 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:17.650+0000 7faf6ffff700 1 --2- 172.21.15.98:0/1524318238 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7faf780ff810 0x7faf78100dc0 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 crypto rx=0x7faf6000c8f0 tx=0x7faf6000cc00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:30:17.655 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:17.650+0000 7faf74d99700 1 -- 172.21.15.98:0/1524318238 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faf600078d0 con 0x7faf780ff810 2023-07-14T19:30:17.655 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:17.650+0000 7faf7d24f700 1 -- 172.21.15.98:0/1524318238 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faf781a49a0 con 0x7faf780ff810 2023-07-14T19:30:17.655 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:17.650+0000 7faf74d99700 1 -- 172.21.15.98:0/1524318238 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7faf6000f450 con 0x7faf780ff810 2023-07-14T19:30:17.655 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:17.650+0000 7faf74d99700 1 -- 172.21.15.98:0/1524318238 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faf60018660 con 0x7faf780ff810 2023-07-14T19:30:17.656 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:17.650+0000 7faf7d24f700 1 -- 172.21.15.98:0/1524318238 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faf781a4ea0 con 0x7faf780ff810 2023-07-14T19:30:17.656 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:17.651+0000 7faf7d24f700 1 -- 172.21.15.98:0/1524318238 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faf7804e340 con 0x7faf780ff810 2023-07-14T19:30:17.656 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:17.652+0000 7faf74d99700 1 -- 172.21.15.98:0/1524318238 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 17) v1 ==== 135766+0+0 (secure 0 0 0) 0x7faf6000f5c0 con 0x7faf780ff810 2023-07-14T19:30:17.656 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:17.653+0000 7faf74d99700 1 --2- 172.21.15.98:0/1524318238 >> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] conn(0x7faf640a11e0 0x7faf640a3690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:30:17.656 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:17.653+0000 7faf76d9d700 1 --2- 172.21.15.98:0/1524318238 >> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] conn(0x7faf640a11e0 0x7faf640a3690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:30:17.656 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:17.653+0000 7faf74d99700 1 -- 172.21.15.98:0/1524318238 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 8211+0+0 (secure 0 0 0) 0x7faf600cc9a0 con 0x7faf780ff810 2023-07-14T19:30:17.656 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:17.654+0000 7faf76d9d700 1 --2- 172.21.15.98:0/1524318238 >> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] conn(0x7faf640a11e0 0x7faf640a3690 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7faf68009a10 tx=0x7faf68009190 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:30:17.657 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:17.657+0000 7faf74d99700 1 -- 172.21.15.98:0/1524318238 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7faf60096030 con 0x7faf780ff810 2023-07-14T19:30:17.888 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:17.888+0000 7faf7d24f700 1 -- 172.21.15.98:0/1524318238 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7faf78101db0 con 0x7faf780ff810 2023-07-14T19:30:17.889 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:17.888+0000 7faf74d99700 1 -- 172.21.15.98:0/1524318238 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v22) v1 ==== 53+0+73 (secure 0 0 0) 0x7faf60099650 con 0x7faf780ff810 2023-07-14T19:30:17.889 INFO:teuthology.orchestra.run.smithi098.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2023-07-14T19:30:17.890 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:17.890+0000 7faf7d24f700 1 -- 172.21.15.98:0/1524318238 >> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] conn(0x7faf640a11e0 msgr2=0x7faf640a3690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:30:17.891 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:17.890+0000 7faf7d24f700 1 --2- 172.21.15.98:0/1524318238 >> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] conn(0x7faf640a11e0 0x7faf640a3690 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7faf68009a10 tx=0x7faf68009190 comp rx=0 tx=0).stop 2023-07-14T19:30:17.891 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:17.890+0000 7faf7d24f700 1 -- 172.21.15.98:0/1524318238 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7faf780ff810 msgr2=0x7faf78100dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:30:17.891 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:17.890+0000 7faf7d24f700 1 --2- 172.21.15.98:0/1524318238 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7faf780ff810 0x7faf78100dc0 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 crypto rx=0x7faf6000c8f0 tx=0x7faf6000cc00 comp rx=0 tx=0).stop 2023-07-14T19:30:17.891 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:17.890+0000 7faf7d24f700 1 -- 172.21.15.98:0/1524318238 shutdown_connections 2023-07-14T19:30:17.891 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:17.890+0000 7faf7d24f700 1 --2- 172.21.15.98:0/1524318238 >> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] conn(0x7faf640a11e0 0x7faf640a3690 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:30:17.891 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:17.890+0000 7faf7d24f700 1 --2- 172.21.15.98:0/1524318238 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7faf78103f10 0x7faf78101340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:30:17.891 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:17.890+0000 7faf7d24f700 1 --2- 172.21.15.98:0/1524318238 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7faf780fee70 0x7faf78100880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:30:17.892 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:17.890+0000 7faf7d24f700 1 --2- 172.21.15.98:0/1524318238 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7faf780ff810 0x7faf78100dc0 unknown :-1 s=CLOSED pgs=390 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:30:17.892 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:17.890+0000 7faf7d24f700 1 -- 172.21.15.98:0/1524318238 >> 172.21.15.98:0/1524318238 conn(0x7faf78075d20 msgr2=0x7faf781029c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:30:17.892 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:17.890+0000 7faf7d24f700 1 -- 172.21.15.98:0/1524318238 shutdown_connections 2023-07-14T19:30:17.892 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:17.890+0000 7faf7d24f700 1 -- 172.21.15.98:0/1524318238 wait complete. 2023-07-14T19:30:18.227 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2023-07-14T19:30:18.228 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2023-07-14T19:30:18.228 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi098.front.sepia.ceph.com 2023-07-14T19:30:18.228 INFO:tasks.cephfs.mount:self.client.name = client.0 2023-07-14T19:30:18.228 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2023-07-14T19:30:18.229 INFO:tasks.cephfs.mount:self.cephfs_name = None 2023-07-14T19:30:18.229 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2023-07-14T19:30:18.229 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2023-07-14T19:30:18.229 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/home/ubuntu/cephtest/mnt.0' 2023-07-14T19:30:18.229 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-07-14T19:30:18.230 DEBUG:teuthology.orchestra.run.smithi098:> ip addr 2023-07-14T19:30:18.247 INFO:teuthology.orchestra.run.smithi098.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2023-07-14T19:30:18.248 INFO:teuthology.orchestra.run.smithi098.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2023-07-14T19:30:18.248 INFO:teuthology.orchestra.run.smithi098.stdout: inet 127.0.0.1/8 scope host lo 2023-07-14T19:30:18.248 INFO:teuthology.orchestra.run.smithi098.stdout: valid_lft forever preferred_lft forever 2023-07-14T19:30:18.249 INFO:teuthology.orchestra.run.smithi098.stdout: inet6 ::1/128 scope host 2023-07-14T19:30:18.249 INFO:teuthology.orchestra.run.smithi098.stdout: valid_lft forever preferred_lft forever 2023-07-14T19:30:18.249 INFO:teuthology.orchestra.run.smithi098.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-07-14T19:30:18.250 INFO:teuthology.orchestra.run.smithi098.stdout: link/ether 0c:c4:7a:d8:d9:b8 brd ff:ff:ff:ff:ff:ff 2023-07-14T19:30:18.250 INFO:teuthology.orchestra.run.smithi098.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-07-14T19:30:18.250 INFO:teuthology.orchestra.run.smithi098.stdout: link/ether 0c:c4:7a:d8:d9:b9 brd ff:ff:ff:ff:ff:ff 2023-07-14T19:30:18.251 INFO:teuthology.orchestra.run.smithi098.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-07-14T19:30:18.251 INFO:teuthology.orchestra.run.smithi098.stdout: link/ether 0c:c4:7a:8f:d2:6a brd ff:ff:ff:ff:ff:ff 2023-07-14T19:30:18.251 INFO:teuthology.orchestra.run.smithi098.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2023-07-14T19:30:18.251 INFO:teuthology.orchestra.run.smithi098.stdout: link/ether 0c:c4:7a:8f:d2:6b brd ff:ff:ff:ff:ff:ff 2023-07-14T19:30:18.252 INFO:teuthology.orchestra.run.smithi098.stdout: inet 172.21.15.98/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2023-07-14T19:30:18.252 INFO:teuthology.orchestra.run.smithi098.stdout: valid_lft 41222sec preferred_lft 41222sec 2023-07-14T19:30:18.252 INFO:teuthology.orchestra.run.smithi098.stdout: inet6 fe80::d4ed:112f:381a:8754/64 scope link noprefixroute 2023-07-14T19:30:18.252 INFO:teuthology.orchestra.run.smithi098.stdout: valid_lft forever preferred_lft forever 2023-07-14T19:30:18.252 INFO:tasks.cephfs.mount:Setuping the 'ceph-brx' with 192.168.255.254/16 2023-07-14T19:30:18.253 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-07-14T19:30:18.253 DEBUG:teuthology.orchestra.run.smithi098:> (cd / && exec bash -c ' 2023-07-14T19:30:18.253 DEBUG:teuthology.orchestra.run.smithi098:> set -e 2023-07-14T19:30:18.253 DEBUG:teuthology.orchestra.run.smithi098:> sudo ip link add name ceph-brx type bridge 2023-07-14T19:30:18.254 DEBUG:teuthology.orchestra.run.smithi098:> sudo ip addr flush dev ceph-brx 2023-07-14T19:30:18.254 DEBUG:teuthology.orchestra.run.smithi098:> sudo ip link set ceph-brx up 2023-07-14T19:30:18.254 DEBUG:teuthology.orchestra.run.smithi098:> sudo ip addr add 192.168.255.254/16 brd 192.168.255.255 dev ceph-brx 2023-07-14T19:30:18.254 DEBUG:teuthology.orchestra.run.smithi098:> ') 2023-07-14T19:30:18.460 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-07-14T19:30:18.461 DEBUG:teuthology.orchestra.run.smithi098:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2023-07-14T19:30:18.488 INFO:teuthology.orchestra.run.smithi098.stdout:1 2023-07-14T19:30:18.490 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-07-14T19:30:18.490 DEBUG:teuthology.orchestra.run.smithi098:> route 2023-07-14T19:30:18.563 INFO:teuthology.orchestra.run.smithi098.stdout:Kernel IP routing table 2023-07-14T19:30:18.563 INFO:teuthology.orchestra.run.smithi098.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2023-07-14T19:30:18.563 INFO:teuthology.orchestra.run.smithi098.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2023-07-14T19:30:18.563 INFO:teuthology.orchestra.run.smithi098.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2023-07-14T19:30:18.564 INFO:teuthology.orchestra.run.smithi098.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2023-07-14T19:30:18.564 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-07-14T19:30:18.564 DEBUG:teuthology.orchestra.run.smithi098:> (cd / && exec bash -c ' 2023-07-14T19:30:18.565 DEBUG:teuthology.orchestra.run.smithi098:> set -e 2023-07-14T19:30:18.565 DEBUG:teuthology.orchestra.run.smithi098:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2023-07-14T19:30:18.565 DEBUG:teuthology.orchestra.run.smithi098:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2023-07-14T19:30:18.565 DEBUG:teuthology.orchestra.run.smithi098:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2023-07-14T19:30:18.566 DEBUG:teuthology.orchestra.run.smithi098:> ') 2023-07-14T19:30:18.831 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-07-14T19:30:18.832 DEBUG:teuthology.orchestra.run.smithi098:> ip netns list 2023-07-14T19:30:18.849 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-07-14T19:30:18.849 DEBUG:teuthology.orchestra.run.smithi098:> ip netns list-id 2023-07-14T19:30:18.906 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-07-14T19:30:18.906 DEBUG:teuthology.orchestra.run.smithi098:> (cd / && exec bash -c ' 2023-07-14T19:30:18.907 DEBUG:teuthology.orchestra.run.smithi098:> set -e 2023-07-14T19:30:18.907 DEBUG:teuthology.orchestra.run.smithi098:> sudo ip netns add ceph-ns--home-ubuntu-cephtest-mnt.0 2023-07-14T19:30:18.907 DEBUG:teuthology.orchestra.run.smithi098:> sudo ip netns set ceph-ns--home-ubuntu-cephtest-mnt.0 0 2023-07-14T19:30:18.907 DEBUG:teuthology.orchestra.run.smithi098:> ') 2023-07-14T19:30:19.000 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--home-ubuntu-cephtest-mnt.0' with 192.168.0.1/16 2023-07-14T19:30:19.000 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-07-14T19:30:19.001 DEBUG:teuthology.orchestra.run.smithi098:> (cd / && exec bash -c ' 2023-07-14T19:30:19.001 DEBUG:teuthology.orchestra.run.smithi098:> set -e 2023-07-14T19:30:19.001 DEBUG:teuthology.orchestra.run.smithi098:> sudo ip link add veth0 netns ceph-ns--home-ubuntu-cephtest-mnt.0 type veth peer name brx.0 2023-07-14T19:30:19.001 DEBUG:teuthology.orchestra.run.smithi098:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip addr add 192.168.0.1/16 brd 192.168.255.255 dev veth0 2023-07-14T19:30:19.002 DEBUG:teuthology.orchestra.run.smithi098:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip link set veth0 up 2023-07-14T19:30:19.002 DEBUG:teuthology.orchestra.run.smithi098:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip link set lo up 2023-07-14T19:30:19.002 DEBUG:teuthology.orchestra.run.smithi098:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip route add default via 192.168.255.254 2023-07-14T19:30:19.002 DEBUG:teuthology.orchestra.run.smithi098:> ') 2023-07-14T19:30:19.031 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:30:18 smithi098 ceph-mon[127486]: pgmap v293: 97 pgs: 97 active+clean; 634 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 5 op/s; 1 B/s, 0 objects/s recovering 2023-07-14T19:30:19.031 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:30:18 smithi098 ceph-mon[127486]: from='client.? 172.21.15.98:0/1524318238' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2023-07-14T19:30:19.191 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:30:18 smithi101 ceph-mon[128807]: pgmap v293: 97 pgs: 97 active+clean; 634 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 5 op/s; 1 B/s, 0 objects/s recovering 2023-07-14T19:30:19.191 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:30:18 smithi101 ceph-mon[128807]: from='client.? 172.21.15.98:0/1524318238' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2023-07-14T19:30:19.217 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-07-14T19:30:19.217 DEBUG:teuthology.orchestra.run.smithi098:> (cd / && exec bash -c ' 2023-07-14T19:30:19.217 DEBUG:teuthology.orchestra.run.smithi098:> set -e 2023-07-14T19:30:19.218 DEBUG:teuthology.orchestra.run.smithi098:> sudo ip link set brx.0 up 2023-07-14T19:30:19.218 DEBUG:teuthology.orchestra.run.smithi098:> sudo ip link set dev brx.0 master ceph-brx 2023-07-14T19:30:19.218 DEBUG:teuthology.orchestra.run.smithi098:> ') 2023-07-14T19:30:19.220 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:30:18 smithi139 ceph-mon[129293]: pgmap v293: 97 pgs: 97 active+clean; 634 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 5 op/s; 1 B/s, 0 objects/s recovering 2023-07-14T19:30:19.220 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:30:18 smithi139 ceph-mon[129293]: from='client.? 172.21.15.98:0/1524318238' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2023-07-14T19:30:19.355 INFO:teuthology.orchestra.run:Running command with timeout 60 2023-07-14T19:30:19.355 DEBUG:teuthology.orchestra.run.smithi098:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2023-07-14T19:30:19.372 INFO:teuthology.orchestra.run.smithi098.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2023-07-14T19:30:19.372 INFO:teuthology.orchestra.run:Running command with timeout 60 2023-07-14T19:30:19.373 DEBUG:teuthology.orchestra.run.smithi098:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2023-07-14T19:30:19.429 WARNING:tasks.cephfs.mount:You cannot specify both: cephfs_mntpt and mount_subvol_num 2023-07-14T19:30:19.429 INFO:tasks.cephfs.mount:Mounting subvol 1 for now 2023-07-14T19:30:19.429 INFO:tasks.cephfs.kernel_mount:mounting using device: 0@.cephfs=/volumes/_nogroup/sv_1/abb1cea8-1576-4a15-9246-a8da768b187f 2023-07-14T19:30:19.430 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-07-14T19:30:19.430 DEBUG:teuthology.orchestra.run.smithi098:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph 0@.cephfs=/volumes/_nogroup/sv_1/abb1cea8-1576-4a15-9246-a8da768b187f /home/ubuntu/cephtest/mnt.0 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,ms_mode=crc,wsync,nofallback 2023-07-14T19:30:19.529 INFO:teuthology.orchestra.run.smithi098.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,ms_mode=crc,wsync,nofallback 2023-07-14T19:30:19.530 INFO:teuthology.orchestra.run.smithi098.stdout:mount.ceph: options "norequire_active_mds,norbytes,ms_mode=crc,wsync". 2023-07-14T19:30:19.670 INFO:teuthology.orchestra.run.smithi098.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,ms_mode=crc,wsync,nofallback 2023-07-14T19:30:19.670 INFO:teuthology.orchestra.run.smithi098.stdout:mount.ceph: options "norequire_active_mds,norbytes,ms_mode=crc,wsync". 2023-07-14T19:30:19.670 INFO:teuthology.orchestra.run.smithi098.stdout:mount.ceph: resolved to: "172.21.15.98:3300,172.21.15.101:3300,172.21.15.139:3300" 2023-07-14T19:30:19.670 INFO:teuthology.orchestra.run.smithi098.stdout:mount.ceph: trying mount with new device syntax: 0@9adea648-227b-11ee-9b33-001a4aab830c.cephfs=/volumes/_nogroup/sv_1/abb1cea8-1576-4a15-9246-a8da768b187f 2023-07-14T19:30:19.670 INFO:teuthology.orchestra.run.smithi098.stdout:mount.ceph: options "norequire_active_mds,norbytes,ms_mode=crc,wsync,name=0,key=0,mon_addr=172.21.15.98:3300/172.21.15.101:3300/172.21.15.139:3300" will pass to kernel 2023-07-14T19:30:19.670 INFO:teuthology.orchestra.run.smithi098.stdout:mount.ceph: v2 mount support check returned 2 2023-07-14T19:30:19.671 INFO:teuthology.orchestra.run.smithi098.stdout:mount.ceph: kernel does not support v2 syntax 2023-07-14T19:30:19.671 INFO:teuthology.orchestra.run.smithi098.stdout:mount.ceph: trying mount with old device syntax: 172.21.15.98:3300,172.21.15.101:3300,172.21.15.139:3300:/volumes/_nogroup/sv_1/abb1cea8-1576-4a15-9246-a8da768b187f 2023-07-14T19:30:19.671 INFO:teuthology.orchestra.run.smithi098.stdout:mount.ceph: options "norequire_active_mds,norbytes,ms_mode=crc,wsync,name=0,key=0,mds_namespace=cephfs,fsid=9adea648-227b-11ee-9b33-001a4aab830c" will pass to kernel 2023-07-14T19:30:19.685 INFO:tasks.cephfs.kernel_mount:mount command passed 2023-07-14T19:30:19.685 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-07-14T19:30:19.685 DEBUG:teuthology.orchestra.run.smithi098:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2023-07-14T19:30:19.716 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-07-14T19:30:19.716 DEBUG:teuthology.orchestra.run.smithi098:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2023-07-14T19:30:19.809 INFO:teuthology.orchestra.run.smithi098.stdout:client24920 2023-07-14T19:30:19.809 DEBUG:teuthology.orchestra.run.smithi098:> set -ex 2023-07-14T19:30:19.810 DEBUG:teuthology.orchestra.run.smithi098:> dd if=/proc/self/mounts of=/dev/stdout 2023-07-14T19:30:19.828 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-07-14T19:30:19.829 DEBUG:teuthology.orchestra.run.smithi098:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2023-07-14T19:30:19.919 INFO:teuthology.orchestra.run.smithi098.stdout:client24920 2023-07-14T19:30:19.920 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-07-14T19:30:19.920 DEBUG:teuthology.orchestra.run.smithi098:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2023-07-14T19:30:19.972 INFO:teuthology.orchestra.run.smithi098.stdout:client24920 2023-07-14T19:30:19.972 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-07-14T19:30:19.973 DEBUG:teuthology.orchestra.run.smithi098:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c 'sudo dd if=/sys/kernel/debug/ceph/9adea648-227b-11ee-9b33-001a4aab830c.client24920/status') 2023-07-14T19:30:20.030 INFO:teuthology.orchestra.run.smithi098.stdout:instance: client.24920 (3)172.21.15.98:0/3170586910 2023-07-14T19:30:20.030 INFO:teuthology.orchestra.run.smithi098.stdout:blocklisted: false 2023-07-14T19:30:20.030 INFO:teuthology.orchestra.run.smithi098.stderr:0+1 records in 2023-07-14T19:30:20.031 INFO:teuthology.orchestra.run.smithi098.stderr:0+1 records out 2023-07-14T19:30:20.031 INFO:teuthology.orchestra.run.smithi098.stderr:71 bytes copied, 2.1916e-05 s, 3.2 MB/s 2023-07-14T19:30:20.032 INFO:teuthology.run_tasks:Running task exec... 2023-07-14T19:30:20.045 INFO:teuthology.task.exec:Executing custom commands... 2023-07-14T19:30:20.046 INFO:teuthology.task.exec:Running commands on role mon.a host ubuntu@smithi098.front.sepia.ceph.com 2023-07-14T19:30:20.046 DEBUG:teuthology.orchestra.run.smithi098:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph fs set cephfs balance_automate false' 2023-07-14T19:30:20.170 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:20.170+0000 7f2527652700 1 -- 172.21.15.98:0/1554436049 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2520105810 msgr2=0x7f2520107cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:30:20.170 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:20.170+0000 7f2527652700 1 --2- 172.21.15.98:0/1554436049 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2520105810 0x7f2520107cc0 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 crypto rx=0x7f2518009ec0 tx=0x7f251800b330 comp rx=0 tx=0).stop 2023-07-14T19:30:20.171 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:20.170+0000 7f2527652700 1 -- 172.21.15.98:0/1554436049 shutdown_connections 2023-07-14T19:30:20.171 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:20.170+0000 7f2527652700 1 --2- 172.21.15.98:0/1554436049 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f2520068710 0x7f2520102820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:30:20.171 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:20.170+0000 7f2527652700 1 --2- 172.21.15.98:0/1554436049 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2520105810 0x7f2520107cc0 unknown :-1 s=CLOSED pgs=391 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:30:20.171 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:20.170+0000 7f2527652700 1 --2- 172.21.15.98:0/1554436049 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f2520102d60 0x7f2520105180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:30:20.171 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:20.170+0000 7f2527652700 1 -- 172.21.15.98:0/1554436049 >> 172.21.15.98:0/1554436049 conn(0x7f25200fa130 msgr2=0x7f25200fc580 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:30:20.171 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:20.170+0000 7f2527652700 1 -- 172.21.15.98:0/1554436049 shutdown_connections 2023-07-14T19:30:20.171 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:20.170+0000 7f2527652700 1 -- 172.21.15.98:0/1554436049 wait complete. 2023-07-14T19:30:20.173 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:20.170+0000 7f2527652700 1 Processor -- start 2023-07-14T19:30:20.173 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:20.171+0000 7f2527652700 1 -- start start 2023-07-14T19:30:20.174 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:20.171+0000 7f2527652700 1 --2- >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f2520068710 0x7f2520197d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:30:20.174 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:20.171+0000 7f2527652700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2520102d60 0x7f25201982a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:30:20.174 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:20.171+0000 7f2527652700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f2520105810 0x7f2520199040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:30:20.174 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:20.171+0000 7f2527652700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2520199750 con 0x7f2520102d60 2023-07-14T19:30:20.174 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:20.171+0000 7f2527652700 1 -- --> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f25201998c0 con 0x7f2520068710 2023-07-14T19:30:20.174 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:20.171+0000 7f2527652700 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2520199a80 con 0x7f2520105810 2023-07-14T19:30:20.174 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:20.171+0000 7f2524bed700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2520102d60 0x7f25201982a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:30:20.175 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:20.171+0000 7f2524bed700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2520102d60 0x7f25201982a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:46988/0 (socket says 172.21.15.98:46988) 2023-07-14T19:30:20.175 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:20.171+0000 7f2524bed700 1 -- 172.21.15.98:0/2319639332 learned_addr learned my addr 172.21.15.98:0/2319639332 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-07-14T19:30:20.175 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:20.171+0000 7f25253ee700 1 --2- 172.21.15.98:0/2319639332 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f2520068710 0x7f2520197d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:30:20.175 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:20.171+0000 7f2525bef700 1 --2- 172.21.15.98:0/2319639332 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f2520105810 0x7f2520199040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:30:20.175 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:20.171+0000 7f2524bed700 1 -- 172.21.15.98:0/2319639332 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f2520105810 msgr2=0x7f2520199040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:30:20.175 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:20.171+0000 7f2524bed700 1 --2- 172.21.15.98:0/2319639332 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f2520105810 0x7f2520199040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:30:20.175 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:20.171+0000 7f2524bed700 1 -- 172.21.15.98:0/2319639332 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f2520068710 msgr2=0x7f2520197d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:30:20.175 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:20.171+0000 7f2524bed700 1 --2- 172.21.15.98:0/2319639332 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f2520068710 0x7f2520197d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:30:20.176 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:20.171+0000 7f2524bed700 1 -- 172.21.15.98:0/2319639332 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2518005e70 con 0x7f2520102d60 2023-07-14T19:30:20.176 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:20.171+0000 7f2525bef700 1 --2- 172.21.15.98:0/2319639332 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f2520105810 0x7f2520199040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-07-14T19:30:20.176 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:20.171+0000 7f2524bed700 1 --2- 172.21.15.98:0/2319639332 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2520102d60 0x7f25201982a0 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 crypto rx=0x7f251c00c9b0 tx=0x7f251c00ccc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:30:20.176 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:20.171+0000 7f25167fc700 1 -- 172.21.15.98:0/2319639332 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f251c007b00 con 0x7f2520102d60 2023-07-14T19:30:20.176 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:20.172+0000 7f25167fc700 1 -- 172.21.15.98:0/2319639332 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f251c007c60 con 0x7f2520102d60 2023-07-14T19:30:20.176 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:20.172+0000 7f2527652700 1 -- 172.21.15.98:0/2319639332 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2520199d60 con 0x7f2520102d60 2023-07-14T19:30:20.176 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:20.172+0000 7f25167fc700 1 -- 172.21.15.98:0/2319639332 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f251c018660 con 0x7f2520102d60 2023-07-14T19:30:20.177 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:20.172+0000 7f2527652700 1 -- 172.21.15.98:0/2319639332 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f25201a04c0 con 0x7f2520102d60 2023-07-14T19:30:20.177 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:20.172+0000 7f2527652700 1 -- 172.21.15.98:0/2319639332 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f252004e420 con 0x7f2520102d60 2023-07-14T19:30:20.177 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:20.174+0000 7f25167fc700 1 -- 172.21.15.98:0/2319639332 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 17) v1 ==== 135766+0+0 (secure 0 0 0) 0x7f251c00f450 con 0x7f2520102d60 2023-07-14T19:30:20.178 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:20.174+0000 7f25167fc700 1 --2- 172.21.15.98:0/2319639332 >> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] conn(0x7f250c0a11e0 0x7f250c0a3690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:30:20.178 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:20.174+0000 7f25253ee700 1 --2- 172.21.15.98:0/2319639332 >> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] conn(0x7f250c0a11e0 0x7f250c0a3690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:30:20.178 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:20.174+0000 7f25167fc700 1 -- 172.21.15.98:0/2319639332 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 8211+0+0 (secure 0 0 0) 0x7f251c0ccd20 con 0x7f2520102d60 2023-07-14T19:30:20.178 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:20.175+0000 7f25253ee700 1 --2- 172.21.15.98:0/2319639332 >> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] conn(0x7f250c0a11e0 0x7f250c0a3690 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f2510005950 tx=0x7f251000a300 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:30:20.178 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:20.177+0000 7f25167fc700 1 -- 172.21.15.98:0/2319639332 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f251c0963b0 con 0x7f2520102d60 2023-07-14T19:30:20.415 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:20.414+0000 7f2527652700 1 -- 172.21.15.98:0/2319639332 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "false"} v 0) v1 -- 0x7f25201a0740 con 0x7f2520102d60 2023-07-14T19:30:20.770 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:20.770+0000 7f25167fc700 1 -- 172.21.15.98:0/2319639332 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "false"}]=0 v23) v1 ==== 118+0+0 (secure 0 0 0) 0x7f251c0999d0 con 0x7f2520102d60 2023-07-14T19:30:20.772 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:20.772+0000 7f2527652700 1 -- 172.21.15.98:0/2319639332 >> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] conn(0x7f250c0a11e0 msgr2=0x7f250c0a3690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:30:20.773 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:20.772+0000 7f2527652700 1 --2- 172.21.15.98:0/2319639332 >> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] conn(0x7f250c0a11e0 0x7f250c0a3690 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f2510005950 tx=0x7f251000a300 comp rx=0 tx=0).stop 2023-07-14T19:30:20.773 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:20.772+0000 7f2527652700 1 -- 172.21.15.98:0/2319639332 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2520102d60 msgr2=0x7f25201982a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:30:20.773 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:20.772+0000 7f2527652700 1 --2- 172.21.15.98:0/2319639332 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2520102d60 0x7f25201982a0 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 crypto rx=0x7f251c00c9b0 tx=0x7f251c00ccc0 comp rx=0 tx=0).stop 2023-07-14T19:30:20.773 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:20.772+0000 7f2527652700 1 -- 172.21.15.98:0/2319639332 shutdown_connections 2023-07-14T19:30:20.773 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:20.772+0000 7f2527652700 1 --2- 172.21.15.98:0/2319639332 >> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] conn(0x7f250c0a11e0 0x7f250c0a3690 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:30:20.773 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:20.772+0000 7f2527652700 1 --2- 172.21.15.98:0/2319639332 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f2520105810 0x7f2520199040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:30:20.773 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:20.772+0000 7f2527652700 1 --2- 172.21.15.98:0/2319639332 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f2520068710 0x7f2520197d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:30:20.774 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:20.772+0000 7f2527652700 1 --2- 172.21.15.98:0/2319639332 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2520102d60 0x7f25201982a0 unknown :-1 s=CLOSED pgs=392 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:30:20.774 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:20.772+0000 7f2527652700 1 -- 172.21.15.98:0/2319639332 >> 172.21.15.98:0/2319639332 conn(0x7f25200fa130 msgr2=0x7f25200fc510 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:30:20.774 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:20.772+0000 7f2527652700 1 -- 172.21.15.98:0/2319639332 shutdown_connections 2023-07-14T19:30:20.774 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:20.772+0000 7f2527652700 1 -- 172.21.15.98:0/2319639332 wait complete. 2023-07-14T19:30:20.786 DEBUG:teuthology.orchestra.run.smithi098:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph fs subvolumegroup pin cephfs _nogroup random 0.01' 2023-07-14T19:30:20.910 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:20.909+0000 7f84ea766700 1 -- 172.21.15.98:0/3484471120 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f84e40ffdd0 msgr2=0x7f84e4100220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:30:20.910 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:20.909+0000 7f84ea766700 1 --2- 172.21.15.98:0/3484471120 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f84e40ffdd0 0x7f84e4100220 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 crypto rx=0x7f84d8009990 tx=0x7f84d8009ca0 comp rx=0 tx=0).stop 2023-07-14T19:30:20.910 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:20.909+0000 7f84ea766700 1 -- 172.21.15.98:0/3484471120 shutdown_connections 2023-07-14T19:30:20.910 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:20.909+0000 7f84ea766700 1 --2- 172.21.15.98:0/3484471120 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f84e40febe0 0x7f84e40feff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:30:20.910 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:20.909+0000 7f84ea766700 1 --2- 172.21.15.98:0/3484471120 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f84e4100760 0x7f84e4107020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:30:20.911 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:20.909+0000 7f84ea766700 1 --2- 172.21.15.98:0/3484471120 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f84e40ffdd0 0x7f84e4100220 unknown :-1 s=CLOSED pgs=393 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:30:20.911 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:20.909+0000 7f84ea766700 1 -- 172.21.15.98:0/3484471120 >> 172.21.15.98:0/3484471120 conn(0x7f84e40fa200 msgr2=0x7f84e40fc630 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:30:20.911 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:20.909+0000 7f84ea766700 1 -- 172.21.15.98:0/3484471120 shutdown_connections 2023-07-14T19:30:20.911 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:20.909+0000 7f84ea766700 1 -- 172.21.15.98:0/3484471120 wait complete. 2023-07-14T19:30:20.913 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:20.910+0000 7f84ea766700 1 Processor -- start 2023-07-14T19:30:20.913 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:20.910+0000 7f84ea766700 1 -- start start 2023-07-14T19:30:20.913 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:20.910+0000 7f84ea766700 1 --2- >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f84e40febe0 0x7f84e4195ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:30:20.913 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:20.910+0000 7f84ea766700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f84e40ffdd0 0x7f84e4195ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:30:20.913 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:20.910+0000 7f84ea766700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f84e4100760 0x7f84e4196e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:30:20.913 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:20.910+0000 7f84ea766700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f84e41974d0 con 0x7f84e4100760 2023-07-14T19:30:20.914 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:20.910+0000 7f84ea766700 1 -- --> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f84e4197640 con 0x7f84e40febe0 2023-07-14T19:30:20.914 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:20.910+0000 7f84ea766700 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f84e4197800 con 0x7f84e40ffdd0 2023-07-14T19:30:20.914 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:20.910+0000 7f84e8d03700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f84e4100760 0x7f84e4196e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:30:20.914 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:20.910+0000 7f84e8d03700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f84e4100760 0x7f84e4196e70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:47002/0 (socket says 172.21.15.98:47002) 2023-07-14T19:30:20.914 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:20.910+0000 7f84e8d03700 1 -- 172.21.15.98:0/673919544 learned_addr learned my addr 172.21.15.98:0/673919544 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-07-14T19:30:20.914 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:20.911+0000 7f84e37fe700 1 --2- 172.21.15.98:0/673919544 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f84e40ffdd0 0x7f84e4195ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:30:20.914 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:20.911+0000 7f84e8d03700 1 -- 172.21.15.98:0/673919544 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f84e40ffdd0 msgr2=0x7f84e4195ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:30:20.914 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:20.911+0000 7f84e8d03700 1 --2- 172.21.15.98:0/673919544 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f84e40ffdd0 0x7f84e4195ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:30:20.915 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:20.911+0000 7f84e8d03700 1 -- 172.21.15.98:0/673919544 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f84e40febe0 msgr2=0x7f84e4195ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:30:20.915 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:20.911+0000 7f84e3fff700 1 --2- 172.21.15.98:0/673919544 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f84e40febe0 0x7f84e4195ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:30:20.915 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:20.911+0000 7f84e8d03700 1 --2- 172.21.15.98:0/673919544 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f84e40febe0 0x7f84e4195ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:30:20.915 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:20.911+0000 7f84e8d03700 1 -- 172.21.15.98:0/673919544 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f84d80095f0 con 0x7f84e4100760 2023-07-14T19:30:20.915 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:20.911+0000 7f84e8d03700 1 --2- 172.21.15.98:0/673919544 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f84e4100760 0x7f84e4196e70 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 crypto rx=0x7f84dc007ee0 tx=0x7f84dc00d3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:30:20.915 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:20.911+0000 7f84e17fa700 1 -- 172.21.15.98:0/673919544 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f84dc017070 con 0x7f84e4100760 2023-07-14T19:30:20.915 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:20.911+0000 7f84e17fa700 1 -- 172.21.15.98:0/673919544 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f84dc00f040 con 0x7f84e4100760 2023-07-14T19:30:20.916 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:20.911+0000 7f84ea766700 1 -- 172.21.15.98:0/673919544 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f84e4197ae0 con 0x7f84e4100760 2023-07-14T19:30:20.916 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:20.911+0000 7f84e17fa700 1 -- 172.21.15.98:0/673919544 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f84dc005740 con 0x7f84e4100760 2023-07-14T19:30:20.916 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:20.911+0000 7f84ea766700 1 -- 172.21.15.98:0/673919544 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f84e419e2a0 con 0x7f84e4100760 2023-07-14T19:30:20.916 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:20.912+0000 7f84ea766700 1 -- 172.21.15.98:0/673919544 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f84e404e420 con 0x7f84e4100760 2023-07-14T19:30:20.917 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:20.913+0000 7f84e17fa700 1 -- 172.21.15.98:0/673919544 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 17) v1 ==== 135766+0+0 (secure 0 0 0) 0x7f84dc0058a0 con 0x7f84e4100760 2023-07-14T19:30:20.917 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:20.914+0000 7f84e17fa700 1 --2- 172.21.15.98:0/673919544 >> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] conn(0x7f84cc0a11e0 0x7f84cc0a3690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:30:20.917 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:20.914+0000 7f84e3fff700 1 --2- 172.21.15.98:0/673919544 >> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] conn(0x7f84cc0a11e0 0x7f84cc0a3690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:30:20.917 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:20.914+0000 7f84e17fa700 1 -- 172.21.15.98:0/673919544 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 8211+0+0 (secure 0 0 0) 0x7f84dc0cc2b0 con 0x7f84e4100760 2023-07-14T19:30:20.917 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:20.914+0000 7f84e3fff700 1 --2- 172.21.15.98:0/673919544 >> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] conn(0x7f84cc0a11e0 0x7f84cc0a3690 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f84d4005fd0 tx=0x7f84d4005f60 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:30:20.918 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:20.918+0000 7f84e17fa700 1 -- 172.21.15.98:0/673919544 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f84dc0958b0 con 0x7f84e4100760 2023-07-14T19:30:21.031 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:30:20 smithi098 ceph-mon[127486]: pgmap v294: 97 pgs: 97 active+clean; 634 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2023-07-14T19:30:21.031 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:30:20 smithi098 ceph-mon[127486]: from='client.? 172.21.15.98:0/2319639332' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "false"}]: dispatch 2023-07-14T19:30:21.141 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:21.141+0000 7f84ea766700 1 -- 172.21.15.98:0/673919544 --> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] -- mgr_command(tid 0: {"prefix": "fs subvolumegroup pin", "vol_name": "cephfs", "group_name": "_nogroup", "pin_type": "random", "pin_setting": "0.01", "target": ["mon-mgr", ""]}) v1 -- 0x7f84e4069090 con 0x7f84cc0a11e0 2023-07-14T19:30:21.142 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:21.142+0000 7f84e17fa700 1 -- 172.21.15.98:0/673919544 <== mgr.14152 v2:172.21.15.98:6800/452042164 1 ==== mgr_command_reply(tid 0: -1 Operation not permitted for group '_nogroup' as it is an internal group.) v1 ==== 80+0+0 (secure 0 0 0) 0x7f84e4069090 con 0x7f84cc0a11e0 2023-07-14T19:30:21.144 INFO:teuthology.orchestra.run.smithi098.stderr:Error EPERM: Operation not permitted for group '_nogroup' as it is an internal group. 2023-07-14T19:30:21.147 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-07-14T19:30:21.148 ERROR:teuthology.run_tasks:Saw exception from tasks. Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_aa6a80e6cc655fff48452a7eca02070e61d074fc/teuthology/run_tasks.py", line 105, in run_tasks manager = run_one_task(taskname, ctx=ctx, config=config) File "/home/teuthworker/src/git.ceph.com_teuthology_aa6a80e6cc655fff48452a7eca02070e61d074fc/teuthology/run_tasks.py", line 84, in run_one_task return task(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_aa6a80e6cc655fff48452a7eca02070e61d074fc/teuthology/task/exec.py", line 66, in task remote.run( File "/home/teuthworker/src/git.ceph.com_teuthology_aa6a80e6cc655fff48452a7eca02070e61d074fc/teuthology/orchestra/remote.py", line 522, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_aa6a80e6cc655fff48452a7eca02070e61d074fc/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_aa6a80e6cc655fff48452a7eca02070e61d074fc/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_aa6a80e6cc655fff48452a7eca02070e61d074fc/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi098 with status 1: "sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph fs subvolumegroup pin cephfs _nogroup random 0.01'" 2023-07-14T19:30:21.210 ERROR:teuthology.run_tasks: Sentry event: https://sentry.ceph.com/organizations/ceph/?query=23a9ab8569fe49bbaf9c6dc2feefbb78 Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_aa6a80e6cc655fff48452a7eca02070e61d074fc/teuthology/run_tasks.py", line 105, in run_tasks manager = run_one_task(taskname, ctx=ctx, config=config) File "/home/teuthworker/src/git.ceph.com_teuthology_aa6a80e6cc655fff48452a7eca02070e61d074fc/teuthology/run_tasks.py", line 84, in run_one_task return task(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_aa6a80e6cc655fff48452a7eca02070e61d074fc/teuthology/task/exec.py", line 66, in task remote.run( File "/home/teuthworker/src/git.ceph.com_teuthology_aa6a80e6cc655fff48452a7eca02070e61d074fc/teuthology/orchestra/remote.py", line 522, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_aa6a80e6cc655fff48452a7eca02070e61d074fc/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_aa6a80e6cc655fff48452a7eca02070e61d074fc/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_aa6a80e6cc655fff48452a7eca02070e61d074fc/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi098 with status 1: "sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph fs subvolumegroup pin cephfs _nogroup random 0.01'" 2023-07-14T19:30:21.212 DEBUG:teuthology.run_tasks:Unwinding manager kclient 2023-07-14T19:30:21.222 INFO:tasks.kclient:Unmounting kernel clients... 2023-07-14T19:30:21.222 DEBUG:teuthology.orchestra.run.smithi098:> set -ex 2023-07-14T19:30:21.222 DEBUG:teuthology.orchestra.run.smithi098:> dd if=/proc/self/mounts of=/dev/stdout 2023-07-14T19:30:26.230 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Jul 14 19:30:21 smithi098 ceph-9adea648-227b-11ee-9b33-001a4aab830c-mgr-x[127734]: 2023-07-14T19:30:21.142+0000 7f245642d700 -1 mgr.server reply reply (1) Operation not permitted Operation not permitted for group '_nogroup' as it is an internal group. 2023-07-14T19:30:26.231 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:30:20 smithi139 ceph-mon[129293]: pgmap v294: 97 pgs: 97 active+clean; 634 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2023-07-14T19:30:26.232 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:30:20 smithi139 ceph-mon[129293]: from='client.? 172.21.15.98:0/2319639332' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "false"}]: dispatch 2023-07-14T19:30:26.232 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:30:21 smithi139 ceph-mon[129293]: from='client.? 172.21.15.98:0/2319639332' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "false"}]': finished 2023-07-14T19:30:26.232 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:30:21 smithi139 ceph-mon[129293]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2023-07-14T19:30:26.232 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:30:22 smithi139 ceph-mon[129293]: from='client.15282 -' entity='client.admin' cmd=[{"prefix": "fs subvolumegroup pin", "vol_name": "cephfs", "group_name": "_nogroup", "pin_type": "random", "pin_setting": "0.01", "target": ["mon-mgr", ""]}]: dispatch 2023-07-14T19:30:26.233 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:30:22 smithi139 ceph-mon[129293]: pgmap v295: 97 pgs: 97 active+clean; 634 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2023-07-14T19:30:26.233 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:30:24 smithi139 ceph-mon[129293]: pgmap v296: 97 pgs: 97 active+clean; 634 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2023-07-14T19:30:26.233 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:30:20 smithi101 ceph-mon[128807]: pgmap v294: 97 pgs: 97 active+clean; 634 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2023-07-14T19:30:26.233 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:30:20 smithi101 ceph-mon[128807]: from='client.? 172.21.15.98:0/2319639332' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "false"}]: dispatch 2023-07-14T19:30:26.234 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:30:21 smithi101 ceph-mon[128807]: from='client.? 172.21.15.98:0/2319639332' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "false"}]': finished 2023-07-14T19:30:26.234 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:30:21 smithi101 ceph-mon[128807]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2023-07-14T19:30:26.234 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:30:22 smithi101 ceph-mon[128807]: from='client.15282 -' entity='client.admin' cmd=[{"prefix": "fs subvolumegroup pin", "vol_name": "cephfs", "group_name": "_nogroup", "pin_type": "random", "pin_setting": "0.01", "target": ["mon-mgr", ""]}]: dispatch 2023-07-14T19:30:26.234 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:30:22 smithi101 ceph-mon[128807]: pgmap v295: 97 pgs: 97 active+clean; 634 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2023-07-14T19:30:26.234 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:30:24 smithi101 ceph-mon[128807]: pgmap v296: 97 pgs: 97 active+clean; 634 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2023-07-14T19:30:26.235 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:30:21 smithi098 ceph-mon[127486]: from='client.? 172.21.15.98:0/2319639332' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "false"}]': finished 2023-07-14T19:30:26.235 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:30:21 smithi098 ceph-mon[127486]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2023-07-14T19:30:26.235 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:30:22 smithi098 ceph-mon[127486]: from='client.15282 -' entity='client.admin' cmd=[{"prefix": "fs subvolumegroup pin", "vol_name": "cephfs", "group_name": "_nogroup", "pin_type": "random", "pin_setting": "0.01", "target": ["mon-mgr", ""]}]: dispatch 2023-07-14T19:30:26.235 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:30:22 smithi098 ceph-mon[127486]: pgmap v295: 97 pgs: 97 active+clean; 634 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2023-07-14T19:30:26.235 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:30:24 smithi098 ceph-mon[127486]: pgmap v296: 97 pgs: 97 active+clean; 634 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2023-07-14T19:30:26.251 DEBUG:teuthology.orchestra.run.smithi098:> set -ex 2023-07-14T19:30:26.251 DEBUG:teuthology.orchestra.run.smithi098:> dd if=/proc/self/mounts of=/dev/stdout 2023-07-14T19:30:26.311 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-07-14T19:30:26.480 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:26.479+0000 7fe3c4a5d700 1 -- 172.21.15.98:0/3782594644 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe3c00fe280 msgr2=0x7fe3c00fe690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:30:26.480 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:26.479+0000 7fe3c4a5d700 1 --2- 172.21.15.98:0/3782594644 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe3c00fe280 0x7fe3c00fe690 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 crypto rx=0x7fe3b4009990 tx=0x7fe3b4009ca0 comp rx=0 tx=0).stop 2023-07-14T19:30:26.480 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:26.479+0000 7fe3c4a5d700 1 -- 172.21.15.98:0/3782594644 shutdown_connections 2023-07-14T19:30:26.481 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:26.479+0000 7fe3c4a5d700 1 --2- 172.21.15.98:0/3782594644 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7fe3c00ffe00 0x7fe3c01066c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:30:26.481 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:26.479+0000 7fe3c4a5d700 1 --2- 172.21.15.98:0/3782594644 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe3c00fe280 0x7fe3c00fe690 unknown :-1 s=CLOSED pgs=395 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:30:26.481 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:26.479+0000 7fe3c4a5d700 1 --2- 172.21.15.98:0/3782594644 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fe3c00ff470 0x7fe3c00ff8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:30:26.481 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:26.479+0000 7fe3c4a5d700 1 -- 172.21.15.98:0/3782594644 >> 172.21.15.98:0/3782594644 conn(0x7fe3c00f98a0 msgr2=0x7fe3c00fbcd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:30:26.481 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:26.479+0000 7fe3c4a5d700 1 -- 172.21.15.98:0/3782594644 shutdown_connections 2023-07-14T19:30:26.482 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:26.479+0000 7fe3c4a5d700 1 -- 172.21.15.98:0/3782594644 wait complete. 2023-07-14T19:30:26.484 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:26.480+0000 7fe3c4a5d700 1 Processor -- start 2023-07-14T19:30:26.484 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:26.480+0000 7fe3c4a5d700 1 -- start start 2023-07-14T19:30:26.484 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:26.480+0000 7fe3c4a5d700 1 --2- >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7fe3c00fe280 0x7fe3c00fbc10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:30:26.485 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:26.480+0000 7fe3c4a5d700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fe3c00ff470 0x7fe3c019ca70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:30:26.485 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:26.480+0000 7fe3c4a5d700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe3c00ffe00 0x7fe3c019efc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:30:26.485 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:26.480+0000 7fe3c4a5d700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe3c019f670 con 0x7fe3c00ffe00 2023-07-14T19:30:26.485 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:26.480+0000 7fe3c4a5d700 1 -- --> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe3c019f7e0 con 0x7fe3c00fe280 2023-07-14T19:30:26.485 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:26.480+0000 7fe3c4a5d700 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe3c019f9a0 con 0x7fe3c00ff470 2023-07-14T19:30:26.486 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:26.480+0000 7fe3bed9d700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe3c00ffe00 0x7fe3c019efc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:30:26.486 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:26.480+0000 7fe3bed9d700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe3c00ffe00 0x7fe3c019efc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:47016/0 (socket says 172.21.15.98:47016) 2023-07-14T19:30:26.486 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:26.480+0000 7fe3bed9d700 1 -- 172.21.15.98:0/861502722 learned_addr learned my addr 172.21.15.98:0/861502722 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-07-14T19:30:26.486 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:26.480+0000 7fe3bdd9b700 1 --2- 172.21.15.98:0/861502722 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fe3c00ff470 0x7fe3c019ca70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:30:26.486 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:26.480+0000 7fe3be59c700 1 --2- 172.21.15.98:0/861502722 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7fe3c00fe280 0x7fe3c00fbc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:30:26.487 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:26.480+0000 7fe3bed9d700 1 -- 172.21.15.98:0/861502722 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fe3c00ff470 msgr2=0x7fe3c019ca70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:30:26.487 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:26.480+0000 7fe3bed9d700 1 --2- 172.21.15.98:0/861502722 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fe3c00ff470 0x7fe3c019ca70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:30:26.487 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:26.480+0000 7fe3bed9d700 1 -- 172.21.15.98:0/861502722 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7fe3c00fe280 msgr2=0x7fe3c00fbc10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:30:26.487 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:26.480+0000 7fe3bed9d700 1 --2- 172.21.15.98:0/861502722 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7fe3c00fe280 0x7fe3c00fbc10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:30:26.487 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:26.480+0000 7fe3bed9d700 1 -- 172.21.15.98:0/861502722 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe3b40095f0 con 0x7fe3c00ffe00 2023-07-14T19:30:26.488 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:26.480+0000 7fe3bdd9b700 1 --2- 172.21.15.98:0/861502722 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fe3c00ff470 0x7fe3c019ca70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-07-14T19:30:26.488 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:26.481+0000 7fe3bed9d700 1 --2- 172.21.15.98:0/861502722 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe3c00ffe00 0x7fe3c019efc0 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 crypto rx=0x7fe3b800b970 tx=0x7fe3b800bd30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:30:26.488 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:26.481+0000 7fe3af7fe700 1 -- 172.21.15.98:0/861502722 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe3b800c9b0 con 0x7fe3c00ffe00 2023-07-14T19:30:26.488 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:26.481+0000 7fe3af7fe700 1 -- 172.21.15.98:0/861502722 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fe3b8014440 con 0x7fe3c00ffe00 2023-07-14T19:30:26.488 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:26.481+0000 7fe3c4a5d700 1 -- 172.21.15.98:0/861502722 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe3c019fc80 con 0x7fe3c00ffe00 2023-07-14T19:30:26.489 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:26.481+0000 7fe3af7fe700 1 -- 172.21.15.98:0/861502722 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe3b80075d0 con 0x7fe3c00ffe00 2023-07-14T19:30:26.489 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:26.481+0000 7fe3c4a5d700 1 -- 172.21.15.98:0/861502722 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe3c01a0210 con 0x7fe3c00ffe00 2023-07-14T19:30:26.489 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:26.482+0000 7fe3c4a5d700 1 -- 172.21.15.98:0/861502722 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe3c0105df0 con 0x7fe3c00ffe00 2023-07-14T19:30:26.489 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:26.483+0000 7fe3af7fe700 1 -- 172.21.15.98:0/861502722 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 17) v1 ==== 135766+0+0 (secure 0 0 0) 0x7fe3b80145b0 con 0x7fe3c00ffe00 2023-07-14T19:30:26.489 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:26.483+0000 7fe3af7fe700 1 --2- 172.21.15.98:0/861502722 >> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] conn(0x7fe3a80a1220 0x7fe3a80a36d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:30:26.490 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:26.483+0000 7fe3be59c700 1 --2- 172.21.15.98:0/861502722 >> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] conn(0x7fe3a80a1220 0x7fe3a80a36d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:30:26.490 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:26.483+0000 7fe3af7fe700 1 -- 172.21.15.98:0/861502722 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 8211+0+0 (secure 0 0 0) 0x7fe3b8006e40 con 0x7fe3c00ffe00 2023-07-14T19:30:26.490 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:26.484+0000 7fe3be59c700 1 --2- 172.21.15.98:0/861502722 >> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] conn(0x7fe3a80a1220 0x7fe3a80a36d0 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7fe3b4006010 tx=0x7fe3b400b540 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:30:26.491 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:26.488+0000 7fe3af7fe700 1 -- 172.21.15.98:0/861502722 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7fe3b804e780 con 0x7fe3c00ffe00 2023-07-14T19:30:26.678 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:26.677+0000 7fe3c4a5d700 1 -- 172.21.15.98:0/861502722 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fe3c002ca50 con 0x7fe3c00ffe00 2023-07-14T19:30:26.679 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:26.679+0000 7fe3af7fe700 1 -- 172.21.15.98:0/861502722 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v85) v1 ==== 74+0+22604 (secure 0 0 0) 0x7fe3b80957d0 con 0x7fe3c00ffe00 2023-07-14T19:30:26.680 INFO:teuthology.orchestra.run.smithi098.stdout: 2023-07-14T19:30:26.680 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":85,"fsid":"9adea648-227b-11ee-9b33-001a4aab830c","created":"2023-07-14T19:22:07.178854+0000","modified":"2023-07-14T19:30:03.314948+0000","last_up_change":"2023-07-14T19:28:19.646827+0000","last_in_change":"2023-07-14T19:28:02.018325+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-07-14T19:25:09.845827+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2023-07-14T19:28:58.952921+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"82","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2023-07-14T19:29:00.228272+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"83","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.690000057220459,"score_stable":1.690000057220459,"optimal_score":1,"raw_score_acting":1.690000057220459,"raw_score_stable":1.690000057220459,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"a3797e89-1754-41cf-9517-227e06ffdd72","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6802","nonce":120507228},{"type":"v1","addr":"172.21.15.98:6803","nonce":120507228}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6804","nonce":120507228},{"type":"v1","addr":"172.21.15.98:6805","nonce":120507228}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6808","nonce":120507228},{"type":"v1","addr":"172.21.15.98:6809","nonce":120507228}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6806","nonce":120507228},{"type":"v1","addr":"172.21.15.98:6807","nonce":120507228}]},"public_addr":"172.21.15.98:6803/120507228","cluster_addr":"172.21.15.98:6805/120507228","heartbeat_back_addr":"172.21.15.98:6809/120507228","heartbeat_front_addr":"172.21.15.98:6807/120507228","state":["exists","up"]},{"osd":1,"uuid":"2b001add-1aaa-4496-a403-7ef9725b80a0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.101:6800","nonce":2450584584},{"type":"v1","addr":"172.21.15.101:6801","nonce":2450584584}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.101:6802","nonce":2450584584},{"type":"v1","addr":"172.21.15.101:6803","nonce":2450584584}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.101:6806","nonce":2450584584},{"type":"v1","addr":"172.21.15.101:6807","nonce":2450584584}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.101:6804","nonce":2450584584},{"type":"v1","addr":"172.21.15.101:6805","nonce":2450584584}]},"public_addr":"172.21.15.101:6801/2450584584","cluster_addr":"172.21.15.101:6803/2450584584","heartbeat_back_addr":"172.21.15.101:6807/2450584584","heartbeat_front_addr":"172.21.15.101:6805/2450584584","state":["exists","up"]},{"osd":2,"uuid":"48e6d499-3ca9-424b-b0fb-3ba5f33b9fc9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":945038995},{"type":"v1","addr":"172.21.15.139:6801","nonce":945038995}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":945038995},{"type":"v1","addr":"172.21.15.139:6803","nonce":945038995}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":945038995},{"type":"v1","addr":"172.21.15.139:6807","nonce":945038995}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":945038995},{"type":"v1","addr":"172.21.15.139:6805","nonce":945038995}]},"public_addr":"172.21.15.139:6801/945038995","cluster_addr":"172.21.15.139:6803/945038995","heartbeat_back_addr":"172.21.15.139:6807/945038995","heartbeat_front_addr":"172.21.15.139:6805/945038995","state":["exists","up"]},{"osd":3,"uuid":"5530b265-31a5-447b-a1ea-17ebae9cfddf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6810","nonce":3095249665},{"type":"v1","addr":"172.21.15.98:6811","nonce":3095249665}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6812","nonce":3095249665},{"type":"v1","addr":"172.21.15.98:6813","nonce":3095249665}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6816","nonce":3095249665},{"type":"v1","addr":"172.21.15.98:6817","nonce":3095249665}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6814","nonce":3095249665},{"type":"v1","addr":"172.21.15.98:6815","nonce":3095249665}]},"public_addr":"172.21.15.98:6811/3095249665","cluster_addr":"172.21.15.98:6813/3095249665","heartbeat_back_addr":"172.21.15.98:6817/3095249665","heartbeat_front_addr":"172.21.15.98:6815/3095249665","state":["exists","up"]},{"osd":4,"uuid":"615d4642-a997-4f95-9e38-9fbdaa831400","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":32,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.101:6808","nonce":2844761406},{"type":"v1","addr":"172.21.15.101:6809","nonce":2844761406}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.101:6810","nonce":2844761406},{"type":"v1","addr":"172.21.15.101:6811","nonce":2844761406}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.101:6814","nonce":2844761406},{"type":"v1","addr":"172.21.15.101:6815","nonce":2844761406}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.101:6812","nonce":2844761406},{"type":"v1","addr":"172.21.15.101:6813","nonce":2844761406}]},"public_addr":"172.21.15.101:6809/2844761406","cluster_addr":"172.21.15.101:6811/2844761406","heartbeat_back_addr":"172.21.15.101:6815/2844761406","heartbeat_front_addr":"172.21.15.101:6813/2844761406","state":["exists","up"]},{"osd":5,"uuid":"415de5fa-607a-45a8-b732-28daf3be6dd3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":37,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":3532263042},{"type":"v1","addr":"172.21.15.139:6809","nonce":3532263042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6810","nonce":3532263042},{"type":"v1","addr":"172.21.15.139:6811","nonce":3532263042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6814","nonce":3532263042},{"type":"v1","addr":"172.21.15.139:6815","nonce":3532263042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":3532263042},{"type":"v1","addr":"172.21.15.139:6813","nonce":3532263042}]},"public_addr":"172.21.15.139:6809/3532263042","cluster_addr":"172.21.15.139:6811/3532263042","heartbeat_back_addr":"172.21.15.139:6815/3532263042","heartbeat_front_addr":"172.21.15.139:6813/3532263042","state":["exists","up"]},{"osd":6,"uuid":"d7f9d4ba-c309-4b75-a836-5a36384b528e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":45,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6818","nonce":658948743},{"type":"v1","addr":"172.21.15.98:6819","nonce":658948743}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6820","nonce":658948743},{"type":"v1","addr":"172.21.15.98:6821","nonce":658948743}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6824","nonce":658948743},{"type":"v1","addr":"172.21.15.98:6825","nonce":658948743}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6822","nonce":658948743},{"type":"v1","addr":"172.21.15.98:6823","nonce":658948743}]},"public_addr":"172.21.15.98:6819/658948743","cluster_addr":"172.21.15.98:6821/658948743","heartbeat_back_addr":"172.21.15.98:6825/658948743","heartbeat_front_addr":"172.21.15.98:6823/658948743","state":["exists","up"]},{"osd":7,"uuid":"eedb94d9-c5cd-4a93-8a8c-2648804b7195","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":50,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.101:6816","nonce":2881214066},{"type":"v1","addr":"172.21.15.101:6817","nonce":2881214066}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.101:6818","nonce":2881214066},{"type":"v1","addr":"172.21.15.101:6819","nonce":2881214066}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.101:6822","nonce":2881214066},{"type":"v1","addr":"172.21.15.101:6823","nonce":2881214066}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.101:6820","nonce":2881214066},{"type":"v1","addr":"172.21.15.101:6821","nonce":2881214066}]},"public_addr":"172.21.15.101:6817/2881214066","cluster_addr":"172.21.15.101:6819/2881214066","heartbeat_back_addr":"172.21.15.101:6823/2881214066","heartbeat_front_addr":"172.21.15.101:6821/2881214066","state":["exists","up"]},{"osd":8,"uuid":"6bfd77c2-5770-466e-aa6f-84b50c74ffb6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":57,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":3669514524},{"type":"v1","addr":"172.21.15.139:6817","nonce":3669514524}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":3669514524},{"type":"v1","addr":"172.21.15.139:6819","nonce":3669514524}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":3669514524},{"type":"v1","addr":"172.21.15.139:6823","nonce":3669514524}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6820","nonce":3669514524},{"type":"v1","addr":"172.21.15.139:6821","nonce":3669514524}]},"public_addr":"172.21.15.139:6817/3669514524","cluster_addr":"172.21.15.139:6819/3669514524","heartbeat_back_addr":"172.21.15.139:6823/3669514524","heartbeat_front_addr":"172.21.15.139:6821/3669514524","state":["exists","up"]},{"osd":9,"uuid":"19c5b54a-2242-4b23-b721-63be9559653d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":65,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6826","nonce":2893994080},{"type":"v1","addr":"172.21.15.98:6827","nonce":2893994080}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6828","nonce":2893994080},{"type":"v1","addr":"172.21.15.98:6829","nonce":2893994080}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6832","nonce":2893994080},{"type":"v1","addr":"172.21.15.98:6833","nonce":2893994080}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6830","nonce":2893994080},{"type":"v1","addr":"172.21.15.98:6831","nonce":2893994080}]},"public_addr":"172.21.15.98:6827/2893994080","cluster_addr":"172.21.15.98:6829/2893994080","heartbeat_back_addr":"172.21.15.98:6833/2893994080","heartbeat_front_addr":"172.21.15.98:6831/2893994080","state":["exists","up"]},{"osd":10,"uuid":"326f0209-ba9d-4b59-97cc-d61a758a8f29","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":70,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.101:6824","nonce":3545244225},{"type":"v1","addr":"172.21.15.101:6825","nonce":3545244225}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.101:6826","nonce":3545244225},{"type":"v1","addr":"172.21.15.101:6827","nonce":3545244225}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.101:6830","nonce":3545244225},{"type":"v1","addr":"172.21.15.101:6831","nonce":3545244225}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.101:6828","nonce":3545244225},{"type":"v1","addr":"172.21.15.101:6829","nonce":3545244225}]},"public_addr":"172.21.15.101:6825/3545244225","cluster_addr":"172.21.15.101:6827/3545244225","heartbeat_back_addr":"172.21.15.101:6831/3545244225","heartbeat_front_addr":"172.21.15.101:6829/3545244225","state":["exists","up"]},{"osd":11,"uuid":"c61c0320-1057-4f17-9bf1-7ad306ed7ed3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":78,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":2630023454},{"type":"v1","addr":"172.21.15.139:6825","nonce":2630023454}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6826","nonce":2630023454},{"type":"v1","addr":"172.21.15.139:6827","nonce":2630023454}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":2630023454},{"type":"v1","addr":"172.21.15.139:6831","nonce":2630023454}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":2630023454},{"type":"v1","addr":"172.21.15.139:6829","nonce":2630023454}]},"public_addr":"172.21.15.139:6825/2630023454","cluster_addr":"172.21.15.139:6827/2630023454","heartbeat_back_addr":"172.21.15.139:6831/2630023454","heartbeat_front_addr":"172.21.15.139:6829/2630023454","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T19:24:23.620028+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T19:24:45.779846+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T19:25:06.877949+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T19:25:27.994199+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T19:25:49.461139+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T19:26:10.409282+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T19:26:32.116390+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T19:26:53.536474+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T19:27:14.384839+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T19:27:35.635921+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T19:27:56.858744+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T19:28:18.342519+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.4","mappings":[{"from":2,"to":11}]},{"pgid":"2.f","mappings":[{"from":8,"to":11}]},{"pgid":"3.0","mappings":[{"from":4,"to":8}]},{"pgid":"3.1","mappings":[{"from":4,"to":9}]},{"pgid":"3.c","mappings":[{"from":3,"to":9}]},{"pgid":"3.17","mappings":[{"from":4,"to":6}]},{"pgid":"3.1e","mappings":[{"from":3,"to":9}]},{"pgid":"3.28","mappings":[{"from":5,"to":7}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.98:0/2164590253":"2023-07-15T19:23:01.520302+0000","172.21.15.98:0/2094734896":"2023-07-15T19:23:01.520302+0000","172.21.15.98:0/784745009":"2023-07-15T19:22:39.481502+0000","172.21.15.98:0/3657190818":"2023-07-15T19:22:39.481502+0000","172.21.15.98:6800/1732769087":"2023-07-15T19:23:01.520302+0000","172.21.15.98:0/3393542419":"2023-07-15T19:22:39.481502+0000","172.21.15.98:6800/749234665":"2023-07-15T19:22:39.481502+0000","172.21.15.98:6801/1732769087":"2023-07-15T19:23:01.520302+0000","172.21.15.98:6801/749234665":"2023-07-15T19:22:39.481502+0000","172.21.15.98:0/187822873":"2023-07-15T19:23:01.520302+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-07-14T19:30:26.682 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:26.681+0000 7fe3c4a5d700 1 -- 172.21.15.98:0/861502722 >> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] conn(0x7fe3a80a1220 msgr2=0x7fe3a80a36d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:30:26.682 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:26.681+0000 7fe3c4a5d700 1 --2- 172.21.15.98:0/861502722 >> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] conn(0x7fe3a80a1220 0x7fe3a80a36d0 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7fe3b4006010 tx=0x7fe3b400b540 comp rx=0 tx=0).stop 2023-07-14T19:30:26.682 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:26.681+0000 7fe3c4a5d700 1 -- 172.21.15.98:0/861502722 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe3c00ffe00 msgr2=0x7fe3c019efc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:30:26.682 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:26.681+0000 7fe3c4a5d700 1 --2- 172.21.15.98:0/861502722 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe3c00ffe00 0x7fe3c019efc0 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 crypto rx=0x7fe3b800b970 tx=0x7fe3b800bd30 comp rx=0 tx=0).stop 2023-07-14T19:30:26.683 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:26.681+0000 7fe3c4a5d700 1 -- 172.21.15.98:0/861502722 shutdown_connections 2023-07-14T19:30:26.683 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:26.681+0000 7fe3c4a5d700 1 --2- 172.21.15.98:0/861502722 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe3c00ffe00 0x7fe3c019efc0 unknown :-1 s=CLOSED pgs=396 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:30:26.683 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:26.681+0000 7fe3c4a5d700 1 --2- 172.21.15.98:0/861502722 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7fe3c00fe280 0x7fe3c00fbc10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:30:26.683 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:26.681+0000 7fe3c4a5d700 1 --2- 172.21.15.98:0/861502722 >> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] conn(0x7fe3a80a1220 0x7fe3a80a36d0 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:30:26.684 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:26.681+0000 7fe3c4a5d700 1 --2- 172.21.15.98:0/861502722 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fe3c00ff470 0x7fe3c019ca70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:30:26.684 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:26.681+0000 7fe3c4a5d700 1 -- 172.21.15.98:0/861502722 >> 172.21.15.98:0/861502722 conn(0x7fe3c00f98a0 msgr2=0x7fe3c0102f30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:30:26.684 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:26.681+0000 7fe3c4a5d700 1 -- 172.21.15.98:0/861502722 shutdown_connections 2023-07-14T19:30:26.684 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:26.681+0000 7fe3c4a5d700 1 -- 172.21.15.98:0/861502722 wait complete. 2023-07-14T19:30:26.695 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-07-14T19:25:09.845827+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2023-07-14T19:28:58.952921+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '82', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.8799999952316284, 'score_stable': 1.8799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2023-07-14T19:29:00.228272+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '83', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.690000057220459, 'score_stable': 1.690000057220459, 'optimal_score': 1, 'raw_score_acting': 1.690000057220459, 'raw_score_stable': 1.690000057220459, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-07-14T19:30:26.696 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-07-14T19:30:26.821 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:26.821+0000 7fa70a176700 1 -- 172.21.15.98:0/3109284812 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa704102fb0 msgr2=0x7fa704105390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:30:26.822 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:26.821+0000 7fa70a176700 1 --2- 172.21.15.98:0/3109284812 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa704102fb0 0x7fa704105390 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 crypto rx=0x7fa6ec009990 tx=0x7fa6ec009ca0 comp rx=0 tx=0).stop 2023-07-14T19:30:26.822 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:26.821+0000 7fa70a176700 1 -- 172.21.15.98:0/3109284812 shutdown_connections 2023-07-14T19:30:26.822 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:26.821+0000 7fa70a176700 1 --2- 172.21.15.98:0/3109284812 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fa7041058d0 0x7fa704107d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:30:26.822 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:26.821+0000 7fa70a176700 1 --2- 172.21.15.98:0/3109284812 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7fa704100690 0x7fa704102a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:30:26.823 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:26.821+0000 7fa70a176700 1 --2- 172.21.15.98:0/3109284812 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa704102fb0 0x7fa704105390 unknown :-1 s=CLOSED pgs=397 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:30:26.823 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:26.821+0000 7fa70a176700 1 -- 172.21.15.98:0/3109284812 >> 172.21.15.98:0/3109284812 conn(0x7fa7040fa130 msgr2=0x7fa7040fc560 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:30:26.823 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:26.821+0000 7fa70a176700 1 -- 172.21.15.98:0/3109284812 shutdown_connections 2023-07-14T19:30:26.823 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:26.821+0000 7fa70a176700 1 -- 172.21.15.98:0/3109284812 wait complete. 2023-07-14T19:30:26.825 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:26.821+0000 7fa70a176700 1 Processor -- start 2023-07-14T19:30:26.825 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:26.822+0000 7fa70a176700 1 -- start start 2023-07-14T19:30:26.826 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:26.822+0000 7fa70a176700 1 --2- >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7fa704100690 0x7fa704100000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:30:26.826 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:26.822+0000 7fa70a176700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa704102fb0 0x7fa7040fe640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:30:26.826 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:26.822+0000 7fa70a176700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fa7041058d0 0x7fa7040feb80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:30:26.826 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:26.822+0000 7fa70a176700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa7040ff320 con 0x7fa704102fb0 2023-07-14T19:30:26.827 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:26.822+0000 7fa70a176700 1 -- --> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa7040ff490 con 0x7fa704100690 2023-07-14T19:30:26.827 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:26.822+0000 7fa70a176700 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa70419f9c0 con 0x7fa7041058d0 2023-07-14T19:30:26.827 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:26.822+0000 7fa702ffd700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa704102fb0 0x7fa7040fe640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:30:26.827 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:26.822+0000 7fa702ffd700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa704102fb0 0x7fa7040fe640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:47028/0 (socket says 172.21.15.98:47028) 2023-07-14T19:30:26.827 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:26.822+0000 7fa702ffd700 1 -- 172.21.15.98:0/4053343784 learned_addr learned my addr 172.21.15.98:0/4053343784 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-07-14T19:30:26.828 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:26.822+0000 7fa7037fe700 1 --2- 172.21.15.98:0/4053343784 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7fa704100690 0x7fa704100000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:30:26.828 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:26.822+0000 7fa703fff700 1 --2- 172.21.15.98:0/4053343784 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fa7041058d0 0x7fa7040feb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:30:26.828 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:26.822+0000 7fa702ffd700 1 -- 172.21.15.98:0/4053343784 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fa7041058d0 msgr2=0x7fa7040feb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:30:26.828 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:26.822+0000 7fa702ffd700 1 --2- 172.21.15.98:0/4053343784 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fa7041058d0 0x7fa7040feb80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:30:26.828 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:26.822+0000 7fa702ffd700 1 -- 172.21.15.98:0/4053343784 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7fa704100690 msgr2=0x7fa704100000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:30:26.829 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:26.822+0000 7fa702ffd700 1 --2- 172.21.15.98:0/4053343784 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7fa704100690 0x7fa704100000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:30:26.829 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:26.822+0000 7fa702ffd700 1 -- 172.21.15.98:0/4053343784 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa6ec0095f0 con 0x7fa704102fb0 2023-07-14T19:30:26.829 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:26.822+0000 7fa702ffd700 1 --2- 172.21.15.98:0/4053343784 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa704102fb0 0x7fa7040fe640 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 crypto rx=0x7fa6ec0059d0 tx=0x7fa6ec0047a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:30:26.829 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:26.822+0000 7fa700ff9700 1 -- 172.21.15.98:0/4053343784 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa6ec01d070 con 0x7fa704102fb0 2023-07-14T19:30:26.829 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:26.823+0000 7fa70a176700 1 -- 172.21.15.98:0/4053343784 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa70419fc40 con 0x7fa704102fb0 2023-07-14T19:30:26.830 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:26.823+0000 7fa700ff9700 1 -- 172.21.15.98:0/4053343784 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fa6ec00fc40 con 0x7fa704102fb0 2023-07-14T19:30:26.830 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:26.823+0000 7fa700ff9700 1 -- 172.21.15.98:0/4053343784 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa6ec017650 con 0x7fa704102fb0 2023-07-14T19:30:26.830 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:26.823+0000 7fa70a176700 1 -- 172.21.15.98:0/4053343784 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa7041a0060 con 0x7fa704102fb0 2023-07-14T19:30:26.830 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:26.824+0000 7fa70a176700 1 -- 172.21.15.98:0/4053343784 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa7040fbca0 con 0x7fa704102fb0 2023-07-14T19:30:26.830 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:26.824+0000 7fa700ff9700 1 -- 172.21.15.98:0/4053343784 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 17) v1 ==== 135766+0+0 (secure 0 0 0) 0x7fa6ec022470 con 0x7fa704102fb0 2023-07-14T19:30:26.830 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:26.825+0000 7fa700ff9700 1 --2- 172.21.15.98:0/4053343784 >> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] conn(0x7fa6f00a11e0 0x7fa6f00a3690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:30:26.831 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:26.825+0000 7fa7037fe700 1 --2- 172.21.15.98:0/4053343784 >> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] conn(0x7fa6f00a11e0 0x7fa6f00a3690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:30:26.831 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:26.825+0000 7fa700ff9700 1 -- 172.21.15.98:0/4053343784 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 8211+0+0 (secure 0 0 0) 0x7fa6ec0cdfe0 con 0x7fa704102fb0 2023-07-14T19:30:26.832 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:26.825+0000 7fa7037fe700 1 --2- 172.21.15.98:0/4053343784 >> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] conn(0x7fa6f00a11e0 0x7fa6f00a3690 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7fa6fc00f540 tx=0x7fa6fc00b040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:30:26.832 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:26.829+0000 7fa700ff9700 1 -- 172.21.15.98:0/4053343784 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7fa6ec097710 con 0x7fa704102fb0 2023-07-14T19:30:27.019 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:27.019+0000 7fa70a176700 1 -- 172.21.15.98:0/4053343784 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fa70410b680 con 0x7fa704102fb0 2023-07-14T19:30:27.020 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:27.019+0000 7fa700ff9700 1 -- 172.21.15.98:0/4053343784 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v85) v1 ==== 93+0+10 (secure 0 0 0) 0x7fa6ec09ad30 con 0x7fa704102fb0 2023-07-14T19:30:27.020 INFO:teuthology.orchestra.run.smithi098.stdout:pg_num: 1 2023-07-14T19:30:27.021 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:27.021+0000 7fa70a176700 1 -- 172.21.15.98:0/4053343784 >> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] conn(0x7fa6f00a11e0 msgr2=0x7fa6f00a3690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:30:27.022 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:27.021+0000 7fa70a176700 1 --2- 172.21.15.98:0/4053343784 >> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] conn(0x7fa6f00a11e0 0x7fa6f00a3690 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7fa6fc00f540 tx=0x7fa6fc00b040 comp rx=0 tx=0).stop 2023-07-14T19:30:27.022 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:27.021+0000 7fa70a176700 1 -- 172.21.15.98:0/4053343784 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa704102fb0 msgr2=0x7fa7040fe640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:30:27.022 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:27.021+0000 7fa70a176700 1 --2- 172.21.15.98:0/4053343784 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa704102fb0 0x7fa7040fe640 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 crypto rx=0x7fa6ec0059d0 tx=0x7fa6ec0047a0 comp rx=0 tx=0).stop 2023-07-14T19:30:27.023 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:27.021+0000 7fa70a176700 1 -- 172.21.15.98:0/4053343784 shutdown_connections 2023-07-14T19:30:27.023 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:27.021+0000 7fa70a176700 1 --2- 172.21.15.98:0/4053343784 >> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] conn(0x7fa6f00a11e0 0x7fa6f00a3690 unknown :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:30:27.023 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:27.021+0000 7fa70a176700 1 --2- 172.21.15.98:0/4053343784 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fa7041058d0 0x7fa7040feb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:30:27.024 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:27.021+0000 7fa70a176700 1 --2- 172.21.15.98:0/4053343784 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7fa704100690 0x7fa704100000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:30:27.024 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:27.021+0000 7fa70a176700 1 --2- 172.21.15.98:0/4053343784 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa704102fb0 0x7fa7040fe640 unknown :-1 s=CLOSED pgs=398 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:30:27.024 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:27.021+0000 7fa70a176700 1 -- 172.21.15.98:0/4053343784 >> 172.21.15.98:0/4053343784 conn(0x7fa7040fa130 msgr2=0x7fa7041012f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:30:27.024 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:27.021+0000 7fa70a176700 1 -- 172.21.15.98:0/4053343784 shutdown_connections 2023-07-14T19:30:27.024 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:27.021+0000 7fa70a176700 1 -- 172.21.15.98:0/4053343784 wait complete. 2023-07-14T19:30:27.030 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:30:26 smithi098 ceph-mon[127486]: pgmap v297: 97 pgs: 97 active+clean; 639 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 597 B/s wr, 6 op/s 2023-07-14T19:30:27.031 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:30:26 smithi098 ceph-mon[127486]: from='client.? 172.21.15.98:0/861502722' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-07-14T19:30:27.034 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-07-14T19:30:27.160 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:27.157+0000 7f9787a94700 1 -- 172.21.15.98:0/1491782555 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9780100690 msgr2=0x7f9780102a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:30:27.160 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:27.157+0000 7f9787a94700 1 --2- 172.21.15.98:0/1491782555 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9780100690 0x7f9780102a70 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 crypto rx=0x7f977c009990 tx=0x7f977c009ca0 comp rx=0 tx=0).stop 2023-07-14T19:30:27.160 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:27.157+0000 7f9787a94700 1 -- 172.21.15.98:0/1491782555 shutdown_connections 2023-07-14T19:30:27.161 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:27.157+0000 7f9787a94700 1 --2- 172.21.15.98:0/1491782555 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f97801058d0 0x7f9780107d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:30:27.161 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:27.157+0000 7f9787a94700 1 --2- 172.21.15.98:0/1491782555 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9780100690 0x7f9780102a70 unknown :-1 s=CLOSED pgs=399 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:30:27.161 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:27.157+0000 7f9787a94700 1 --2- 172.21.15.98:0/1491782555 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f9780102fb0 0x7f9780105390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:30:27.161 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:27.157+0000 7f9787a94700 1 -- 172.21.15.98:0/1491782555 >> 172.21.15.98:0/1491782555 conn(0x7f97800fa130 msgr2=0x7f97800fc560 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:30:27.161 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:27.157+0000 7f9787a94700 1 -- 172.21.15.98:0/1491782555 shutdown_connections 2023-07-14T19:30:27.162 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:27.158+0000 7f9787a94700 1 -- 172.21.15.98:0/1491782555 wait complete. 2023-07-14T19:30:27.162 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:27.158+0000 7f9787a94700 1 Processor -- start 2023-07-14T19:30:27.162 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:27.158+0000 7f9787a94700 1 -- start start 2023-07-14T19:30:27.162 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:27.158+0000 7f9787a94700 1 --2- >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f9780100690 0x7f9780195990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:30:27.162 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:27.158+0000 7f9787a94700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9780102fb0 0x7f9780195ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:30:27.163 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:27.158+0000 7f9787a94700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f97801058d0 0x7f978019cf40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:30:27.163 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:27.158+0000 7f9787a94700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f978019d510 con 0x7f9780102fb0 2023-07-14T19:30:27.163 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:27.158+0000 7f9787a94700 1 -- --> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f978019d650 con 0x7f9780100690 2023-07-14T19:30:27.163 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:27.158+0000 7f9787a94700 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f978019d7e0 con 0x7f97801058d0 2023-07-14T19:30:27.163 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:27.158+0000 7f978502f700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9780102fb0 0x7f9780195ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:30:27.163 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:27.158+0000 7f978502f700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9780102fb0 0x7f9780195ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:47042/0 (socket says 172.21.15.98:47042) 2023-07-14T19:30:27.164 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:27.158+0000 7f978502f700 1 -- 172.21.15.98:0/1298141492 learned_addr learned my addr 172.21.15.98:0/1298141492 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-07-14T19:30:27.164 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:27.159+0000 7f9785830700 1 --2- 172.21.15.98:0/1298141492 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f9780100690 0x7f9780195990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:30:27.165 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:27.159+0000 7f9786031700 1 --2- 172.21.15.98:0/1298141492 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f97801058d0 0x7f978019cf40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:30:27.165 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:27.159+0000 7f978502f700 1 -- 172.21.15.98:0/1298141492 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f97801058d0 msgr2=0x7f978019cf40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:30:27.166 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:27.159+0000 7f978502f700 1 --2- 172.21.15.98:0/1298141492 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f97801058d0 0x7f978019cf40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:30:27.166 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:27.159+0000 7f978502f700 1 -- 172.21.15.98:0/1298141492 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f9780100690 msgr2=0x7f9780195990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:30:27.166 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:27.159+0000 7f978502f700 1 --2- 172.21.15.98:0/1298141492 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f9780100690 0x7f9780195990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:30:27.167 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:27.159+0000 7f978502f700 1 -- 172.21.15.98:0/1298141492 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f977c0095f0 con 0x7f9780102fb0 2023-07-14T19:30:27.167 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:27.159+0000 7f9785830700 1 --2- 172.21.15.98:0/1298141492 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f9780100690 0x7f9780195990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-07-14T19:30:27.168 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:27.159+0000 7f9786031700 1 --2- 172.21.15.98:0/1298141492 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f97801058d0 0x7f978019cf40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-07-14T19:30:27.168 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:27.159+0000 7f978502f700 1 --2- 172.21.15.98:0/1298141492 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9780102fb0 0x7f9780195ed0 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 crypto rx=0x7f977000c920 tx=0x7f977000cce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:30:27.169 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:27.159+0000 7f9776ffd700 1 -- 172.21.15.98:0/1298141492 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9770007b00 con 0x7f9780102fb0 2023-07-14T19:30:27.169 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:27.159+0000 7f9776ffd700 1 -- 172.21.15.98:0/1298141492 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f9770004cf0 con 0x7f9780102fb0 2023-07-14T19:30:27.170 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:27.159+0000 7f9776ffd700 1 -- 172.21.15.98:0/1298141492 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9770005610 con 0x7f9780102fb0 2023-07-14T19:30:27.170 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:27.159+0000 7f9787a94700 1 -- 172.21.15.98:0/1298141492 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f978019dac0 con 0x7f9780102fb0 2023-07-14T19:30:27.170 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:27.159+0000 7f9787a94700 1 -- 172.21.15.98:0/1298141492 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f978019df80 con 0x7f9780102fb0 2023-07-14T19:30:27.172 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:27.160+0000 7f9787a94700 1 -- 172.21.15.98:0/1298141492 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f97800fbca0 con 0x7f9780102fb0 2023-07-14T19:30:27.172 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:27.161+0000 7f9776ffd700 1 -- 172.21.15.98:0/1298141492 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 17) v1 ==== 135766+0+0 (secure 0 0 0) 0x7f9770004750 con 0x7f9780102fb0 2023-07-14T19:30:27.173 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:27.162+0000 7f9776ffd700 1 --2- 172.21.15.98:0/1298141492 >> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] conn(0x7f976c0a11e0 0x7f976c0a3690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:30:27.173 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:27.162+0000 7f9785830700 1 --2- 172.21.15.98:0/1298141492 >> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] conn(0x7f976c0a11e0 0x7f976c0a3690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:30:27.173 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:27.162+0000 7f9776ffd700 1 -- 172.21.15.98:0/1298141492 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 8211+0+0 (secure 0 0 0) 0x7f97700cca90 con 0x7f9780102fb0 2023-07-14T19:30:27.174 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:27.162+0000 7f9785830700 1 --2- 172.21.15.98:0/1298141492 >> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] conn(0x7f976c0a11e0 0x7f976c0a3690 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f977c00b5c0 tx=0x7f977c005fb0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:30:27.174 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:27.165+0000 7f9776ffd700 1 -- 172.21.15.98:0/1298141492 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f9770096030 con 0x7f9780102fb0 2023-07-14T19:30:27.191 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:30:26 smithi101 ceph-mon[128807]: pgmap v297: 97 pgs: 97 active+clean; 639 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 597 B/s wr, 6 op/s 2023-07-14T19:30:27.191 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:30:26 smithi101 ceph-mon[128807]: from='client.? 172.21.15.98:0/861502722' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-07-14T19:30:27.220 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:30:26 smithi139 ceph-mon[129293]: pgmap v297: 97 pgs: 97 active+clean; 639 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 597 B/s wr, 6 op/s 2023-07-14T19:30:27.221 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:30:26 smithi139 ceph-mon[129293]: from='client.? 172.21.15.98:0/861502722' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-07-14T19:30:27.356 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:27.356+0000 7f9787a94700 1 -- 172.21.15.98:0/1298141492 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f978002ca70 con 0x7f9780102fb0 2023-07-14T19:30:27.357 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:27.356+0000 7f9776ffd700 1 -- 172.21.15.98:0/1298141492 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v85) v1 ==== 104+0+11 (secure 0 0 0) 0x7f977001d0a0 con 0x7f9780102fb0 2023-07-14T19:30:27.357 INFO:teuthology.orchestra.run.smithi098.stdout:pg_num: 32 2023-07-14T19:30:27.358 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:27.358+0000 7f9787a94700 1 -- 172.21.15.98:0/1298141492 >> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] conn(0x7f976c0a11e0 msgr2=0x7f976c0a3690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:30:27.359 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:27.358+0000 7f9787a94700 1 --2- 172.21.15.98:0/1298141492 >> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] conn(0x7f976c0a11e0 0x7f976c0a3690 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f977c00b5c0 tx=0x7f977c005fb0 comp rx=0 tx=0).stop 2023-07-14T19:30:27.359 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:27.358+0000 7f9787a94700 1 -- 172.21.15.98:0/1298141492 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9780102fb0 msgr2=0x7f9780195ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:30:27.359 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:27.358+0000 7f9787a94700 1 --2- 172.21.15.98:0/1298141492 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9780102fb0 0x7f9780195ed0 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 crypto rx=0x7f977000c920 tx=0x7f977000cce0 comp rx=0 tx=0).stop 2023-07-14T19:30:27.359 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:27.358+0000 7f9787a94700 1 -- 172.21.15.98:0/1298141492 shutdown_connections 2023-07-14T19:30:27.360 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:27.358+0000 7f9787a94700 1 --2- 172.21.15.98:0/1298141492 >> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] conn(0x7f976c0a11e0 0x7f976c0a3690 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:30:27.360 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:27.358+0000 7f9787a94700 1 --2- 172.21.15.98:0/1298141492 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f97801058d0 0x7f978019cf40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:30:27.360 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:27.358+0000 7f9787a94700 1 --2- 172.21.15.98:0/1298141492 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f9780100690 0x7f9780195990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:30:27.360 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:27.358+0000 7f9787a94700 1 --2- 172.21.15.98:0/1298141492 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9780102fb0 0x7f9780195ed0 unknown :-1 s=CLOSED pgs=400 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:30:27.361 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:27.358+0000 7f9787a94700 1 -- 172.21.15.98:0/1298141492 >> 172.21.15.98:0/1298141492 conn(0x7f97800fa130 msgr2=0x7f9780101310 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:30:27.361 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:27.358+0000 7f9787a94700 1 -- 172.21.15.98:0/1298141492 shutdown_connections 2023-07-14T19:30:27.362 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:27.358+0000 7f9787a94700 1 -- 172.21.15.98:0/1298141492 wait complete. 2023-07-14T19:30:27.371 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-07-14T19:30:27.495 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:27.494+0000 7fb8d887e700 1 -- 172.21.15.98:0/4217677088 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb8d41000e0 msgr2=0x7fb8d41004f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:30:27.495 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:27.494+0000 7fb8d887e700 1 --2- 172.21.15.98:0/4217677088 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb8d41000e0 0x7fb8d41004f0 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 crypto rx=0x7fb8c4009990 tx=0x7fb8c4009ca0 comp rx=0 tx=0).stop 2023-07-14T19:30:27.496 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:27.495+0000 7fb8d887e700 1 -- 172.21.15.98:0/4217677088 shutdown_connections 2023-07-14T19:30:27.496 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:27.495+0000 7fb8d887e700 1 --2- 172.21.15.98:0/4217677088 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fb8d40ff710 0x7fb8d4106fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:30:27.496 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:27.495+0000 7fb8d887e700 1 --2- 172.21.15.98:0/4217677088 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb8d41000e0 0x7fb8d41004f0 unknown :-1 s=CLOSED pgs=401 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:30:27.496 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:27.495+0000 7fb8d887e700 1 --2- 172.21.15.98:0/4217677088 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7fb8d40fed80 0x7fb8d40ff1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:30:27.497 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:27.495+0000 7fb8d887e700 1 -- 172.21.15.98:0/4217677088 >> 172.21.15.98:0/4217677088 conn(0x7fb8d40f98a0 msgr2=0x7fb8d40fbcd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:30:27.497 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:27.495+0000 7fb8d887e700 1 -- 172.21.15.98:0/4217677088 shutdown_connections 2023-07-14T19:30:27.497 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:27.495+0000 7fb8d887e700 1 -- 172.21.15.98:0/4217677088 wait complete. 2023-07-14T19:30:27.499 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:27.495+0000 7fb8d887e700 1 Processor -- start 2023-07-14T19:30:27.499 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:27.495+0000 7fb8d887e700 1 -- start start 2023-07-14T19:30:27.500 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:27.495+0000 7fb8d887e700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb8d40fed80 0x7fb8d41948b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:30:27.500 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:27.495+0000 7fb8d887e700 1 --2- >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7fb8d40ff710 0x7fb8d4194df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:30:27.500 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:27.495+0000 7fb8d887e700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fb8d41000e0 0x7fb8d419be60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:30:27.500 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:27.495+0000 7fb8d887e700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb8d419c430 con 0x7fb8d40fed80 2023-07-14T19:30:27.501 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:27.495+0000 7fb8d887e700 1 -- --> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb8d419c570 con 0x7fb8d40ff710 2023-07-14T19:30:27.501 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:27.495+0000 7fb8d887e700 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb8d419c700 con 0x7fb8d41000e0 2023-07-14T19:30:27.501 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:27.496+0000 7fb8d259c700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb8d40fed80 0x7fb8d41948b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:30:27.502 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:27.496+0000 7fb8d259c700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb8d40fed80 0x7fb8d41948b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:47060/0 (socket says 172.21.15.98:47060) 2023-07-14T19:30:27.502 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:27.496+0000 7fb8d259c700 1 -- 172.21.15.98:0/1559145163 learned_addr learned my addr 172.21.15.98:0/1559145163 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-07-14T19:30:27.502 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:27.496+0000 7fb8d2d9d700 1 --2- 172.21.15.98:0/1559145163 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fb8d41000e0 0x7fb8d419be60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:30:27.502 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:27.496+0000 7fb8d1d9b700 1 --2- 172.21.15.98:0/1559145163 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7fb8d40ff710 0x7fb8d4194df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:30:27.503 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:27.496+0000 7fb8d259c700 1 -- 172.21.15.98:0/1559145163 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fb8d41000e0 msgr2=0x7fb8d419be60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:30:27.503 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:27.496+0000 7fb8d259c700 1 --2- 172.21.15.98:0/1559145163 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fb8d41000e0 0x7fb8d419be60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:30:27.503 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:27.496+0000 7fb8d259c700 1 -- 172.21.15.98:0/1559145163 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7fb8d40ff710 msgr2=0x7fb8d4194df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:30:27.503 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:27.496+0000 7fb8d259c700 1 --2- 172.21.15.98:0/1559145163 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7fb8d40ff710 0x7fb8d4194df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:30:27.504 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:27.496+0000 7fb8d259c700 1 -- 172.21.15.98:0/1559145163 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb8c40095f0 con 0x7fb8d40fed80 2023-07-14T19:30:27.504 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:27.496+0000 7fb8d259c700 1 --2- 172.21.15.98:0/1559145163 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb8d40fed80 0x7fb8d41948b0 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 crypto rx=0x7fb8c40095c0 tx=0x7fb8c400f690 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:30:27.504 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:27.496+0000 7fb8c37fe700 1 -- 172.21.15.98:0/1559145163 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb8c401d070 con 0x7fb8d40fed80 2023-07-14T19:30:27.504 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:27.496+0000 7fb8d887e700 1 -- 172.21.15.98:0/1559145163 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb8d419c980 con 0x7fb8d40fed80 2023-07-14T19:30:27.504 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:27.496+0000 7fb8c37fe700 1 -- 172.21.15.98:0/1559145163 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fb8c4022470 con 0x7fb8d40fed80 2023-07-14T19:30:27.505 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:27.496+0000 7fb8c37fe700 1 -- 172.21.15.98:0/1559145163 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb8c4017650 con 0x7fb8d40fed80 2023-07-14T19:30:27.505 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:27.496+0000 7fb8d887e700 1 -- 172.21.15.98:0/1559145163 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb8d419cda0 con 0x7fb8d40fed80 2023-07-14T19:30:27.505 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:27.497+0000 7fb8d887e700 1 -- 172.21.15.98:0/1559145163 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb8d4105b20 con 0x7fb8d40fed80 2023-07-14T19:30:27.505 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:27.498+0000 7fb8c37fe700 1 -- 172.21.15.98:0/1559145163 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 17) v1 ==== 135766+0+0 (secure 0 0 0) 0x7fb8c400fad0 con 0x7fb8d40fed80 2023-07-14T19:30:27.506 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:27.499+0000 7fb8c37fe700 1 --2- 172.21.15.98:0/1559145163 >> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] conn(0x7fb8bc0a11e0 0x7fb8bc0a3690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:30:27.506 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:27.499+0000 7fb8c37fe700 1 -- 172.21.15.98:0/1559145163 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 8211+0+0 (secure 0 0 0) 0x7fb8c40cde90 con 0x7fb8d40fed80 2023-07-14T19:30:27.506 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:27.499+0000 7fb8d1d9b700 1 --2- 172.21.15.98:0/1559145163 >> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] conn(0x7fb8bc0a11e0 0x7fb8bc0a3690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:30:27.507 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:27.499+0000 7fb8d1d9b700 1 --2- 172.21.15.98:0/1559145163 >> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] conn(0x7fb8bc0a11e0 0x7fb8bc0a3690 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7fb8c800bbd0 tx=0x7fb8c800b5b0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:30:27.507 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:27.503+0000 7fb8c37fe700 1 -- 172.21.15.98:0/1559145163 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7fb8c4097640 con 0x7fb8d40fed80 2023-07-14T19:30:27.694 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:27.694+0000 7fb8d887e700 1 -- 172.21.15.98:0/1559145163 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7fb8d402cb00 con 0x7fb8d40fed80 2023-07-14T19:30:27.695 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:27.694+0000 7fb8c37fe700 1 -- 172.21.15.98:0/1559145163 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v85) v1 ==== 100+0+11 (secure 0 0 0) 0x7fb8c4027020 con 0x7fb8d40fed80 2023-07-14T19:30:27.695 INFO:teuthology.orchestra.run.smithi098.stdout:pg_num: 64 2023-07-14T19:30:27.697 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:27.696+0000 7fb8d887e700 1 -- 172.21.15.98:0/1559145163 >> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] conn(0x7fb8bc0a11e0 msgr2=0x7fb8bc0a3690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:30:27.697 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:27.696+0000 7fb8d887e700 1 --2- 172.21.15.98:0/1559145163 >> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] conn(0x7fb8bc0a11e0 0x7fb8bc0a3690 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7fb8c800bbd0 tx=0x7fb8c800b5b0 comp rx=0 tx=0).stop 2023-07-14T19:30:27.698 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:27.696+0000 7fb8d887e700 1 -- 172.21.15.98:0/1559145163 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb8d40fed80 msgr2=0x7fb8d41948b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:30:27.698 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:27.696+0000 7fb8d887e700 1 --2- 172.21.15.98:0/1559145163 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb8d40fed80 0x7fb8d41948b0 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 crypto rx=0x7fb8c40095c0 tx=0x7fb8c400f690 comp rx=0 tx=0).stop 2023-07-14T19:30:27.698 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:27.696+0000 7fb8d887e700 1 -- 172.21.15.98:0/1559145163 shutdown_connections 2023-07-14T19:30:27.698 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:27.696+0000 7fb8d887e700 1 --2- 172.21.15.98:0/1559145163 >> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] conn(0x7fb8bc0a11e0 0x7fb8bc0a3690 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:30:27.699 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:27.697+0000 7fb8d887e700 1 --2- 172.21.15.98:0/1559145163 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fb8d41000e0 0x7fb8d419be60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:30:27.699 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:27.697+0000 7fb8d887e700 1 --2- 172.21.15.98:0/1559145163 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb8d40fed80 0x7fb8d41948b0 unknown :-1 s=CLOSED pgs=402 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:30:27.699 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:27.697+0000 7fb8d887e700 1 --2- 172.21.15.98:0/1559145163 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7fb8d40ff710 0x7fb8d4194df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:30:27.699 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:27.697+0000 7fb8d887e700 1 -- 172.21.15.98:0/1559145163 >> 172.21.15.98:0/1559145163 conn(0x7fb8d40f98a0 msgr2=0x7fb8d40fbb70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:30:27.699 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:27.697+0000 7fb8d887e700 1 -- 172.21.15.98:0/1559145163 shutdown_connections 2023-07-14T19:30:27.700 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:27.697+0000 7fb8d887e700 1 -- 172.21.15.98:0/1559145163 wait complete. 2023-07-14T19:30:27.709 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2023-07-14T19:30:27.832 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:27.832+0000 7f467cb94700 1 -- 172.21.15.98:0/2295176682 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4678101790 msgr2=0x7f4678108050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:30:27.833 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:27.832+0000 7f467cb94700 1 --2- 172.21.15.98:0/2295176682 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4678101790 0x7f4678108050 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 crypto rx=0x7f467000b0a0 tx=0x7f467000b3b0 comp rx=0 tx=0).stop 2023-07-14T19:30:27.833 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:27.832+0000 7f467cb94700 1 -- 172.21.15.98:0/2295176682 shutdown_connections 2023-07-14T19:30:27.833 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:27.832+0000 7f467cb94700 1 --2- 172.21.15.98:0/2295176682 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f46780ffc10 0x7f4678100020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:30:27.834 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:27.832+0000 7f467cb94700 1 --2- 172.21.15.98:0/2295176682 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4678101790 0x7f4678108050 unknown :-1 s=CLOSED pgs=403 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:30:27.834 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:27.832+0000 7f467cb94700 1 --2- 172.21.15.98:0/2295176682 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f4678100e00 0x7f4678101250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:30:27.834 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:27.832+0000 7f467cb94700 1 -- 172.21.15.98:0/2295176682 >> 172.21.15.98:0/2295176682 conn(0x7f46780fb250 msgr2=0x7f46780fd660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:30:27.834 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:27.832+0000 7f467cb94700 1 -- 172.21.15.98:0/2295176682 shutdown_connections 2023-07-14T19:30:27.834 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:27.832+0000 7f467cb94700 1 -- 172.21.15.98:0/2295176682 wait complete. 2023-07-14T19:30:27.836 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:27.833+0000 7f467cb94700 1 Processor -- start 2023-07-14T19:30:27.837 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:27.833+0000 7f467cb94700 1 -- start start 2023-07-14T19:30:27.837 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:27.833+0000 7f467cb94700 1 --2- >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f46780ffc10 0x7f4678195a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:30:27.837 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:27.833+0000 7f467cb94700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4678100e00 0x7f4678195f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:30:27.837 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:27.833+0000 7f467cb94700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f4678101790 0x7f467819cfc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:30:27.838 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:27.833+0000 7f467cb94700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f467819d590 con 0x7f4678100e00 2023-07-14T19:30:27.838 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:27.833+0000 7f467cb94700 1 -- --> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f467819d6d0 con 0x7f46780ffc10 2023-07-14T19:30:27.838 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:27.833+0000 7f467cb94700 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f467819d860 con 0x7f4678101790 2023-07-14T19:30:27.838 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:27.833+0000 7f4675d9b700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4678100e00 0x7f4678195f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:30:27.839 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:27.833+0000 7f4675d9b700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4678100e00 0x7f4678195f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:47076/0 (socket says 172.21.15.98:47076) 2023-07-14T19:30:27.839 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:27.833+0000 7f4675d9b700 1 -- 172.21.15.98:0/1687764691 learned_addr learned my addr 172.21.15.98:0/1687764691 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-07-14T19:30:27.839 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:27.834+0000 7f467659c700 1 --2- 172.21.15.98:0/1687764691 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f46780ffc10 0x7f4678195a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:30:27.839 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:27.834+0000 7f4676d9d700 1 --2- 172.21.15.98:0/1687764691 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f4678101790 0x7f467819cfc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:30:27.839 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:27.834+0000 7f4675d9b700 1 -- 172.21.15.98:0/1687764691 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f4678101790 msgr2=0x7f467819cfc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:30:27.840 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:27.834+0000 7f4675d9b700 1 --2- 172.21.15.98:0/1687764691 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f4678101790 0x7f467819cfc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:30:27.840 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:27.834+0000 7f4675d9b700 1 -- 172.21.15.98:0/1687764691 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f46780ffc10 msgr2=0x7f4678195a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:30:27.840 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:27.834+0000 7f4675d9b700 1 --2- 172.21.15.98:0/1687764691 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f46780ffc10 0x7f4678195a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:30:27.840 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:27.834+0000 7f4675d9b700 1 -- 172.21.15.98:0/1687764691 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4670009d00 con 0x7f4678100e00 2023-07-14T19:30:27.841 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:27.834+0000 7f4676d9d700 1 --2- 172.21.15.98:0/1687764691 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f4678101790 0x7f467819cfc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-07-14T19:30:27.841 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:27.834+0000 7f4675d9b700 1 --2- 172.21.15.98:0/1687764691 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4678100e00 0x7f4678195f50 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 crypto rx=0x7f466800c920 tx=0x7f466800cce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:30:27.841 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:27.834+0000 7f46677fe700 1 -- 172.21.15.98:0/1687764691 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4668007b00 con 0x7f4678100e00 2023-07-14T19:30:27.841 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:27.834+0000 7f46677fe700 1 -- 172.21.15.98:0/1687764691 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f4668004cf0 con 0x7f4678100e00 2023-07-14T19:30:27.842 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:27.834+0000 7f467cb94700 1 -- 172.21.15.98:0/1687764691 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f467819db40 con 0x7f4678100e00 2023-07-14T19:30:27.842 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:27.834+0000 7f46677fe700 1 -- 172.21.15.98:0/1687764691 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4668005610 con 0x7f4678100e00 2023-07-14T19:30:27.842 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:27.834+0000 7f467cb94700 1 -- 172.21.15.98:0/1687764691 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f467819e050 con 0x7f4678100e00 2023-07-14T19:30:27.842 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:27.835+0000 7f467cb94700 1 -- 172.21.15.98:0/1687764691 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f467810b3f0 con 0x7f4678100e00 2023-07-14T19:30:27.842 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:27.836+0000 7f46677fe700 1 -- 172.21.15.98:0/1687764691 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 17) v1 ==== 135766+0+0 (secure 0 0 0) 0x7f4668004750 con 0x7f4678100e00 2023-07-14T19:30:27.843 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:27.837+0000 7f46677fe700 1 --2- 172.21.15.98:0/1687764691 >> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] conn(0x7f46600a11e0 0x7f46600a3690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:30:27.843 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:27.837+0000 7f467659c700 1 --2- 172.21.15.98:0/1687764691 >> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] conn(0x7f46600a11e0 0x7f46600a3690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:30:27.844 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:27.837+0000 7f46677fe700 1 -- 172.21.15.98:0/1687764691 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 8211+0+0 (secure 0 0 0) 0x7f46680cca90 con 0x7f4678100e00 2023-07-14T19:30:27.844 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:27.837+0000 7f467659c700 1 --2- 172.21.15.98:0/1687764691 >> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] conn(0x7f46600a11e0 0x7f46600a3690 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f466c005fd0 tx=0x7f466c005dd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:30:27.845 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:27.840+0000 7f46677fe700 1 -- 172.21.15.98:0/1687764691 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f4668096030 con 0x7f4678100e00 2023-07-14T19:30:28.031 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:30:27 smithi098 ceph-mon[127486]: from='client.? 172.21.15.98:0/4053343784' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2023-07-14T19:30:28.031 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:30:27 smithi098 ceph-mon[127486]: from='client.? 172.21.15.98:0/1298141492' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2023-07-14T19:30:28.031 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:30:27 smithi098 ceph-mon[127486]: from='client.? 172.21.15.98:0/1559145163' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2023-07-14T19:30:28.044 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:28.044+0000 7f467cb94700 1 -- 172.21.15.98:0/1687764691 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7f4678106a90 con 0x7f4678100e00 2023-07-14T19:30:28.045 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:28.044+0000 7f46677fe700 1 -- 172.21.15.98:0/1687764691 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 10 entries v85) v1 ==== 81+0+588 (secure 0 0 0) 0x7f466801d020 con 0x7f4678100e00 2023-07-14T19:30:28.045 INFO:teuthology.orchestra.run.smithi098.stdout:172.21.15.98:0/2164590253 2023-07-15T19:23:01.520302+0000 2023-07-14T19:30:28.045 INFO:teuthology.orchestra.run.smithi098.stdout:172.21.15.98:0/2094734896 2023-07-15T19:23:01.520302+0000 2023-07-14T19:30:28.045 INFO:teuthology.orchestra.run.smithi098.stdout:172.21.15.98:0/784745009 2023-07-15T19:22:39.481502+0000 2023-07-14T19:30:28.046 INFO:teuthology.orchestra.run.smithi098.stdout:172.21.15.98:0/3657190818 2023-07-15T19:22:39.481502+0000 2023-07-14T19:30:28.046 INFO:teuthology.orchestra.run.smithi098.stdout:172.21.15.98:6800/1732769087 2023-07-15T19:23:01.520302+0000 2023-07-14T19:30:28.046 INFO:teuthology.orchestra.run.smithi098.stdout:172.21.15.98:0/3393542419 2023-07-15T19:22:39.481502+0000 2023-07-14T19:30:28.046 INFO:teuthology.orchestra.run.smithi098.stdout:172.21.15.98:6800/749234665 2023-07-15T19:22:39.481502+0000 2023-07-14T19:30:28.046 INFO:teuthology.orchestra.run.smithi098.stdout:172.21.15.98:6801/1732769087 2023-07-15T19:23:01.520302+0000 2023-07-14T19:30:28.047 INFO:teuthology.orchestra.run.smithi098.stdout:172.21.15.98:6801/749234665 2023-07-15T19:22:39.481502+0000 2023-07-14T19:30:28.047 INFO:teuthology.orchestra.run.smithi098.stdout:172.21.15.98:0/187822873 2023-07-15T19:23:01.520302+0000 2023-07-14T19:30:28.048 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:28.046+0000 7f467cb94700 1 -- 172.21.15.98:0/1687764691 >> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] conn(0x7f46600a11e0 msgr2=0x7f46600a3690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:30:28.048 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:28.046+0000 7f467cb94700 1 --2- 172.21.15.98:0/1687764691 >> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] conn(0x7f46600a11e0 0x7f46600a3690 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f466c005fd0 tx=0x7f466c005dd0 comp rx=0 tx=0).stop 2023-07-14T19:30:28.048 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:28.046+0000 7f467cb94700 1 -- 172.21.15.98:0/1687764691 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4678100e00 msgr2=0x7f4678195f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:30:28.048 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:28.046+0000 7f467cb94700 1 --2- 172.21.15.98:0/1687764691 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4678100e00 0x7f4678195f50 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 crypto rx=0x7f466800c920 tx=0x7f466800cce0 comp rx=0 tx=0).stop 2023-07-14T19:30:28.049 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:28.046+0000 7f467cb94700 1 -- 172.21.15.98:0/1687764691 shutdown_connections 2023-07-14T19:30:28.049 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:28.046+0000 7f467cb94700 1 --2- 172.21.15.98:0/1687764691 >> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] conn(0x7f46600a11e0 0x7f46600a3690 unknown :-1 s=CLOSED pgs=240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:30:28.049 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:28.046+0000 7f467cb94700 1 --2- 172.21.15.98:0/1687764691 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f4678101790 0x7f467819cfc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:30:28.049 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:28.046+0000 7f467cb94700 1 --2- 172.21.15.98:0/1687764691 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f46780ffc10 0x7f4678195a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:30:28.049 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:28.046+0000 7f467cb94700 1 --2- 172.21.15.98:0/1687764691 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4678100e00 0x7f4678195f50 unknown :-1 s=CLOSED pgs=404 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:30:28.050 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:28.046+0000 7f467cb94700 1 -- 172.21.15.98:0/1687764691 >> 172.21.15.98:0/1687764691 conn(0x7f46780fb250 msgr2=0x7f46780fd3e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:30:28.050 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:28.046+0000 7f467cb94700 1 -- 172.21.15.98:0/1687764691 shutdown_connections 2023-07-14T19:30:28.050 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:28.046+0000 7f467cb94700 1 -- 172.21.15.98:0/1687764691 wait complete. 2023-07-14T19:30:28.050 INFO:teuthology.orchestra.run.smithi098.stderr:listed 10 entries 2023-07-14T19:30:28.059 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.0... 2023-07-14T19:30:28.059 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-07-14T19:30:28.060 DEBUG:teuthology.orchestra.run.smithi098:> sudo umount /home/ubuntu/cephtest/mnt.0 2023-07-14T19:30:28.133 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi098.front.sepia.ceph.com 2023-07-14T19:30:28.134 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-07-14T19:30:28.134 DEBUG:teuthology.orchestra.run.smithi098:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2023-07-14T19:30:28.168 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-07-14T19:30:28.169 DEBUG:teuthology.orchestra.run.smithi098:> ip netns list 2023-07-14T19:30:28.191 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:30:27 smithi101 ceph-mon[128807]: from='client.? 172.21.15.98:0/4053343784' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2023-07-14T19:30:28.191 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:30:27 smithi101 ceph-mon[128807]: from='client.? 172.21.15.98:0/1298141492' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2023-07-14T19:30:28.192 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:30:27 smithi101 ceph-mon[128807]: from='client.? 172.21.15.98:0/1559145163' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2023-07-14T19:30:28.220 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:30:27 smithi139 ceph-mon[129293]: from='client.? 172.21.15.98:0/4053343784' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2023-07-14T19:30:28.221 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:30:27 smithi139 ceph-mon[129293]: from='client.? 172.21.15.98:0/1298141492' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2023-07-14T19:30:28.221 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:30:27 smithi139 ceph-mon[129293]: from='client.? 172.21.15.98:0/1559145163' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2023-07-14T19:30:28.225 INFO:teuthology.orchestra.run.smithi098.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2023-07-14T19:30:28.225 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-07-14T19:30:28.226 DEBUG:teuthology.orchestra.run.smithi098:> sudo ip netns delete ceph-ns--home-ubuntu-cephtest-mnt.0 2023-07-14T19:30:28.305 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-07-14T19:30:28.306 DEBUG:teuthology.orchestra.run.smithi098:> sudo ip link delete ceph-brx 2023-07-14T19:30:28.421 DEBUG:teuthology.run_tasks:Unwinding manager cephadm 2023-07-14T19:30:28.430 INFO:tasks.cephadm:Teardown begin 2023-07-14T19:30:28.431 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2023-07-14T19:30:28.432 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-07-14T19:30:28.554 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:28.554+0000 7f2fc78c2700 1 -- 172.21.15.98:0/339116077 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2fc00ffcf0 msgr2=0x7f2fc0100140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:30:28.555 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:28.554+0000 7f2fc78c2700 1 --2- 172.21.15.98:0/339116077 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2fc00ffcf0 0x7f2fc0100140 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 crypto rx=0x7f2fb4009990 tx=0x7f2fb4009ca0 comp rx=0 tx=0).stop 2023-07-14T19:30:28.555 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:28.554+0000 7f2fc78c2700 1 -- 172.21.15.98:0/339116077 shutdown_connections 2023-07-14T19:30:28.555 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:28.554+0000 7f2fc78c2700 1 --2- 172.21.15.98:0/339116077 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f2fc00feb00 0x7f2fc00fef10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:30:28.556 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:28.554+0000 7f2fc78c2700 1 --2- 172.21.15.98:0/339116077 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f2fc0100680 0x7f2fc0106f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:30:28.556 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:28.554+0000 7f2fc78c2700 1 --2- 172.21.15.98:0/339116077 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2fc00ffcf0 0x7f2fc0100140 unknown :-1 s=CLOSED pgs=405 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:30:28.556 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:28.554+0000 7f2fc78c2700 1 -- 172.21.15.98:0/339116077 >> 172.21.15.98:0/339116077 conn(0x7f2fc00fa100 msgr2=0x7f2fc00fc550 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:30:28.556 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:28.554+0000 7f2fc78c2700 1 -- 172.21.15.98:0/339116077 shutdown_connections 2023-07-14T19:30:28.556 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:28.554+0000 7f2fc78c2700 1 -- 172.21.15.98:0/339116077 wait complete. 2023-07-14T19:30:28.557 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:28.554+0000 7f2fc78c2700 1 Processor -- start 2023-07-14T19:30:28.557 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:28.554+0000 7f2fc78c2700 1 -- start start 2023-07-14T19:30:28.559 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:28.555+0000 7f2fc78c2700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f2fc00feb00 0x7f2fc0068330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:30:28.559 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:28.555+0000 7f2fc78c2700 1 --2- >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f2fc00ffcf0 0x7f2fc019a6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:30:28.559 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:28.555+0000 7f2fc78c2700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2fc0100680 0x7f2fc019cd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:30:28.560 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:28.555+0000 7f2fc78c2700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2fc019d4d0 con 0x7f2fc0100680 2023-07-14T19:30:28.560 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:28.555+0000 7f2fc78c2700 1 -- --> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2fc019d640 con 0x7f2fc00ffcf0 2023-07-14T19:30:28.560 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:28.555+0000 7f2fc78c2700 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2fc019d800 con 0x7f2fc00feb00 2023-07-14T19:30:28.560 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:28.555+0000 7f2fc5e5f700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2fc0100680 0x7f2fc019cd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:30:28.560 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:28.555+0000 7f2fc5e5f700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2fc0100680 0x7f2fc019cd00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:47094/0 (socket says 172.21.15.98:47094) 2023-07-14T19:30:28.561 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:28.555+0000 7f2fc5e5f700 1 -- 172.21.15.98:0/95834015 learned_addr learned my addr 172.21.15.98:0/95834015 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-07-14T19:30:28.561 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:28.555+0000 7f2fc565e700 1 --2- 172.21.15.98:0/95834015 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f2fc00feb00 0x7f2fc0068330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:30:28.561 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:28.555+0000 7f2fc4e5d700 1 --2- 172.21.15.98:0/95834015 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f2fc00ffcf0 0x7f2fc019a6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:30:28.561 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:28.555+0000 7f2fc5e5f700 1 -- 172.21.15.98:0/95834015 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f2fc00feb00 msgr2=0x7f2fc0068330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:30:28.561 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:28.555+0000 7f2fc5e5f700 1 --2- 172.21.15.98:0/95834015 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f2fc00feb00 0x7f2fc0068330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:30:28.562 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:28.555+0000 7f2fc5e5f700 1 -- 172.21.15.98:0/95834015 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f2fc00ffcf0 msgr2=0x7f2fc019a6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:30:28.562 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:28.555+0000 7f2fc5e5f700 1 --2- 172.21.15.98:0/95834015 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f2fc00ffcf0 0x7f2fc019a6b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:30:28.562 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:28.555+0000 7f2fc5e5f700 1 -- 172.21.15.98:0/95834015 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2fb40095f0 con 0x7f2fc0100680 2023-07-14T19:30:28.562 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:28.555+0000 7f2fc4e5d700 1 --2- 172.21.15.98:0/95834015 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f2fc00ffcf0 0x7f2fc019a6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-07-14T19:30:28.562 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:28.555+0000 7f2fc5e5f700 1 --2- 172.21.15.98:0/95834015 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2fc0100680 0x7f2fc019cd00 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 crypto rx=0x7f2fb8007ee0 tx=0x7f2fb800d3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:30:28.563 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:28.555+0000 7f2fb27fc700 1 -- 172.21.15.98:0/95834015 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2fb8017070 con 0x7f2fc0100680 2023-07-14T19:30:28.563 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:28.555+0000 7f2fb27fc700 1 -- 172.21.15.98:0/95834015 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f2fb800f040 con 0x7f2fc0100680 2023-07-14T19:30:28.563 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:28.555+0000 7f2fc78c2700 1 -- 172.21.15.98:0/95834015 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2fc019dae0 con 0x7f2fc0100680 2023-07-14T19:30:28.563 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:28.555+0000 7f2fb27fc700 1 -- 172.21.15.98:0/95834015 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2fb8005740 con 0x7f2fc0100680 2023-07-14T19:30:28.563 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:28.555+0000 7f2fc78c2700 1 -- 172.21.15.98:0/95834015 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2fc019dff0 con 0x7f2fc0100680 2023-07-14T19:30:28.564 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:28.556+0000 7f2fc78c2700 1 -- 172.21.15.98:0/95834015 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2fc01060a0 con 0x7f2fc0100680 2023-07-14T19:30:28.564 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:28.557+0000 7f2fb27fc700 1 -- 172.21.15.98:0/95834015 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 17) v1 ==== 135766+0+0 (secure 0 0 0) 0x7f2fb80058a0 con 0x7f2fc0100680 2023-07-14T19:30:28.564 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:28.558+0000 7f2fb27fc700 1 --2- 172.21.15.98:0/95834015 >> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] conn(0x7f2fac0a12a0 0x7f2fac0a3750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:30:28.564 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:28.558+0000 7f2fc565e700 1 --2- 172.21.15.98:0/95834015 >> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] conn(0x7f2fac0a12a0 0x7f2fac0a3750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:30:28.564 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:28.558+0000 7f2fb27fc700 1 -- 172.21.15.98:0/95834015 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 8211+0+0 (secure 0 0 0) 0x7f2fb80cc880 con 0x7f2fc0100680 2023-07-14T19:30:28.565 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:28.558+0000 7f2fc565e700 1 --2- 172.21.15.98:0/95834015 >> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] conn(0x7f2fac0a12a0 0x7f2fac0a3750 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f2fbc005950 tx=0x7f2fbc0058e0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:30:28.565 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:28.561+0000 7f2fb27fc700 1 -- 172.21.15.98:0/95834015 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f2fb8095e80 con 0x7f2fc0100680 2023-07-14T19:30:28.795 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:28.794+0000 7f2fc78c2700 1 -- 172.21.15.98:0/95834015 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f2fc002c8e0 con 0x7f2fc0100680 2023-07-14T19:30:28.796 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:28.795+0000 7f2fb27fc700 1 -- 172.21.15.98:0/95834015 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 23 v23) v1 ==== 94+0+10826 (secure 0 0 0) 0x7f2fb80b9030 con 0x7f2fc0100680 2023-07-14T19:30:28.796 INFO:teuthology.orchestra.run.smithi098.stdout: 2023-07-14T19:30:28.796 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":23,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14565,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6835/586280675","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6834","nonce":586280675},{"type":"v1","addr":"172.21.15.98:6835","nonce":586280675}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":4},{"gid":14577,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6837/487577257","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6836","nonce":487577257},{"type":"v1","addr":"172.21.15.98:6837","nonce":487577257}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":5},{"gid":14634,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6839/4085731209","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6838","nonce":4085731209},{"type":"v1","addr":"172.21.15.98:6839","nonce":4085731209}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":11},{"gid":14646,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6841/1679318259","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6840","nonce":1679318259},{"type":"v1","addr":"172.21.15.98:6841","nonce":1679318259}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":12},{"gid":24421,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.139:6833/3505477820","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6832","nonce":3505477820},{"type":"v1","addr":"172.21.15.139:6833","nonce":3505477820}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":24437,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.101:6833/3474460171","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.101:6832","nonce":3474460171},{"type":"v1","addr":"172.21.15.101:6833","nonce":3474460171}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":23,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2023-07-14T19:29:01.033156+0000","modified":"2023-07-14T19:30:20.766354+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24521,"mds_1":24493,"mds_2":24479},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24455":{"gid":24455,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.101:6835/1583264648","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.101:6834","nonce":1583264648},{"type":"v1","addr":"172.21.15.101:6835","nonce":1583264648}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24460":{"gid":24460,"name":"f","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.139:6835/2315281724","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6834","nonce":2315281724},{"type":"v1","addr":"172.21.15.139:6835","nonce":2315281724}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24469":{"gid":24469,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.139:6837/3597660552","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6836","nonce":3597660552},{"type":"v1","addr":"172.21.15.139:6837","nonce":3597660552}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24479":{"gid":24479,"name":"h","rank":2,"incarnation":21,"state":"up:active","state_seq":6,"addr":"172.21.15.101:6837/485741528","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.101:6836","nonce":485741528},{"type":"v1","addr":"172.21.15.101:6837","nonce":485741528}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24493":{"gid":24493,"name":"i","rank":1,"incarnation":19,"state":"up:active","state_seq":5,"addr":"172.21.15.139:6839/1258854143","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6838","nonce":1258854143},{"type":"v1","addr":"172.21.15.139:6839","nonce":1258854143}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24521":{"gid":24521,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.101:6839/48393381","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.101:6838","nonce":48393381},{"type":"v1","addr":"172.21.15.101:6839","nonce":48393381}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-07-14T19:30:28.798 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:28.797+0000 7f2fc78c2700 1 -- 172.21.15.98:0/95834015 >> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] conn(0x7f2fac0a12a0 msgr2=0x7f2fac0a3750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:30:28.798 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:28.797+0000 7f2fc78c2700 1 --2- 172.21.15.98:0/95834015 >> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] conn(0x7f2fac0a12a0 0x7f2fac0a3750 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f2fbc005950 tx=0x7f2fbc0058e0 comp rx=0 tx=0).stop 2023-07-14T19:30:28.799 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:28.797+0000 7f2fc78c2700 1 -- 172.21.15.98:0/95834015 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2fc0100680 msgr2=0x7f2fc019cd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:30:28.799 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:28.797+0000 7f2fc78c2700 1 --2- 172.21.15.98:0/95834015 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2fc0100680 0x7f2fc019cd00 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 crypto rx=0x7f2fb8007ee0 tx=0x7f2fb800d3b0 comp rx=0 tx=0).stop 2023-07-14T19:30:28.800 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:28.797+0000 7f2fc78c2700 1 -- 172.21.15.98:0/95834015 shutdown_connections 2023-07-14T19:30:28.800 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:28.797+0000 7f2fc78c2700 1 --2- 172.21.15.98:0/95834015 >> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] conn(0x7f2fac0a12a0 0x7f2fac0a3750 unknown :-1 s=CLOSED pgs=241 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:30:28.801 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:28.797+0000 7f2fc78c2700 1 --2- 172.21.15.98:0/95834015 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f2fc00feb00 0x7f2fc0068330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:30:28.801 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:28.797+0000 7f2fc78c2700 1 --2- 172.21.15.98:0/95834015 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2fc0100680 0x7f2fc019cd00 unknown :-1 s=CLOSED pgs=406 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:30:28.802 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:28.797+0000 7f2fc78c2700 1 --2- 172.21.15.98:0/95834015 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f2fc00ffcf0 0x7f2fc019a6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:30:28.802 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:28.797+0000 7f2fc78c2700 1 -- 172.21.15.98:0/95834015 >> 172.21.15.98:0/95834015 conn(0x7f2fc00fa100 msgr2=0x7f2fc01031e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:30:28.802 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:28.797+0000 7f2fc78c2700 1 -- 172.21.15.98:0/95834015 shutdown_connections 2023-07-14T19:30:28.802 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:28.797+0000 7f2fc78c2700 1 -- 172.21.15.98:0/95834015 wait complete. 2023-07-14T19:30:28.803 INFO:teuthology.orchestra.run.smithi098.stderr:dumped fsmap epoch 23 2023-07-14T19:30:28.810 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-07-14T19:30:28.935 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:28.932+0000 7fd197ec4700 1 -- 172.21.15.98:0/2948777618 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd1900ff760 msgr2=0x7fd190107010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:30:28.936 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:28.932+0000 7fd197ec4700 1 --2- 172.21.15.98:0/2948777618 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd1900ff760 0x7fd190107010 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 crypto rx=0x7fd18800b0a0 tx=0x7fd18800b3b0 comp rx=0 tx=0).stop 2023-07-14T19:30:28.936 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:28.932+0000 7fd197ec4700 1 -- 172.21.15.98:0/2948777618 shutdown_connections 2023-07-14T19:30:28.936 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:28.932+0000 7fd197ec4700 1 --2- 172.21.15.98:0/2948777618 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fd190100160 0x7fd190100570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:30:28.937 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:28.932+0000 7fd197ec4700 1 --2- 172.21.15.98:0/2948777618 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd1900ff760 0x7fd190107010 unknown :-1 s=CLOSED pgs=407 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:30:28.937 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:28.932+0000 7fd197ec4700 1 --2- 172.21.15.98:0/2948777618 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7fd1900fedb0 0x7fd1900ff220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:30:28.938 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:28.932+0000 7fd197ec4700 1 -- 172.21.15.98:0/2948777618 >> 172.21.15.98:0/2948777618 conn(0x7fd1900f98d0 msgr2=0x7fd1900fbd20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:30:28.938 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:28.932+0000 7fd197ec4700 1 -- 172.21.15.98:0/2948777618 shutdown_connections 2023-07-14T19:30:28.938 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:28.932+0000 7fd197ec4700 1 -- 172.21.15.98:0/2948777618 wait complete. 2023-07-14T19:30:28.939 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:28.932+0000 7fd197ec4700 1 Processor -- start 2023-07-14T19:30:28.939 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:28.932+0000 7fd197ec4700 1 -- start start 2023-07-14T19:30:28.940 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:28.932+0000 7fd197ec4700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fd1900fedb0 0x7fd1901937c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:30:28.940 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:28.933+0000 7fd197ec4700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd1900ff760 0x7fd190193d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:30:28.940 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:28.933+0000 7fd197ec4700 1 --2- >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7fd190100160 0x7fd19019ad70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:30:28.941 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:28.933+0000 7fd197ec4700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd19019b340 con 0x7fd1900ff760 2023-07-14T19:30:28.941 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:28.933+0000 7fd197ec4700 1 -- --> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd19019b480 con 0x7fd190100160 2023-07-14T19:30:28.941 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:28.933+0000 7fd197ec4700 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd19019b610 con 0x7fd1900fedb0 2023-07-14T19:30:28.942 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:28.933+0000 7fd19545f700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd1900ff760 0x7fd190193d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:30:28.942 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:28.933+0000 7fd19545f700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd1900ff760 0x7fd190193d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:47104/0 (socket says 172.21.15.98:47104) 2023-07-14T19:30:28.943 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:28.933+0000 7fd19545f700 1 -- 172.21.15.98:0/3857552320 learned_addr learned my addr 172.21.15.98:0/3857552320 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-07-14T19:30:28.943 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:28.933+0000 7fd195c60700 1 --2- 172.21.15.98:0/3857552320 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fd1900fedb0 0x7fd1901937c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:30:28.943 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:28.933+0000 7fd196461700 1 --2- 172.21.15.98:0/3857552320 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7fd190100160 0x7fd19019ad70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:30:28.944 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:28.933+0000 7fd19545f700 1 -- 172.21.15.98:0/3857552320 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fd1900fedb0 msgr2=0x7fd1901937c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:30:28.944 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:28.933+0000 7fd19545f700 1 --2- 172.21.15.98:0/3857552320 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fd1900fedb0 0x7fd1901937c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:30:28.945 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:28.933+0000 7fd19545f700 1 -- 172.21.15.98:0/3857552320 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7fd190100160 msgr2=0x7fd19019ad70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:30:28.945 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:28.933+0000 7fd19545f700 1 --2- 172.21.15.98:0/3857552320 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7fd190100160 0x7fd19019ad70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:30:28.945 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:28.933+0000 7fd19545f700 1 -- 172.21.15.98:0/3857552320 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd188009d00 con 0x7fd1900ff760 2023-07-14T19:30:28.946 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:28.933+0000 7fd19545f700 1 --2- 172.21.15.98:0/3857552320 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd1900ff760 0x7fd190193d00 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 crypto rx=0x7fd180007bd0 tx=0x7fd180007ee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:30:28.946 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:28.933+0000 7fd186ffd700 1 -- 172.21.15.98:0/3857552320 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd180010040 con 0x7fd1900ff760 2023-07-14T19:30:28.946 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:28.933+0000 7fd186ffd700 1 -- 172.21.15.98:0/3857552320 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fd18000fe90 con 0x7fd1900ff760 2023-07-14T19:30:28.947 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:28.933+0000 7fd186ffd700 1 -- 172.21.15.98:0/3857552320 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd1800143d0 con 0x7fd1900ff760 2023-07-14T19:30:28.947 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:28.933+0000 7fd197ec4700 1 -- 172.21.15.98:0/3857552320 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd19019b8f0 con 0x7fd1900ff760 2023-07-14T19:30:28.948 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:28.933+0000 7fd197ec4700 1 -- 172.21.15.98:0/3857552320 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd19019bdb0 con 0x7fd1900ff760 2023-07-14T19:30:28.948 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:28.934+0000 7fd197ec4700 1 -- 172.21.15.98:0/3857552320 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd190106030 con 0x7fd1900ff760 2023-07-14T19:30:28.950 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:28.935+0000 7fd186ffd700 1 -- 172.21.15.98:0/3857552320 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 17) v1 ==== 135766+0+0 (secure 0 0 0) 0x7fd180015470 con 0x7fd1900ff760 2023-07-14T19:30:28.950 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:28.936+0000 7fd186ffd700 1 --2- 172.21.15.98:0/3857552320 >> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] conn(0x7fd17c0a1220 0x7fd17c0a36d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:30:28.950 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:28.936+0000 7fd195c60700 1 --2- 172.21.15.98:0/3857552320 >> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] conn(0x7fd17c0a1220 0x7fd17c0a36d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:30:28.951 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:28.936+0000 7fd186ffd700 1 -- 172.21.15.98:0/3857552320 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 8211+0+0 (secure 0 0 0) 0x7fd1800cd550 con 0x7fd1900ff760 2023-07-14T19:30:28.951 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:28.936+0000 7fd195c60700 1 --2- 172.21.15.98:0/3857552320 >> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] conn(0x7fd17c0a1220 0x7fd17c0a36d0 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7fd1900687e0 tx=0x7fd18c009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:30:28.951 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:28.939+0000 7fd186ffd700 1 -- 172.21.15.98:0/3857552320 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7fd180096af0 con 0x7fd1900ff760 2023-07-14T19:30:29.172 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:29.172+0000 7fd197ec4700 1 -- 172.21.15.98:0/3857552320 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd19002c8e0 con 0x7fd1900ff760 2023-07-14T19:30:29.173 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:29.173+0000 7fd186ffd700 1 -- 172.21.15.98:0/3857552320 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 23 v23) v1 ==== 94+0+10826 (secure 0 0 0) 0x7fd18009a110 con 0x7fd1900ff760 2023-07-14T19:30:29.174 INFO:teuthology.orchestra.run.smithi098.stdout: 2023-07-14T19:30:29.174 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":23,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14565,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6835/586280675","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6834","nonce":586280675},{"type":"v1","addr":"172.21.15.98:6835","nonce":586280675}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":4},{"gid":14577,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6837/487577257","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6836","nonce":487577257},{"type":"v1","addr":"172.21.15.98:6837","nonce":487577257}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":5},{"gid":14634,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6839/4085731209","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6838","nonce":4085731209},{"type":"v1","addr":"172.21.15.98:6839","nonce":4085731209}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":11},{"gid":14646,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6841/1679318259","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6840","nonce":1679318259},{"type":"v1","addr":"172.21.15.98:6841","nonce":1679318259}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":12},{"gid":24421,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.139:6833/3505477820","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6832","nonce":3505477820},{"type":"v1","addr":"172.21.15.139:6833","nonce":3505477820}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":24437,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.101:6833/3474460171","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.101:6832","nonce":3474460171},{"type":"v1","addr":"172.21.15.101:6833","nonce":3474460171}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":23,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2023-07-14T19:29:01.033156+0000","modified":"2023-07-14T19:30:20.766354+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24521,"mds_1":24493,"mds_2":24479},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24455":{"gid":24455,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.101:6835/1583264648","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.101:6834","nonce":1583264648},{"type":"v1","addr":"172.21.15.101:6835","nonce":1583264648}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24460":{"gid":24460,"name":"f","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.139:6835/2315281724","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6834","nonce":2315281724},{"type":"v1","addr":"172.21.15.139:6835","nonce":2315281724}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24469":{"gid":24469,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.139:6837/3597660552","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6836","nonce":3597660552},{"type":"v1","addr":"172.21.15.139:6837","nonce":3597660552}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24479":{"gid":24479,"name":"h","rank":2,"incarnation":21,"state":"up:active","state_seq":6,"addr":"172.21.15.101:6837/485741528","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.101:6836","nonce":485741528},{"type":"v1","addr":"172.21.15.101:6837","nonce":485741528}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24493":{"gid":24493,"name":"i","rank":1,"incarnation":19,"state":"up:active","state_seq":5,"addr":"172.21.15.139:6839/1258854143","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6838","nonce":1258854143},{"type":"v1","addr":"172.21.15.139:6839","nonce":1258854143}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24521":{"gid":24521,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.101:6839/48393381","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.101:6838","nonce":48393381},{"type":"v1","addr":"172.21.15.101:6839","nonce":48393381}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-07-14T19:30:29.175 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:29.175+0000 7fd197ec4700 1 -- 172.21.15.98:0/3857552320 >> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] conn(0x7fd17c0a1220 msgr2=0x7fd17c0a36d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:30:29.176 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:29.175+0000 7fd197ec4700 1 --2- 172.21.15.98:0/3857552320 >> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] conn(0x7fd17c0a1220 0x7fd17c0a36d0 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7fd1900687e0 tx=0x7fd18c009260 comp rx=0 tx=0).stop 2023-07-14T19:30:29.176 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:29.175+0000 7fd197ec4700 1 -- 172.21.15.98:0/3857552320 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd1900ff760 msgr2=0x7fd190193d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:30:29.176 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:29.175+0000 7fd197ec4700 1 --2- 172.21.15.98:0/3857552320 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd1900ff760 0x7fd190193d00 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 crypto rx=0x7fd180007bd0 tx=0x7fd180007ee0 comp rx=0 tx=0).stop 2023-07-14T19:30:29.176 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:29.175+0000 7fd197ec4700 1 -- 172.21.15.98:0/3857552320 shutdown_connections 2023-07-14T19:30:29.177 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:29.175+0000 7fd197ec4700 1 --2- 172.21.15.98:0/3857552320 >> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] conn(0x7fd17c0a1220 0x7fd17c0a36d0 unknown :-1 s=CLOSED pgs=242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:30:29.177 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:29.175+0000 7fd197ec4700 1 --2- 172.21.15.98:0/3857552320 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7fd1900fedb0 0x7fd1901937c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:30:29.177 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:29.175+0000 7fd197ec4700 1 --2- 172.21.15.98:0/3857552320 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7fd190100160 0x7fd19019ad70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:30:29.177 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:29.175+0000 7fd197ec4700 1 --2- 172.21.15.98:0/3857552320 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd1900ff760 0x7fd190193d00 unknown :-1 s=CLOSED pgs=408 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:30:29.177 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:29.175+0000 7fd197ec4700 1 -- 172.21.15.98:0/3857552320 >> 172.21.15.98:0/3857552320 conn(0x7fd1900f98d0 msgr2=0x7fd190103170 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:30:29.178 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:29.175+0000 7fd197ec4700 1 -- 172.21.15.98:0/3857552320 shutdown_connections 2023-07-14T19:30:29.178 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:29.175+0000 7fd197ec4700 1 -- 172.21.15.98:0/3857552320 wait complete. 2023-07-14T19:30:29.178 INFO:teuthology.orchestra.run.smithi098.stderr:dumped fsmap epoch 23 2023-07-14T19:30:29.187 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:30:28 smithi098 ceph-mon[127486]: pgmap v298: 97 pgs: 97 active+clean; 639 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 597 B/s wr, 6 op/s 2023-07-14T19:30:29.187 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:30:28 smithi098 ceph-mon[127486]: from='client.? 172.21.15.98:0/1687764691' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2023-07-14T19:30:29.188 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-07-14T19:30:29.191 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:30:28 smithi101 ceph-mon[128807]: pgmap v298: 97 pgs: 97 active+clean; 639 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 597 B/s wr, 6 op/s 2023-07-14T19:30:29.191 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:30:28 smithi101 ceph-mon[128807]: from='client.? 172.21.15.98:0/1687764691' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2023-07-14T19:30:29.220 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:30:28 smithi139 ceph-mon[129293]: pgmap v298: 97 pgs: 97 active+clean; 639 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 597 B/s wr, 6 op/s 2023-07-14T19:30:29.220 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:30:28 smithi139 ceph-mon[129293]: from='client.? 172.21.15.98:0/1687764691' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2023-07-14T19:30:29.309 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:29.308+0000 7f05ae660700 1 -- 172.21.15.98:0/4070201599 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f05a81017c0 msgr2=0x7f05a8108080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:30:29.309 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:29.308+0000 7f05ae660700 1 --2- 172.21.15.98:0/4070201599 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f05a81017c0 0x7f05a8108080 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 crypto rx=0x7f05a000b0a0 tx=0x7f05a000b3b0 comp rx=0 tx=0).stop 2023-07-14T19:30:29.309 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:29.308+0000 7f05ae660700 1 -- 172.21.15.98:0/4070201599 shutdown_connections 2023-07-14T19:30:29.310 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:29.308+0000 7f05ae660700 1 --2- 172.21.15.98:0/4070201599 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f05a81017c0 0x7f05a8108080 unknown :-1 s=CLOSED pgs=409 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:30:29.310 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:29.308+0000 7f05ae660700 1 --2- 172.21.15.98:0/4070201599 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f05a80ffc40 0x7f05a8100050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:30:29.310 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:29.308+0000 7f05ae660700 1 --2- 172.21.15.98:0/4070201599 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f05a8100e30 0x7f05a8101280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:30:29.310 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:29.308+0000 7f05ae660700 1 -- 172.21.15.98:0/4070201599 >> 172.21.15.98:0/4070201599 conn(0x7f05a80fb240 msgr2=0x7f05a80fd690 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:30:29.310 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:29.308+0000 7f05ae660700 1 -- 172.21.15.98:0/4070201599 shutdown_connections 2023-07-14T19:30:29.311 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:29.308+0000 7f05ae660700 1 -- 172.21.15.98:0/4070201599 wait complete. 2023-07-14T19:30:29.311 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:29.309+0000 7f05ae660700 1 Processor -- start 2023-07-14T19:30:29.311 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:29.309+0000 7f05ae660700 1 -- start start 2023-07-14T19:30:29.313 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:29.309+0000 7f05ae660700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f05a80ffc40 0x7f05a8195910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:30:29.313 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:29.309+0000 7f05ae660700 1 --2- >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f05a8100e30 0x7f05a8195e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:30:29.314 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:29.309+0000 7f05ae660700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f05a81017c0 0x7f05a819cec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:30:29.314 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:29.309+0000 7f05ae660700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f05a819d490 con 0x7f05a80ffc40 2023-07-14T19:30:29.314 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:29.309+0000 7f05ae660700 1 -- --> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f05a819d5d0 con 0x7f05a8100e30 2023-07-14T19:30:29.314 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:29.309+0000 7f05ae660700 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f05a819d760 con 0x7f05a81017c0 2023-07-14T19:30:29.314 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:29.309+0000 7f05a7fff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f05a80ffc40 0x7f05a8195910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:30:29.315 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:29.309+0000 7f05a7fff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f05a80ffc40 0x7f05a8195910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:47116/0 (socket says 172.21.15.98:47116) 2023-07-14T19:30:29.315 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:29.309+0000 7f05a7fff700 1 -- 172.21.15.98:0/1264251855 learned_addr learned my addr 172.21.15.98:0/1264251855 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-07-14T19:30:29.315 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:29.309+0000 7f05acbfd700 1 --2- 172.21.15.98:0/1264251855 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f05a81017c0 0x7f05a819cec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:30:29.315 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:29.309+0000 7f05a7fff700 1 -- 172.21.15.98:0/1264251855 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f05a81017c0 msgr2=0x7f05a819cec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:30:29.315 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:29.309+0000 7f05a7fff700 1 --2- 172.21.15.98:0/1264251855 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f05a81017c0 0x7f05a819cec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:30:29.316 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:29.309+0000 7f05a7fff700 1 -- 172.21.15.98:0/1264251855 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f05a8100e30 msgr2=0x7f05a8195e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:30:29.316 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:29.309+0000 7f05a77fe700 1 --2- 172.21.15.98:0/1264251855 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f05a8100e30 0x7f05a8195e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:30:29.316 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:29.309+0000 7f05a7fff700 1 --2- 172.21.15.98:0/1264251855 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f05a8100e30 0x7f05a8195e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:30:29.316 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:29.309+0000 7f05a7fff700 1 -- 172.21.15.98:0/1264251855 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f05a0009d00 con 0x7f05a80ffc40 2023-07-14T19:30:29.316 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:29.310+0000 7f05a7fff700 1 --2- 172.21.15.98:0/1264251855 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f05a80ffc40 0x7f05a8195910 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 crypto rx=0x7f059000c990 tx=0x7f059000cd50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:30:29.317 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:29.310+0000 7f05a57fa700 1 -- 172.21.15.98:0/1264251855 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0590007b00 con 0x7f05a80ffc40 2023-07-14T19:30:29.317 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:29.310+0000 7f05ae660700 1 -- 172.21.15.98:0/1264251855 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f05a819da40 con 0x7f05a80ffc40 2023-07-14T19:30:29.317 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:29.310+0000 7f05a57fa700 1 -- 172.21.15.98:0/1264251855 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f0590007c60 con 0x7f05a80ffc40 2023-07-14T19:30:29.317 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:29.310+0000 7f05a57fa700 1 -- 172.21.15.98:0/1264251855 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0590018660 con 0x7f05a80ffc40 2023-07-14T19:30:29.317 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:29.310+0000 7f05ae660700 1 -- 172.21.15.98:0/1264251855 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f05a819dfa0 con 0x7f05a80ffc40 2023-07-14T19:30:29.318 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:29.311+0000 7f05ae660700 1 -- 172.21.15.98:0/1264251855 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f05a810b470 con 0x7f05a80ffc40 2023-07-14T19:30:29.318 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:29.312+0000 7f05a57fa700 1 -- 172.21.15.98:0/1264251855 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 17) v1 ==== 135766+0+0 (secure 0 0 0) 0x7f059000f450 con 0x7f05a80ffc40 2023-07-14T19:30:29.318 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:29.312+0000 7f05a57fa700 1 --2- 172.21.15.98:0/1264251855 >> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] conn(0x7f05940a1220 0x7f05940a36d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:30:29.318 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:29.312+0000 7f05a57fa700 1 -- 172.21.15.98:0/1264251855 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 8211+0+0 (secure 0 0 0) 0x7f05900ccc60 con 0x7f05a80ffc40 2023-07-14T19:30:29.318 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:29.312+0000 7f05a77fe700 1 --2- 172.21.15.98:0/1264251855 >> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] conn(0x7f05940a1220 0x7f05940a36d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:30:29.319 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:29.313+0000 7f05a77fe700 1 --2- 172.21.15.98:0/1264251855 >> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] conn(0x7f05940a1220 0x7f05940a36d0 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f059800ba60 tx=0x7f0598005d50 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:30:29.319 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:29.316+0000 7f05a57fa700 1 -- 172.21.15.98:0/1264251855 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f0590096290 con 0x7f05a80ffc40 2023-07-14T19:30:29.507 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:29.507+0000 7f05ae660700 1 -- 172.21.15.98:0/1264251855 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f05a810b690 con 0x7f05a80ffc40 2023-07-14T19:30:29.508 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:29.508+0000 7f05a57fa700 1 -- 172.21.15.98:0/1264251855 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v85) v1 ==== 74+0+22604 (secure 0 0 0) 0x7f05900998b0 con 0x7f05a80ffc40 2023-07-14T19:30:29.509 INFO:teuthology.orchestra.run.smithi098.stdout: 2023-07-14T19:30:29.509 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":85,"fsid":"9adea648-227b-11ee-9b33-001a4aab830c","created":"2023-07-14T19:22:07.178854+0000","modified":"2023-07-14T19:30:03.314948+0000","last_up_change":"2023-07-14T19:28:19.646827+0000","last_in_change":"2023-07-14T19:28:02.018325+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-07-14T19:25:09.845827+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2023-07-14T19:28:58.952921+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"82","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2023-07-14T19:29:00.228272+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"83","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.690000057220459,"score_stable":1.690000057220459,"optimal_score":1,"raw_score_acting":1.690000057220459,"raw_score_stable":1.690000057220459,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"a3797e89-1754-41cf-9517-227e06ffdd72","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6802","nonce":120507228},{"type":"v1","addr":"172.21.15.98:6803","nonce":120507228}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6804","nonce":120507228},{"type":"v1","addr":"172.21.15.98:6805","nonce":120507228}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6808","nonce":120507228},{"type":"v1","addr":"172.21.15.98:6809","nonce":120507228}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6806","nonce":120507228},{"type":"v1","addr":"172.21.15.98:6807","nonce":120507228}]},"public_addr":"172.21.15.98:6803/120507228","cluster_addr":"172.21.15.98:6805/120507228","heartbeat_back_addr":"172.21.15.98:6809/120507228","heartbeat_front_addr":"172.21.15.98:6807/120507228","state":["exists","up"]},{"osd":1,"uuid":"2b001add-1aaa-4496-a403-7ef9725b80a0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.101:6800","nonce":2450584584},{"type":"v1","addr":"172.21.15.101:6801","nonce":2450584584}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.101:6802","nonce":2450584584},{"type":"v1","addr":"172.21.15.101:6803","nonce":2450584584}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.101:6806","nonce":2450584584},{"type":"v1","addr":"172.21.15.101:6807","nonce":2450584584}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.101:6804","nonce":2450584584},{"type":"v1","addr":"172.21.15.101:6805","nonce":2450584584}]},"public_addr":"172.21.15.101:6801/2450584584","cluster_addr":"172.21.15.101:6803/2450584584","heartbeat_back_addr":"172.21.15.101:6807/2450584584","heartbeat_front_addr":"172.21.15.101:6805/2450584584","state":["exists","up"]},{"osd":2,"uuid":"48e6d499-3ca9-424b-b0fb-3ba5f33b9fc9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":945038995},{"type":"v1","addr":"172.21.15.139:6801","nonce":945038995}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":945038995},{"type":"v1","addr":"172.21.15.139:6803","nonce":945038995}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":945038995},{"type":"v1","addr":"172.21.15.139:6807","nonce":945038995}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":945038995},{"type":"v1","addr":"172.21.15.139:6805","nonce":945038995}]},"public_addr":"172.21.15.139:6801/945038995","cluster_addr":"172.21.15.139:6803/945038995","heartbeat_back_addr":"172.21.15.139:6807/945038995","heartbeat_front_addr":"172.21.15.139:6805/945038995","state":["exists","up"]},{"osd":3,"uuid":"5530b265-31a5-447b-a1ea-17ebae9cfddf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6810","nonce":3095249665},{"type":"v1","addr":"172.21.15.98:6811","nonce":3095249665}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6812","nonce":3095249665},{"type":"v1","addr":"172.21.15.98:6813","nonce":3095249665}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6816","nonce":3095249665},{"type":"v1","addr":"172.21.15.98:6817","nonce":3095249665}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6814","nonce":3095249665},{"type":"v1","addr":"172.21.15.98:6815","nonce":3095249665}]},"public_addr":"172.21.15.98:6811/3095249665","cluster_addr":"172.21.15.98:6813/3095249665","heartbeat_back_addr":"172.21.15.98:6817/3095249665","heartbeat_front_addr":"172.21.15.98:6815/3095249665","state":["exists","up"]},{"osd":4,"uuid":"615d4642-a997-4f95-9e38-9fbdaa831400","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":32,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.101:6808","nonce":2844761406},{"type":"v1","addr":"172.21.15.101:6809","nonce":2844761406}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.101:6810","nonce":2844761406},{"type":"v1","addr":"172.21.15.101:6811","nonce":2844761406}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.101:6814","nonce":2844761406},{"type":"v1","addr":"172.21.15.101:6815","nonce":2844761406}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.101:6812","nonce":2844761406},{"type":"v1","addr":"172.21.15.101:6813","nonce":2844761406}]},"public_addr":"172.21.15.101:6809/2844761406","cluster_addr":"172.21.15.101:6811/2844761406","heartbeat_back_addr":"172.21.15.101:6815/2844761406","heartbeat_front_addr":"172.21.15.101:6813/2844761406","state":["exists","up"]},{"osd":5,"uuid":"415de5fa-607a-45a8-b732-28daf3be6dd3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":37,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":3532263042},{"type":"v1","addr":"172.21.15.139:6809","nonce":3532263042}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6810","nonce":3532263042},{"type":"v1","addr":"172.21.15.139:6811","nonce":3532263042}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6814","nonce":3532263042},{"type":"v1","addr":"172.21.15.139:6815","nonce":3532263042}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":3532263042},{"type":"v1","addr":"172.21.15.139:6813","nonce":3532263042}]},"public_addr":"172.21.15.139:6809/3532263042","cluster_addr":"172.21.15.139:6811/3532263042","heartbeat_back_addr":"172.21.15.139:6815/3532263042","heartbeat_front_addr":"172.21.15.139:6813/3532263042","state":["exists","up"]},{"osd":6,"uuid":"d7f9d4ba-c309-4b75-a836-5a36384b528e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":45,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6818","nonce":658948743},{"type":"v1","addr":"172.21.15.98:6819","nonce":658948743}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6820","nonce":658948743},{"type":"v1","addr":"172.21.15.98:6821","nonce":658948743}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6824","nonce":658948743},{"type":"v1","addr":"172.21.15.98:6825","nonce":658948743}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6822","nonce":658948743},{"type":"v1","addr":"172.21.15.98:6823","nonce":658948743}]},"public_addr":"172.21.15.98:6819/658948743","cluster_addr":"172.21.15.98:6821/658948743","heartbeat_back_addr":"172.21.15.98:6825/658948743","heartbeat_front_addr":"172.21.15.98:6823/658948743","state":["exists","up"]},{"osd":7,"uuid":"eedb94d9-c5cd-4a93-8a8c-2648804b7195","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":50,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.101:6816","nonce":2881214066},{"type":"v1","addr":"172.21.15.101:6817","nonce":2881214066}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.101:6818","nonce":2881214066},{"type":"v1","addr":"172.21.15.101:6819","nonce":2881214066}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.101:6822","nonce":2881214066},{"type":"v1","addr":"172.21.15.101:6823","nonce":2881214066}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.101:6820","nonce":2881214066},{"type":"v1","addr":"172.21.15.101:6821","nonce":2881214066}]},"public_addr":"172.21.15.101:6817/2881214066","cluster_addr":"172.21.15.101:6819/2881214066","heartbeat_back_addr":"172.21.15.101:6823/2881214066","heartbeat_front_addr":"172.21.15.101:6821/2881214066","state":["exists","up"]},{"osd":8,"uuid":"6bfd77c2-5770-466e-aa6f-84b50c74ffb6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":57,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":3669514524},{"type":"v1","addr":"172.21.15.139:6817","nonce":3669514524}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":3669514524},{"type":"v1","addr":"172.21.15.139:6819","nonce":3669514524}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":3669514524},{"type":"v1","addr":"172.21.15.139:6823","nonce":3669514524}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6820","nonce":3669514524},{"type":"v1","addr":"172.21.15.139:6821","nonce":3669514524}]},"public_addr":"172.21.15.139:6817/3669514524","cluster_addr":"172.21.15.139:6819/3669514524","heartbeat_back_addr":"172.21.15.139:6823/3669514524","heartbeat_front_addr":"172.21.15.139:6821/3669514524","state":["exists","up"]},{"osd":9,"uuid":"19c5b54a-2242-4b23-b721-63be9559653d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":65,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6826","nonce":2893994080},{"type":"v1","addr":"172.21.15.98:6827","nonce":2893994080}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6828","nonce":2893994080},{"type":"v1","addr":"172.21.15.98:6829","nonce":2893994080}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6832","nonce":2893994080},{"type":"v1","addr":"172.21.15.98:6833","nonce":2893994080}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6830","nonce":2893994080},{"type":"v1","addr":"172.21.15.98:6831","nonce":2893994080}]},"public_addr":"172.21.15.98:6827/2893994080","cluster_addr":"172.21.15.98:6829/2893994080","heartbeat_back_addr":"172.21.15.98:6833/2893994080","heartbeat_front_addr":"172.21.15.98:6831/2893994080","state":["exists","up"]},{"osd":10,"uuid":"326f0209-ba9d-4b59-97cc-d61a758a8f29","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":70,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.101:6824","nonce":3545244225},{"type":"v1","addr":"172.21.15.101:6825","nonce":3545244225}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.101:6826","nonce":3545244225},{"type":"v1","addr":"172.21.15.101:6827","nonce":3545244225}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.101:6830","nonce":3545244225},{"type":"v1","addr":"172.21.15.101:6831","nonce":3545244225}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.101:6828","nonce":3545244225},{"type":"v1","addr":"172.21.15.101:6829","nonce":3545244225}]},"public_addr":"172.21.15.101:6825/3545244225","cluster_addr":"172.21.15.101:6827/3545244225","heartbeat_back_addr":"172.21.15.101:6831/3545244225","heartbeat_front_addr":"172.21.15.101:6829/3545244225","state":["exists","up"]},{"osd":11,"uuid":"c61c0320-1057-4f17-9bf1-7ad306ed7ed3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":78,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6824","nonce":2630023454},{"type":"v1","addr":"172.21.15.139:6825","nonce":2630023454}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6826","nonce":2630023454},{"type":"v1","addr":"172.21.15.139:6827","nonce":2630023454}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6830","nonce":2630023454},{"type":"v1","addr":"172.21.15.139:6831","nonce":2630023454}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6828","nonce":2630023454},{"type":"v1","addr":"172.21.15.139:6829","nonce":2630023454}]},"public_addr":"172.21.15.139:6825/2630023454","cluster_addr":"172.21.15.139:6827/2630023454","heartbeat_back_addr":"172.21.15.139:6831/2630023454","heartbeat_front_addr":"172.21.15.139:6829/2630023454","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T19:24:23.620028+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T19:24:45.779846+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T19:25:06.877949+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T19:25:27.994199+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T19:25:49.461139+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T19:26:10.409282+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T19:26:32.116390+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T19:26:53.536474+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T19:27:14.384839+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T19:27:35.635921+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T19:27:56.858744+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T19:28:18.342519+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.4","mappings":[{"from":2,"to":11}]},{"pgid":"2.f","mappings":[{"from":8,"to":11}]},{"pgid":"3.0","mappings":[{"from":4,"to":8}]},{"pgid":"3.1","mappings":[{"from":4,"to":9}]},{"pgid":"3.c","mappings":[{"from":3,"to":9}]},{"pgid":"3.17","mappings":[{"from":4,"to":6}]},{"pgid":"3.1e","mappings":[{"from":3,"to":9}]},{"pgid":"3.28","mappings":[{"from":5,"to":7}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.98:0/2164590253":"2023-07-15T19:23:01.520302+0000","172.21.15.98:0/2094734896":"2023-07-15T19:23:01.520302+0000","172.21.15.98:0/784745009":"2023-07-15T19:22:39.481502+0000","172.21.15.98:0/3657190818":"2023-07-15T19:22:39.481502+0000","172.21.15.98:6800/1732769087":"2023-07-15T19:23:01.520302+0000","172.21.15.98:0/3393542419":"2023-07-15T19:22:39.481502+0000","172.21.15.98:6800/749234665":"2023-07-15T19:22:39.481502+0000","172.21.15.98:6801/1732769087":"2023-07-15T19:23:01.520302+0000","172.21.15.98:6801/749234665":"2023-07-15T19:22:39.481502+0000","172.21.15.98:0/187822873":"2023-07-15T19:23:01.520302+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-07-14T19:30:29.511 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:29.510+0000 7f05ae660700 1 -- 172.21.15.98:0/1264251855 >> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] conn(0x7f05940a1220 msgr2=0x7f05940a36d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:30:29.511 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:29.510+0000 7f05ae660700 1 --2- 172.21.15.98:0/1264251855 >> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] conn(0x7f05940a1220 0x7f05940a36d0 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f059800ba60 tx=0x7f0598005d50 comp rx=0 tx=0).stop 2023-07-14T19:30:29.511 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:29.510+0000 7f05ae660700 1 -- 172.21.15.98:0/1264251855 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f05a80ffc40 msgr2=0x7f05a8195910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:30:29.511 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:29.510+0000 7f05ae660700 1 --2- 172.21.15.98:0/1264251855 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f05a80ffc40 0x7f05a8195910 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 crypto rx=0x7f059000c990 tx=0x7f059000cd50 comp rx=0 tx=0).stop 2023-07-14T19:30:29.512 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:29.510+0000 7f05ae660700 1 -- 172.21.15.98:0/1264251855 shutdown_connections 2023-07-14T19:30:29.512 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:29.510+0000 7f05ae660700 1 --2- 172.21.15.98:0/1264251855 >> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] conn(0x7f05940a1220 0x7f05940a36d0 unknown :-1 s=CLOSED pgs=243 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:30:29.512 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:29.510+0000 7f05ae660700 1 --2- 172.21.15.98:0/1264251855 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f05a81017c0 0x7f05a819cec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:30:29.512 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:29.510+0000 7f05ae660700 1 --2- 172.21.15.98:0/1264251855 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f05a80ffc40 0x7f05a8195910 unknown :-1 s=CLOSED pgs=410 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:30:29.513 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:29.510+0000 7f05ae660700 1 --2- 172.21.15.98:0/1264251855 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f05a8100e30 0x7f05a8195e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:30:29.513 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:29.510+0000 7f05ae660700 1 -- 172.21.15.98:0/1264251855 >> 172.21.15.98:0/1264251855 conn(0x7f05a80fb240 msgr2=0x7f05a80fd270 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:30:29.513 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:29.510+0000 7f05ae660700 1 -- 172.21.15.98:0/1264251855 shutdown_connections 2023-07-14T19:30:29.513 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:29.510+0000 7f05ae660700 1 -- 172.21.15.98:0/1264251855 wait complete. 2023-07-14T19:30:29.523 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2023-07-14T19:30:29.646 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:29.645+0000 7f83ca0ed700 1 -- 172.21.15.98:0/2047612535 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f83c40ffd00 msgr2=0x7f83c4100150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:30:29.646 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:29.645+0000 7f83ca0ed700 1 --2- 172.21.15.98:0/2047612535 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f83c40ffd00 0x7f83c4100150 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 crypto rx=0x7f83b4009990 tx=0x7f83b4009ca0 comp rx=0 tx=0).stop 2023-07-14T19:30:29.647 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:29.645+0000 7f83ca0ed700 1 -- 172.21.15.98:0/2047612535 shutdown_connections 2023-07-14T19:30:29.647 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:29.645+0000 7f83ca0ed700 1 --2- 172.21.15.98:0/2047612535 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f83c40feb10 0x7f83c40fef20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:30:29.647 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:29.645+0000 7f83ca0ed700 1 --2- 172.21.15.98:0/2047612535 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f83c4100690 0x7f83c4106f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:30:29.647 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:29.645+0000 7f83ca0ed700 1 --2- 172.21.15.98:0/2047612535 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f83c40ffd00 0x7f83c4100150 unknown :-1 s=CLOSED pgs=411 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:30:29.647 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:29.645+0000 7f83ca0ed700 1 -- 172.21.15.98:0/2047612535 >> 172.21.15.98:0/2047612535 conn(0x7f83c40fa130 msgr2=0x7f83c40fc560 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:30:29.648 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:29.645+0000 7f83ca0ed700 1 -- 172.21.15.98:0/2047612535 shutdown_connections 2023-07-14T19:30:29.648 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:29.645+0000 7f83ca0ed700 1 -- 172.21.15.98:0/2047612535 wait complete. 2023-07-14T19:30:29.650 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:29.646+0000 7f83ca0ed700 1 Processor -- start 2023-07-14T19:30:29.650 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:29.646+0000 7f83ca0ed700 1 -- start start 2023-07-14T19:30:29.650 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:29.647+0000 7f83ca0ed700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f83c40feb10 0x7f83c41948f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:30:29.650 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:29.647+0000 7f83ca0ed700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f83c40ffd00 0x7f83c4194e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:30:29.651 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:29.647+0000 7f83ca0ed700 1 --2- >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f83c4100690 0x7f83c419bea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:30:29.651 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:29.647+0000 7f83ca0ed700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83c419c470 con 0x7f83c40ffd00 2023-07-14T19:30:29.651 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:29.647+0000 7f83ca0ed700 1 -- --> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83c419c5b0 con 0x7f83c4100690 2023-07-14T19:30:29.651 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:29.647+0000 7f83ca0ed700 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83c419c740 con 0x7f83c40feb10 2023-07-14T19:30:29.651 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:29.647+0000 7f83c2ffd700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f83c40ffd00 0x7f83c4194e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:30:29.651 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:29.647+0000 7f83c2ffd700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f83c40ffd00 0x7f83c4194e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:47132/0 (socket says 172.21.15.98:47132) 2023-07-14T19:30:29.652 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:29.647+0000 7f83c2ffd700 1 -- 172.21.15.98:0/4187650013 learned_addr learned my addr 172.21.15.98:0/4187650013 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-07-14T19:30:29.652 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:29.647+0000 7f83c3fff700 1 --2- 172.21.15.98:0/4187650013 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f83c4100690 0x7f83c419bea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:30:29.652 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:29.647+0000 7f83c37fe700 1 --2- 172.21.15.98:0/4187650013 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f83c40feb10 0x7f83c41948f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:30:29.652 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:29.647+0000 7f83c2ffd700 1 -- 172.21.15.98:0/4187650013 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f83c40feb10 msgr2=0x7f83c41948f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:30:29.653 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:29.647+0000 7f83c2ffd700 1 --2- 172.21.15.98:0/4187650013 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f83c40feb10 0x7f83c41948f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:30:29.653 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:29.647+0000 7f83c2ffd700 1 -- 172.21.15.98:0/4187650013 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f83c4100690 msgr2=0x7f83c419bea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:30:29.653 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:29.647+0000 7f83c2ffd700 1 --2- 172.21.15.98:0/4187650013 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f83c4100690 0x7f83c419bea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:30:29.653 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:29.647+0000 7f83c2ffd700 1 -- 172.21.15.98:0/4187650013 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f83b40095f0 con 0x7f83c40ffd00 2023-07-14T19:30:29.653 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:29.647+0000 7f83c2ffd700 1 --2- 172.21.15.98:0/4187650013 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f83c40ffd00 0x7f83c4194e30 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 crypto rx=0x7f83b400c010 tx=0x7f83b4004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:30:29.653 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:29.647+0000 7f83c0ff9700 1 -- 172.21.15.98:0/4187650013 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f83b401d070 con 0x7f83c40ffd00 2023-07-14T19:30:29.654 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:29.647+0000 7f83ca0ed700 1 -- 172.21.15.98:0/4187650013 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f83c419c9c0 con 0x7f83c40ffd00 2023-07-14T19:30:29.654 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:29.647+0000 7f83c0ff9700 1 -- 172.21.15.98:0/4187650013 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f83b40054f0 con 0x7f83c40ffd00 2023-07-14T19:30:29.654 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:29.647+0000 7f83c0ff9700 1 -- 172.21.15.98:0/4187650013 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f83b4004ca0 con 0x7f83c40ffd00 2023-07-14T19:30:29.654 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:29.647+0000 7f83ca0ed700 1 -- 172.21.15.98:0/4187650013 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f83c419cd80 con 0x7f83c40ffd00 2023-07-14T19:30:29.654 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:29.648+0000 7f83ca0ed700 1 -- 172.21.15.98:0/4187650013 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f83c41060a0 con 0x7f83c40ffd00 2023-07-14T19:30:29.655 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:29.649+0000 7f83c0ff9700 1 -- 172.21.15.98:0/4187650013 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 17) v1 ==== 135766+0+0 (secure 0 0 0) 0x7f83b400fbc0 con 0x7f83c40ffd00 2023-07-14T19:30:29.655 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:29.650+0000 7f83c0ff9700 1 --2- 172.21.15.98:0/4187650013 >> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] conn(0x7f83ac0a11e0 0x7f83ac0a3690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:30:29.656 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:29.650+0000 7f83c0ff9700 1 -- 172.21.15.98:0/4187650013 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 8211+0+0 (secure 0 0 0) 0x7f83b40cddb0 con 0x7f83c40ffd00 2023-07-14T19:30:29.656 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:29.650+0000 7f83c37fe700 1 --2- 172.21.15.98:0/4187650013 >> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] conn(0x7f83ac0a11e0 0x7f83ac0a3690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:30:29.656 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:29.650+0000 7f83c37fe700 1 --2- 172.21.15.98:0/4187650013 >> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] conn(0x7f83ac0a11e0 0x7f83ac0a3690 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f83b8005950 tx=0x7f83b80058e0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:30:29.656 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:29.654+0000 7f83c0ff9700 1 -- 172.21.15.98:0/4187650013 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f83b405a780 con 0x7f83c40ffd00 2023-07-14T19:30:29.890 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:29.889+0000 7f83ca0ed700 1 -- 172.21.15.98:0/4187650013 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) v1 -- 0x7f83c402cb00 con 0x7f83c40ffd00 2023-07-14T19:30:29.897 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:29.897+0000 7f83c0ff9700 1 -- 172.21.15.98:0/4187650013 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v24) v1 ==== 161+0+0 (secure 0 0 0) 0x7f83b4097440 con 0x7f83c40ffd00 2023-07-14T19:30:29.900 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:29.900+0000 7f83ca0ed700 1 -- 172.21.15.98:0/4187650013 >> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] conn(0x7f83ac0a11e0 msgr2=0x7f83ac0a3690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:30:29.900 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:29.900+0000 7f83ca0ed700 1 --2- 172.21.15.98:0/4187650013 >> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] conn(0x7f83ac0a11e0 0x7f83ac0a3690 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f83b8005950 tx=0x7f83b80058e0 comp rx=0 tx=0).stop 2023-07-14T19:30:29.901 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:29.900+0000 7f83ca0ed700 1 -- 172.21.15.98:0/4187650013 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f83c40ffd00 msgr2=0x7f83c4194e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:30:29.901 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:29.900+0000 7f83ca0ed700 1 --2- 172.21.15.98:0/4187650013 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f83c40ffd00 0x7f83c4194e30 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 crypto rx=0x7f83b400c010 tx=0x7f83b4004730 comp rx=0 tx=0).stop 2023-07-14T19:30:29.901 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:29.900+0000 7f83ca0ed700 1 -- 172.21.15.98:0/4187650013 shutdown_connections 2023-07-14T19:30:29.901 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:29.900+0000 7f83ca0ed700 1 --2- 172.21.15.98:0/4187650013 >> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] conn(0x7f83ac0a11e0 0x7f83ac0a3690 unknown :-1 s=CLOSED pgs=244 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:30:29.901 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:29.900+0000 7f83ca0ed700 1 --2- 172.21.15.98:0/4187650013 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f83c40feb10 0x7f83c41948f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:30:29.902 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:29.900+0000 7f83ca0ed700 1 --2- 172.21.15.98:0/4187650013 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f83c4100690 0x7f83c419bea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:30:29.902 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:29.900+0000 7f83ca0ed700 1 --2- 172.21.15.98:0/4187650013 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f83c40ffd00 0x7f83c4194e30 unknown :-1 s=CLOSED pgs=412 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:30:29.902 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:29.900+0000 7f83ca0ed700 1 -- 172.21.15.98:0/4187650013 >> 172.21.15.98:0/4187650013 conn(0x7f83c40fa130 msgr2=0x7f83c41031e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:30:29.902 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:29.900+0000 7f83ca0ed700 1 -- 172.21.15.98:0/4187650013 shutdown_connections 2023-07-14T19:30:29.902 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:29.900+0000 7f83ca0ed700 1 -- 172.21.15.98:0/4187650013 wait complete. 2023-07-14T19:30:29.903 INFO:teuthology.orchestra.run.smithi098.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2023-07-14T19:30:29.913 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2023-07-14T19:30:30.037 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:30.036+0000 7ff7a31a0700 1 -- 172.21.15.98:0/3379762665 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff79c0fea00 msgr2=0x7ff79c0fee10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:30:30.038 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:30.036+0000 7ff7a31a0700 1 --2- 172.21.15.98:0/3379762665 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff79c0fea00 0x7ff79c0fee10 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 crypto rx=0x7ff790009990 tx=0x7ff790009ca0 comp rx=0 tx=0).stop 2023-07-14T19:30:30.038 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:30.036+0000 7ff7a31a0700 1 -- 172.21.15.98:0/3379762665 shutdown_connections 2023-07-14T19:30:30.038 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:30.036+0000 7ff7a31a0700 1 --2- 172.21.15.98:0/3379762665 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7ff79c100580 0x7ff79c106e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:30:30.038 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:30.036+0000 7ff7a31a0700 1 --2- 172.21.15.98:0/3379762665 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff79c0fea00 0x7ff79c0fee10 unknown :-1 s=CLOSED pgs=418 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:30:30.038 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:30.036+0000 7ff7a31a0700 1 --2- 172.21.15.98:0/3379762665 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7ff79c0ffbf0 0x7ff79c100040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:30:30.039 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:30.036+0000 7ff7a31a0700 1 -- 172.21.15.98:0/3379762665 >> 172.21.15.98:0/3379762665 conn(0x7ff79c0fa000 msgr2=0x7ff79c0fc450 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:30:30.039 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:30.036+0000 7ff7a31a0700 1 -- 172.21.15.98:0/3379762665 shutdown_connections 2023-07-14T19:30:30.039 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:30.036+0000 7ff7a31a0700 1 -- 172.21.15.98:0/3379762665 wait complete. 2023-07-14T19:30:30.041 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:30.037+0000 7ff7a31a0700 1 Processor -- start 2023-07-14T19:30:30.042 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:30.037+0000 7ff7a31a0700 1 -- start start 2023-07-14T19:30:30.042 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:30.037+0000 7ff7a31a0700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7ff79c0fea00 0x7ff79c068330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:30:30.042 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:30.037+0000 7ff7a31a0700 1 --2- >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7ff79c0ffbf0 0x7ff79c19a470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:30:30.042 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:30.037+0000 7ff7a31a0700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff79c100580 0x7ff79c19cac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:30:30.042 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:30.037+0000 7ff7a31a0700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff79c19d290 con 0x7ff79c100580 2023-07-14T19:30:30.043 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:30.037+0000 7ff7a31a0700 1 -- --> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff79c19d400 con 0x7ff79c0ffbf0 2023-07-14T19:30:30.043 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:30.037+0000 7ff7a31a0700 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff79c19d5c0 con 0x7ff79c0fea00 2023-07-14T19:30:30.043 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:30.037+0000 7ff7a299f700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff79c100580 0x7ff79c19cac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:30:30.043 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:30.037+0000 7ff7a299f700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff79c100580 0x7ff79c19cac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:47148/0 (socket says 172.21.15.98:47148) 2023-07-14T19:30:30.043 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:30.037+0000 7ff7a299f700 1 -- 172.21.15.98:0/302434192 learned_addr learned my addr 172.21.15.98:0/302434192 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-07-14T19:30:30.044 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:30.037+0000 7ff7a199d700 1 --2- 172.21.15.98:0/302434192 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7ff79c0ffbf0 0x7ff79c19a470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:30:30.044 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:30.037+0000 7ff7a219e700 1 --2- 172.21.15.98:0/302434192 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7ff79c0fea00 0x7ff79c068330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:30:30.044 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:30.038+0000 7ff7a299f700 1 -- 172.21.15.98:0/302434192 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7ff79c0fea00 msgr2=0x7ff79c068330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:30:30.044 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:30.038+0000 7ff7a299f700 1 --2- 172.21.15.98:0/302434192 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7ff79c0fea00 0x7ff79c068330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:30:30.044 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:30.038+0000 7ff7a299f700 1 -- 172.21.15.98:0/302434192 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7ff79c0ffbf0 msgr2=0x7ff79c19a470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:30:30.045 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:30.038+0000 7ff7a299f700 1 --2- 172.21.15.98:0/302434192 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7ff79c0ffbf0 0x7ff79c19a470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:30:30.045 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:30.038+0000 7ff7a299f700 1 -- 172.21.15.98:0/302434192 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff7900095f0 con 0x7ff79c100580 2023-07-14T19:30:30.045 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:30.038+0000 7ff7a199d700 1 --2- 172.21.15.98:0/302434192 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7ff79c0ffbf0 0x7ff79c19a470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-07-14T19:30:30.045 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:30.038+0000 7ff7a299f700 1 --2- 172.21.15.98:0/302434192 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff79c100580 0x7ff79c19cac0 secure :-1 s=READY pgs=419 cs=0 l=1 rev1=1 crypto rx=0x7ff794007ee0 tx=0x7ff79400d3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:30:30.045 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:30.038+0000 7ff78f7fe700 1 -- 172.21.15.98:0/302434192 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff79400ddd0 con 0x7ff79c100580 2023-07-14T19:30:30.046 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:30.038+0000 7ff78f7fe700 1 -- 172.21.15.98:0/302434192 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7ff79400f040 con 0x7ff79c100580 2023-07-14T19:30:30.046 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:30.038+0000 7ff7a31a0700 1 -- 172.21.15.98:0/302434192 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff79c19d8a0 con 0x7ff79c100580 2023-07-14T19:30:30.046 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:30.038+0000 7ff78f7fe700 1 -- 172.21.15.98:0/302434192 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff794015400 con 0x7ff79c100580 2023-07-14T19:30:30.046 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:30.038+0000 7ff7a31a0700 1 -- 172.21.15.98:0/302434192 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff79c19dd60 con 0x7ff79c100580 2023-07-14T19:30:30.046 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:30.039+0000 7ff7a31a0700 1 -- 172.21.15.98:0/302434192 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff79c10a240 con 0x7ff79c100580 2023-07-14T19:30:30.047 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:30.040+0000 7ff78f7fe700 1 -- 172.21.15.98:0/302434192 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 17) v1 ==== 135766+0+0 (secure 0 0 0) 0x7ff794004000 con 0x7ff79c100580 2023-07-14T19:30:30.047 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:30.041+0000 7ff78f7fe700 1 --2- 172.21.15.98:0/302434192 >> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] conn(0x7ff7880a1290 0x7ff7880a3740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:30:30.047 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:30.041+0000 7ff7a219e700 1 --2- 172.21.15.98:0/302434192 >> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] conn(0x7ff7880a1290 0x7ff7880a3740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:30:30.047 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:30.041+0000 7ff78f7fe700 1 -- 172.21.15.98:0/302434192 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 8469+0+0 (secure 0 0 0) 0x7ff7940ccd50 con 0x7ff79c100580 2023-07-14T19:30:30.047 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:30.041+0000 7ff7a219e700 1 --2- 172.21.15.98:0/302434192 >> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] conn(0x7ff7880a1290 0x7ff7880a3740 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7ff79000b5c0 tx=0x7ff790005fb0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:30:30.048 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:30.044+0000 7ff78f7fe700 1 -- 172.21.15.98:0/302434192 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7ff794096250 con 0x7ff79c100580 2023-07-14T19:30:30.191 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:30:29 smithi101 ceph-mon[128807]: from='client.? 172.21.15.98:0/95834015' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-07-14T19:30:30.191 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:30:29 smithi101 ceph-mon[128807]: from='client.? 172.21.15.98:0/3857552320' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-07-14T19:30:30.192 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:30:29 smithi101 ceph-mon[128807]: from='client.? 172.21.15.98:0/1264251855' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-07-14T19:30:30.192 INFO:journalctl@ceph.mds.b.smithi101.stdout:Jul 14 19:30:29 smithi101 ceph-9adea648-227b-11ee-9b33-001a4aab830c-mds-b[154069]: ignoring --setuser ceph since I am not root 2023-07-14T19:30:30.193 INFO:journalctl@ceph.mds.b.smithi101.stdout:Jul 14 19:30:29 smithi101 ceph-9adea648-227b-11ee-9b33-001a4aab830c-mds-b[154069]: ignoring --setgroup ceph since I am not root 2023-07-14T19:30:30.193 INFO:journalctl@ceph.mds.b.smithi101.stdout:Jul 14 19:30:29 smithi101 ceph-mds[154093]: -- 172.21.15.101:0/410678625 <== mon.2 v2:172.21.15.101:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55a6ab1091e0 con 0x55a6ab0df000 2023-07-14T19:30:30.193 INFO:journalctl@ceph.mds.b.smithi101.stdout:Jul 14 19:30:29 smithi101 ceph-9adea648-227b-11ee-9b33-001a4aab830c-mds-b[154069]: starting mds.b at 2023-07-14T19:30:30.194 INFO:journalctl@ceph.mds.e.smithi101.stdout:Jul 14 19:30:29 smithi101 ceph-9adea648-227b-11ee-9b33-001a4aab830c-mds-e[153382]: ignoring --setuser ceph since I am not root 2023-07-14T19:30:30.194 INFO:journalctl@ceph.mds.e.smithi101.stdout:Jul 14 19:30:29 smithi101 ceph-9adea648-227b-11ee-9b33-001a4aab830c-mds-e[153382]: ignoring --setgroup ceph since I am not root 2023-07-14T19:30:30.194 INFO:journalctl@ceph.mds.e.smithi101.stdout:Jul 14 19:30:29 smithi101 ceph-mds[153406]: -- 172.21.15.101:0/324527540 <== mon.0 v2:172.21.15.98:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x5606aa0e91e0 con 0x5606aa0be800 2023-07-14T19:30:30.194 INFO:journalctl@ceph.mds.e.smithi101.stdout:Jul 14 19:30:29 smithi101 ceph-9adea648-227b-11ee-9b33-001a4aab830c-mds-e[153382]: starting mds.e at 2023-07-14T19:30:30.195 INFO:journalctl@ceph.mds.h.smithi101.stdout:Jul 14 19:30:29 smithi101 ceph-9adea648-227b-11ee-9b33-001a4aab830c-mds-h[153725]: ignoring --setuser ceph since I am not root 2023-07-14T19:30:30.195 INFO:journalctl@ceph.mds.h.smithi101.stdout:Jul 14 19:30:29 smithi101 ceph-9adea648-227b-11ee-9b33-001a4aab830c-mds-h[153725]: ignoring --setgroup ceph since I am not root 2023-07-14T19:30:30.195 INFO:journalctl@ceph.mds.h.smithi101.stdout:Jul 14 19:30:29 smithi101 ceph-mds[153750]: -- 172.21.15.101:0/3997182198 <== mon.0 v2:172.21.15.98:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55d3b247f1e0 con 0x55d3b2454800 2023-07-14T19:30:30.195 INFO:journalctl@ceph.mds.h.smithi101.stdout:Jul 14 19:30:29 smithi101 ceph-9adea648-227b-11ee-9b33-001a4aab830c-mds-h[153725]: starting mds.h at 2023-07-14T19:30:30.220 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:30:29 smithi139 ceph-mon[129293]: from='client.? 172.21.15.98:0/95834015' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-07-14T19:30:30.220 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:30:29 smithi139 ceph-mon[129293]: from='client.? 172.21.15.98:0/3857552320' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-07-14T19:30:30.221 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:30:29 smithi139 ceph-mon[129293]: from='client.? 172.21.15.98:0/1264251855' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-07-14T19:30:30.221 INFO:journalctl@ceph.mds.c.smithi139.stdout:Jul 14 19:30:29 smithi139 ceph-9adea648-227b-11ee-9b33-001a4aab830c-mds-c[155201]: ignoring --setuser ceph since I am not root 2023-07-14T19:30:30.221 INFO:journalctl@ceph.mds.c.smithi139.stdout:Jul 14 19:30:29 smithi139 ceph-9adea648-227b-11ee-9b33-001a4aab830c-mds-c[155201]: ignoring --setgroup ceph since I am not root 2023-07-14T19:30:30.222 INFO:journalctl@ceph.mds.c.smithi139.stdout:Jul 14 19:30:29 smithi139 ceph-mds[155225]: -- 172.21.15.139:0/2554447716 <== mon.1 v2:172.21.15.139:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x558c73ab91e0 con 0x558c73a8ec00 2023-07-14T19:30:30.222 INFO:journalctl@ceph.mds.c.smithi139.stdout:Jul 14 19:30:29 smithi139 ceph-9adea648-227b-11ee-9b33-001a4aab830c-mds-c[155201]: starting mds.c at 2023-07-14T19:30:30.222 INFO:journalctl@ceph.mds.i.smithi139.stdout:Jul 14 19:30:29 smithi139 ceph-9adea648-227b-11ee-9b33-001a4aab830c-mds-i[155543]: ignoring --setuser ceph since I am not root 2023-07-14T19:30:30.222 INFO:journalctl@ceph.mds.i.smithi139.stdout:Jul 14 19:30:29 smithi139 ceph-9adea648-227b-11ee-9b33-001a4aab830c-mds-i[155543]: ignoring --setgroup ceph since I am not root 2023-07-14T19:30:30.223 INFO:journalctl@ceph.mds.i.smithi139.stdout:Jul 14 19:30:29 smithi139 ceph-mds[155567]: -- 172.21.15.139:0/2437202984 <== mon.0 v2:172.21.15.98:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55c5b64171e0 con 0x55c5b63ec800 2023-07-14T19:30:30.223 INFO:journalctl@ceph.mds.i.smithi139.stdout:Jul 14 19:30:29 smithi139 ceph-9adea648-227b-11ee-9b33-001a4aab830c-mds-i[155543]: starting mds.i at 2023-07-14T19:30:30.223 INFO:journalctl@ceph.mds.f.smithi139.stdout:Jul 14 19:30:29 smithi139 ceph-9adea648-227b-11ee-9b33-001a4aab830c-mds-f[154861]: ignoring --setuser ceph since I am not root 2023-07-14T19:30:30.224 INFO:journalctl@ceph.mds.f.smithi139.stdout:Jul 14 19:30:29 smithi139 ceph-9adea648-227b-11ee-9b33-001a4aab830c-mds-f[154861]: ignoring --setgroup ceph since I am not root 2023-07-14T19:30:30.224 INFO:journalctl@ceph.mds.f.smithi139.stdout:Jul 14 19:30:29 smithi139 ceph-mds[154885]: -- 172.21.15.139:0/628169577 <== mon.2 v2:172.21.15.101:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55a9f482f1e0 con 0x55a9f4804800 2023-07-14T19:30:30.224 INFO:journalctl@ceph.mds.f.smithi139.stdout:Jul 14 19:30:29 smithi139 ceph-9adea648-227b-11ee-9b33-001a4aab830c-mds-f[154861]: starting mds.f at 2023-07-14T19:30:30.276 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:30.276+0000 7ff7a31a0700 1 -- 172.21.15.98:0/302434192 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) v1 -- 0x7ff79c1058a0 con 0x7ff79c100580 2023-07-14T19:30:30.280 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:30:29 smithi098 ceph-mon[127486]: from='client.? 172.21.15.98:0/95834015' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-07-14T19:30:30.281 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:30:29 smithi098 ceph-mon[127486]: from='client.? 172.21.15.98:0/3857552320' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-07-14T19:30:30.281 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:30:29 smithi098 ceph-mon[127486]: from='client.? 172.21.15.98:0/1264251855' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-07-14T19:30:30.281 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:30:29 smithi098 ceph-9adea648-227b-11ee-9b33-001a4aab830c-mon-a[127462]: 2023-07-14T19:30:29.890+0000 7f6e5f2b7700 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2023-07-14T19:30:30.899 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:30.898+0000 7ff78f7fe700 1 -- 172.21.15.98:0/302434192 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v25) v1 ==== 104+0+0 (secure 0 0 0) 0x7ff794099870 con 0x7ff79c100580 2023-07-14T19:30:30.902 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:30.901+0000 7ff7a31a0700 1 -- 172.21.15.98:0/302434192 >> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] conn(0x7ff7880a1290 msgr2=0x7ff7880a3740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:30:30.902 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:30.901+0000 7ff7a31a0700 1 --2- 172.21.15.98:0/302434192 >> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] conn(0x7ff7880a1290 0x7ff7880a3740 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7ff79000b5c0 tx=0x7ff790005fb0 comp rx=0 tx=0).stop 2023-07-14T19:30:30.903 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:30.901+0000 7ff7a31a0700 1 -- 172.21.15.98:0/302434192 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff79c100580 msgr2=0x7ff79c19cac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:30:30.903 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:30.901+0000 7ff7a31a0700 1 --2- 172.21.15.98:0/302434192 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff79c100580 0x7ff79c19cac0 secure :-1 s=READY pgs=419 cs=0 l=1 rev1=1 crypto rx=0x7ff794007ee0 tx=0x7ff79400d3b0 comp rx=0 tx=0).stop 2023-07-14T19:30:30.903 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:30.901+0000 7ff7a31a0700 1 -- 172.21.15.98:0/302434192 shutdown_connections 2023-07-14T19:30:30.903 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:30.901+0000 7ff7a31a0700 1 --2- 172.21.15.98:0/302434192 >> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] conn(0x7ff7880a1290 0x7ff7880a3740 unknown :-1 s=CLOSED pgs=245 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:30:30.903 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:30.901+0000 7ff7a31a0700 1 --2- 172.21.15.98:0/302434192 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7ff79c0fea00 0x7ff79c068330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:30:30.904 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:30.901+0000 7ff7a31a0700 1 --2- 172.21.15.98:0/302434192 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff79c100580 0x7ff79c19cac0 unknown :-1 s=CLOSED pgs=419 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:30:30.904 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:30.901+0000 7ff7a31a0700 1 --2- 172.21.15.98:0/302434192 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7ff79c0ffbf0 0x7ff79c19a470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:30:30.904 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:30.901+0000 7ff7a31a0700 1 -- 172.21.15.98:0/302434192 >> 172.21.15.98:0/302434192 conn(0x7ff79c0fa000 msgr2=0x7ff79c0fc230 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:30:30.904 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:30.902+0000 7ff7a31a0700 1 -- 172.21.15.98:0/302434192 shutdown_connections 2023-07-14T19:30:30.905 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:30.902+0000 7ff7a31a0700 1 -- 172.21.15.98:0/302434192 wait complete. 2023-07-14T19:30:30.917 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2023-07-14T19:30:31.046 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:31.045+0000 7f48bbc78700 1 -- 172.21.15.98:0/2693707494 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f48b41000f0 msgr2=0x7f48b4100500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:30:31.046 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:31.045+0000 7f48bbc78700 1 --2- 172.21.15.98:0/2693707494 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f48b41000f0 0x7f48b4100500 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 crypto rx=0x7f48a4009990 tx=0x7f48a4009ca0 comp rx=0 tx=0).stop 2023-07-14T19:30:31.047 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:31.045+0000 7f48bbc78700 1 -- 172.21.15.98:0/2693707494 shutdown_connections 2023-07-14T19:30:31.047 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:31.045+0000 7f48bbc78700 1 --2- 172.21.15.98:0/2693707494 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f48b40ff6f0 0x7f48b4106fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:30:31.047 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:31.045+0000 7f48bbc78700 1 --2- 172.21.15.98:0/2693707494 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f48b41000f0 0x7f48b4100500 unknown :-1 s=CLOSED pgs=420 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:30:31.047 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:31.045+0000 7f48bbc78700 1 --2- 172.21.15.98:0/2693707494 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f48b40fed40 0x7f48b40ff1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:30:31.047 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:31.045+0000 7f48bbc78700 1 -- 172.21.15.98:0/2693707494 >> 172.21.15.98:0/2693707494 conn(0x7f48b40f98e0 msgr2=0x7f48b40fbd10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:30:31.048 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:31.045+0000 7f48bbc78700 1 -- 172.21.15.98:0/2693707494 shutdown_connections 2023-07-14T19:30:31.048 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:31.045+0000 7f48bbc78700 1 -- 172.21.15.98:0/2693707494 wait complete. 2023-07-14T19:30:31.050 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:31.046+0000 7f48bbc78700 1 Processor -- start 2023-07-14T19:30:31.050 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:31.046+0000 7f48bbc78700 1 -- start start 2023-07-14T19:30:31.051 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:31.046+0000 7f48bbc78700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f48b40fed40 0x7f48b41959d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:30:31.051 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:31.046+0000 7f48bbc78700 1 --2- >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f48b40ff6f0 0x7f48b4195f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:30:31.051 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:31.046+0000 7f48bbc78700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f48b41000f0 0x7f48b419cf80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:30:31.051 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:31.046+0000 7f48bbc78700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f48b419d550 con 0x7f48b41000f0 2023-07-14T19:30:31.051 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:31.046+0000 7f48bbc78700 1 -- --> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f48b419d690 con 0x7f48b40ff6f0 2023-07-14T19:30:31.052 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:31.046+0000 7f48bbc78700 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f48b419d820 con 0x7f48b40fed40 2023-07-14T19:30:31.052 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:31.047+0000 7f48ba215700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f48b41000f0 0x7f48b419cf80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:30:31.052 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:31.047+0000 7f48ba215700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f48b41000f0 0x7f48b419cf80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:47162/0 (socket says 172.21.15.98:47162) 2023-07-14T19:30:31.052 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:31.047+0000 7f48ba215700 1 -- 172.21.15.98:0/3818728386 learned_addr learned my addr 172.21.15.98:0/3818728386 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-07-14T19:30:31.053 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:31.047+0000 7f48b9a14700 1 --2- 172.21.15.98:0/3818728386 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f48b40fed40 0x7f48b41959d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:30:31.053 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:31.047+0000 7f48b9213700 1 --2- 172.21.15.98:0/3818728386 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f48b40ff6f0 0x7f48b4195f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:30:31.053 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:31.047+0000 7f48ba215700 1 -- 172.21.15.98:0/3818728386 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f48b40fed40 msgr2=0x7f48b41959d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:30:31.053 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:31.047+0000 7f48ba215700 1 --2- 172.21.15.98:0/3818728386 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f48b40fed40 0x7f48b41959d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:30:31.053 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:31.047+0000 7f48ba215700 1 -- 172.21.15.98:0/3818728386 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f48b40ff6f0 msgr2=0x7f48b4195f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:30:31.054 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:31.047+0000 7f48ba215700 1 --2- 172.21.15.98:0/3818728386 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f48b40ff6f0 0x7f48b4195f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:30:31.054 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:31.047+0000 7f48ba215700 1 -- 172.21.15.98:0/3818728386 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f48a40095f0 con 0x7f48b41000f0 2023-07-14T19:30:31.054 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:31.047+0000 7f48ba215700 1 --2- 172.21.15.98:0/3818728386 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f48b41000f0 0x7f48b419cf80 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 crypto rx=0x7f48ac00ada0 tx=0x7f48ac00c330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:30:31.054 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:31.047+0000 7f48aaffd700 1 -- 172.21.15.98:0/3818728386 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f48ac00f070 con 0x7f48b41000f0 2023-07-14T19:30:31.055 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:31.047+0000 7f48aaffd700 1 -- 172.21.15.98:0/3818728386 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f48ac007b30 con 0x7f48b41000f0 2023-07-14T19:30:31.055 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:31.047+0000 7f48bbc78700 1 -- 172.21.15.98:0/3818728386 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f48b419db00 con 0x7f48b41000f0 2023-07-14T19:30:31.055 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:31.047+0000 7f48aaffd700 1 -- 172.21.15.98:0/3818728386 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f48ac0053e0 con 0x7f48b41000f0 2023-07-14T19:30:31.055 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:31.047+0000 7f48bbc78700 1 -- 172.21.15.98:0/3818728386 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f48b419dfc0 con 0x7f48b41000f0 2023-07-14T19:30:31.055 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:31.048+0000 7f48bbc78700 1 -- 172.21.15.98:0/3818728386 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f48b41061a0 con 0x7f48b41000f0 2023-07-14T19:30:31.056 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:31.049+0000 7f48aaffd700 1 -- 172.21.15.98:0/3818728386 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 17) v1 ==== 135766+0+0 (secure 0 0 0) 0x7f48ac00c980 con 0x7f48b41000f0 2023-07-14T19:30:31.056 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:31.050+0000 7f48aaffd700 1 --2- 172.21.15.98:0/3818728386 >> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] conn(0x7f48a00a11e0 0x7f48a00a3690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:30:31.056 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:31.050+0000 7f48b9a14700 1 --2- 172.21.15.98:0/3818728386 >> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] conn(0x7f48a00a11e0 0x7f48a00a3690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:30:31.056 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:31.050+0000 7f48aaffd700 1 -- 172.21.15.98:0/3818728386 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 8469+0+0 (secure 0 0 0) 0x7f48ac0cc6d0 con 0x7f48b41000f0 2023-07-14T19:30:31.057 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:31.050+0000 7f48b9a14700 1 --2- 172.21.15.98:0/3818728386 >> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] conn(0x7f48a00a11e0 0x7f48a00a3690 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f48a400b5c0 tx=0x7f48a4005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:30:31.057 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:31.053+0000 7f48aaffd700 1 -- 172.21.15.98:0/3818728386 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f48ac095c60 con 0x7f48b41000f0 2023-07-14T19:30:31.191 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:30:30 smithi101 ceph-mon[128807]: pgmap v299: 97 pgs: 97 active+clean; 639 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 682 B/s wr, 6 op/s 2023-07-14T19:30:31.192 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:30:30 smithi101 ceph-mon[128807]: from='client.? 172.21.15.98:0/4187650013' entity='client.admin' cmd=[{"prefix": "fs fail", "fs_name": "cephfs"}]: dispatch 2023-07-14T19:30:31.192 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:30:30 smithi101 ceph-mon[128807]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2023-07-14T19:30:31.192 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:30:30 smithi101 ceph-mon[128807]: Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2023-07-14T19:30:31.192 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:30:30 smithi101 ceph-mon[128807]: osdmap e86: 12 total, 12 up, 12 in 2023-07-14T19:30:31.193 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:30:30 smithi101 ceph-mon[128807]: from='client.? 172.21.15.98:0/4187650013' entity='client.admin' cmd='[{"prefix": "fs fail", "fs_name": "cephfs"}]': finished 2023-07-14T19:30:31.193 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:30:30 smithi101 ceph-mon[128807]: fsmap cephfs:0/3 6 up:standby, 3 failed 2023-07-14T19:30:31.193 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:30:30 smithi101 ceph-mon[128807]: from='client.? 172.21.15.98:0/302434192' entity='client.admin' cmd=[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]: dispatch 2023-07-14T19:30:31.220 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:30:30 smithi139 ceph-mon[129293]: pgmap v299: 97 pgs: 97 active+clean; 639 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 682 B/s wr, 6 op/s 2023-07-14T19:30:31.221 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:30:30 smithi139 ceph-mon[129293]: from='client.? 172.21.15.98:0/4187650013' entity='client.admin' cmd=[{"prefix": "fs fail", "fs_name": "cephfs"}]: dispatch 2023-07-14T19:30:31.221 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:30:30 smithi139 ceph-mon[129293]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2023-07-14T19:30:31.221 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:30:30 smithi139 ceph-mon[129293]: Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2023-07-14T19:30:31.222 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:30:30 smithi139 ceph-mon[129293]: osdmap e86: 12 total, 12 up, 12 in 2023-07-14T19:30:31.222 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:30:30 smithi139 ceph-mon[129293]: from='client.? 172.21.15.98:0/4187650013' entity='client.admin' cmd='[{"prefix": "fs fail", "fs_name": "cephfs"}]': finished 2023-07-14T19:30:31.223 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:30:30 smithi139 ceph-mon[129293]: fsmap cephfs:0/3 6 up:standby, 3 failed 2023-07-14T19:30:31.223 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:30:30 smithi139 ceph-mon[129293]: from='client.? 172.21.15.98:0/302434192' entity='client.admin' cmd=[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]: dispatch 2023-07-14T19:30:31.246 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:31.245+0000 7f48bbc78700 1 -- 172.21.15.98:0/3818728386 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f48b402c8e0 con 0x7f48b41000f0 2023-07-14T19:30:31.281 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:30:30 smithi098 ceph-mon[127486]: pgmap v299: 97 pgs: 97 active+clean; 639 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 682 B/s wr, 6 op/s 2023-07-14T19:30:31.281 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:30:30 smithi098 ceph-mon[127486]: from='client.? 172.21.15.98:0/4187650013' entity='client.admin' cmd=[{"prefix": "fs fail", "fs_name": "cephfs"}]: dispatch 2023-07-14T19:30:31.282 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:30:30 smithi098 ceph-mon[127486]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2023-07-14T19:30:31.282 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:30:30 smithi098 ceph-mon[127486]: Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2023-07-14T19:30:31.283 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:30:30 smithi098 ceph-mon[127486]: osdmap e86: 12 total, 12 up, 12 in 2023-07-14T19:30:31.283 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:30:30 smithi098 ceph-mon[127486]: from='client.? 172.21.15.98:0/4187650013' entity='client.admin' cmd='[{"prefix": "fs fail", "fs_name": "cephfs"}]': finished 2023-07-14T19:30:31.283 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:30:30 smithi098 ceph-mon[127486]: fsmap cephfs:0/3 6 up:standby, 3 failed 2023-07-14T19:30:31.284 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:30:30 smithi098 ceph-mon[127486]: from='client.? 172.21.15.98:0/302434192' entity='client.admin' cmd=[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]: dispatch 2023-07-14T19:30:31.901 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:31.901+0000 7f48aaffd700 1 -- 172.21.15.98:0/3818728386 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v87) v1 ==== 181+0+0 (secure 0 0 0) 0x7f48ac0840d0 con 0x7f48b41000f0 2023-07-14T19:30:31.904 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:31.903+0000 7f48bbc78700 1 -- 172.21.15.98:0/3818728386 >> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] conn(0x7f48a00a11e0 msgr2=0x7f48a00a3690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:30:31.905 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:31.903+0000 7f48bbc78700 1 --2- 172.21.15.98:0/3818728386 >> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] conn(0x7f48a00a11e0 0x7f48a00a3690 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f48a400b5c0 tx=0x7f48a4005fd0 comp rx=0 tx=0).stop 2023-07-14T19:30:31.905 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:31.903+0000 7f48bbc78700 1 -- 172.21.15.98:0/3818728386 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f48b41000f0 msgr2=0x7f48b419cf80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:30:31.905 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:31.903+0000 7f48bbc78700 1 --2- 172.21.15.98:0/3818728386 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f48b41000f0 0x7f48b419cf80 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 crypto rx=0x7f48ac00ada0 tx=0x7f48ac00c330 comp rx=0 tx=0).stop 2023-07-14T19:30:31.905 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:31.904+0000 7f48bbc78700 1 -- 172.21.15.98:0/3818728386 shutdown_connections 2023-07-14T19:30:31.905 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:31.904+0000 7f48bbc78700 1 --2- 172.21.15.98:0/3818728386 >> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] conn(0x7f48a00a11e0 0x7f48a00a3690 unknown :-1 s=CLOSED pgs=252 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:30:31.906 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:31.904+0000 7f48bbc78700 1 --2- 172.21.15.98:0/3818728386 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f48b40fed40 0x7f48b41959d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:30:31.906 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:31.904+0000 7f48bbc78700 1 --2- 172.21.15.98:0/3818728386 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f48b41000f0 0x7f48b419cf80 unknown :-1 s=CLOSED pgs=421 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:30:31.906 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:31.904+0000 7f48bbc78700 1 --2- 172.21.15.98:0/3818728386 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f48b40ff6f0 0x7f48b4195f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:30:31.906 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:31.904+0000 7f48bbc78700 1 -- 172.21.15.98:0/3818728386 >> 172.21.15.98:0/3818728386 conn(0x7f48b40f98e0 msgr2=0x7f48b41032e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:30:31.906 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:31.904+0000 7f48bbc78700 1 -- 172.21.15.98:0/3818728386 shutdown_connections 2023-07-14T19:30:31.907 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:31.904+0000 7f48bbc78700 1 -- 172.21.15.98:0/3818728386 wait complete. 2023-07-14T19:30:31.907 INFO:teuthology.orchestra.run.smithi098.stderr:pool 'cephfs_metadata' removed 2023-07-14T19:30:31.917 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2023-07-14T19:30:32.044 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:32.043+0000 7f00a132f700 1 -- 172.21.15.98:0/1442087666 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f009c0feb10 msgr2=0x7f009c0fef20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:30:32.045 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:32.043+0000 7f00a132f700 1 --2- 172.21.15.98:0/1442087666 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f009c0feb10 0x7f009c0fef20 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 crypto rx=0x7f0090009990 tx=0x7f0090009ca0 comp rx=0 tx=0).stop 2023-07-14T19:30:32.045 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:32.043+0000 7f00a132f700 1 -- 172.21.15.98:0/1442087666 shutdown_connections 2023-07-14T19:30:32.046 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:32.043+0000 7f00a132f700 1 --2- 172.21.15.98:0/1442087666 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f009c100690 0x7f009c106f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:30:32.046 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:32.043+0000 7f00a132f700 1 --2- 172.21.15.98:0/1442087666 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f009c0feb10 0x7f009c0fef20 unknown :-1 s=CLOSED pgs=422 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:30:32.046 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:32.043+0000 7f00a132f700 1 --2- 172.21.15.98:0/1442087666 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f009c0ffd00 0x7f009c100150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:30:32.046 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:32.043+0000 7f00a132f700 1 -- 172.21.15.98:0/1442087666 >> 172.21.15.98:0/1442087666 conn(0x7f009c0fa130 msgr2=0x7f009c0fc560 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:30:32.046 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:32.043+0000 7f00a132f700 1 -- 172.21.15.98:0/1442087666 shutdown_connections 2023-07-14T19:30:32.047 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:32.043+0000 7f00a132f700 1 -- 172.21.15.98:0/1442087666 wait complete. 2023-07-14T19:30:32.047 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:32.043+0000 7f00a132f700 1 Processor -- start 2023-07-14T19:30:32.049 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:32.043+0000 7f00a132f700 1 -- start start 2023-07-14T19:30:32.049 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:32.044+0000 7f00a132f700 1 --2- >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f009c0feb10 0x7f009c195980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:30:32.050 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:32.044+0000 7f00a132f700 1 --2- >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f009c0ffd00 0x7f009c195ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:30:32.050 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:32.044+0000 7f00a132f700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f009c100690 0x7f009c19cf30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:30:32.050 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:32.044+0000 7f00a132f700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f009c19d500 con 0x7f009c100690 2023-07-14T19:30:32.050 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:32.044+0000 7f00a132f700 1 -- --> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f009c19d640 con 0x7f009c0ffd00 2023-07-14T19:30:32.050 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:32.044+0000 7f00a132f700 1 -- --> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f009c19d7d0 con 0x7f009c0feb10 2023-07-14T19:30:32.051 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:32.044+0000 7f009b7fe700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f009c100690 0x7f009c19cf30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:30:32.051 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:32.044+0000 7f009b7fe700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f009c100690 0x7f009c19cf30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:47176/0 (socket says 172.21.15.98:47176) 2023-07-14T19:30:32.051 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:32.044+0000 7f009b7fe700 1 -- 172.21.15.98:0/2126444364 learned_addr learned my addr 172.21.15.98:0/2126444364 (peer_addr_for_me v2:172.21.15.98:0/0) 2023-07-14T19:30:32.051 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:32.044+0000 7f009affd700 1 --2- 172.21.15.98:0/2126444364 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f009c0feb10 0x7f009c195980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:30:32.051 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:32.044+0000 7f009a7fc700 1 --2- 172.21.15.98:0/2126444364 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f009c0ffd00 0x7f009c195ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:30:32.052 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:32.044+0000 7f009b7fe700 1 -- 172.21.15.98:0/2126444364 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f009c0feb10 msgr2=0x7f009c195980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:30:32.052 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:32.044+0000 7f009b7fe700 1 --2- 172.21.15.98:0/2126444364 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f009c0feb10 0x7f009c195980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:30:32.052 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:32.044+0000 7f009b7fe700 1 -- 172.21.15.98:0/2126444364 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f009c0ffd00 msgr2=0x7f009c195ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:30:32.052 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:32.044+0000 7f009b7fe700 1 --2- 172.21.15.98:0/2126444364 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f009c0ffd00 0x7f009c195ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:30:32.053 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:32.044+0000 7f009b7fe700 1 -- 172.21.15.98:0/2126444364 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f00900095f0 con 0x7f009c100690 2023-07-14T19:30:32.053 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:32.044+0000 7f009b7fe700 1 --2- 172.21.15.98:0/2126444364 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f009c100690 0x7f009c19cf30 secure :-1 s=READY pgs=423 cs=0 l=1 rev1=1 crypto rx=0x7f0094007ee0 tx=0x7f009400d3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:30:32.053 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:32.044+0000 7f0083fff700 1 -- 172.21.15.98:0/2126444364 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0094017070 con 0x7f009c100690 2023-07-14T19:30:32.053 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:32.044+0000 7f0083fff700 1 -- 172.21.15.98:0/2126444364 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f009400f040 con 0x7f009c100690 2023-07-14T19:30:32.053 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:32.044+0000 7f00a132f700 1 -- 172.21.15.98:0/2126444364 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f009c19dab0 con 0x7f009c100690 2023-07-14T19:30:32.054 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:32.044+0000 7f0083fff700 1 -- 172.21.15.98:0/2126444364 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0094005580 con 0x7f009c100690 2023-07-14T19:30:32.054 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:32.044+0000 7f00a132f700 1 -- 172.21.15.98:0/2126444364 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f009c19dfc0 con 0x7f009c100690 2023-07-14T19:30:32.054 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:32.045+0000 7f00a132f700 1 -- 172.21.15.98:0/2126444364 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f009c10a2e0 con 0x7f009c100690 2023-07-14T19:30:32.054 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:32.046+0000 7f0083fff700 1 -- 172.21.15.98:0/2126444364 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 17) v1 ==== 135766+0+0 (secure 0 0 0) 0x7f009400db20 con 0x7f009c100690 2023-07-14T19:30:32.055 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:32.047+0000 7f0083fff700 1 --2- 172.21.15.98:0/2126444364 >> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] conn(0x7f00840a11e0 0x7f00840a3690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:30:32.055 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:32.047+0000 7f009affd700 1 --2- 172.21.15.98:0/2126444364 >> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] conn(0x7f00840a11e0 0x7f00840a3690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:30:32.055 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:32.047+0000 7f0083fff700 1 -- 172.21.15.98:0/2126444364 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 7973+0+0 (secure 0 0 0) 0x7f00940cbff0 con 0x7f009c100690 2023-07-14T19:30:32.055 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:32.047+0000 7f009affd700 1 --2- 172.21.15.98:0/2126444364 >> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] conn(0x7f00840a11e0 0x7f00840a3690 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f009000b5c0 tx=0x7f0090005960 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:30:32.056 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:32.051+0000 7f0083fff700 1 -- 172.21.15.98:0/2126444364 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f0094095790 con 0x7f009c100690 2023-07-14T19:30:32.191 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:30:31 smithi101 ceph-mon[128807]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2023-07-14T19:30:32.191 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:30:31 smithi101 ceph-mon[128807]: Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2023-07-14T19:30:32.192 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:30:31 smithi101 ceph-mon[128807]: Cluster is now healthy 2023-07-14T19:30:32.192 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:30:31 smithi101 ceph-mon[128807]: mds.? [v2:172.21.15.101:6835/3226284830,v1:172.21.15.101:6837/3226284830] up:boot 2023-07-14T19:30:32.192 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:30:31 smithi101 ceph-mon[128807]: mds.? [v2:172.21.15.139:6838/4008487421,v1:172.21.15.139:6839/4008487421] up:boot 2023-07-14T19:30:32.192 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:30:31 smithi101 ceph-mon[128807]: mds.? [v2:172.21.15.101:6838/2160653963,v1:172.21.15.101:6839/2160653963] up:boot 2023-07-14T19:30:32.193 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:30:31 smithi101 ceph-mon[128807]: mds.? [v2:172.21.15.101:6834/3960790974,v1:172.21.15.101:6836/3960790974] up:boot 2023-07-14T19:30:32.193 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:30:31 smithi101 ceph-mon[128807]: mds.? [v2:172.21.15.139:6834/2077338435,v1:172.21.15.139:6835/2077338435] up:boot 2023-07-14T19:30:32.193 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:30:31 smithi101 ceph-mon[128807]: mds.? [v2:172.21.15.139:6836/3057937674,v1:172.21.15.139:6837/3057937674] up:boot 2023-07-14T19:30:32.193 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:30:31 smithi101 ceph-mon[128807]: from='client.? 172.21.15.98:0/302434192' entity='client.admin' cmd='[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]': finished 2023-07-14T19:30:32.194 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:30:31 smithi101 ceph-mon[128807]: fsmap 12 up:standby 2023-07-14T19:30:32.194 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:30:31 smithi101 ceph-mon[128807]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2023-07-14T19:30:32.194 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:30:31 smithi101 ceph-mon[128807]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2023-07-14T19:30:32.194 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:30:31 smithi101 ceph-mon[128807]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2023-07-14T19:30:32.194 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:30:31 smithi101 ceph-mon[128807]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2023-07-14T19:30:32.195 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:30:31 smithi101 ceph-mon[128807]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2023-07-14T19:30:32.195 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:30:31 smithi101 ceph-mon[128807]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2023-07-14T19:30:32.195 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:30:31 smithi101 ceph-mon[128807]: from='client.? 172.21.15.98:0/3818728386' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]: dispatch 2023-07-14T19:30:32.220 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:30:31 smithi139 ceph-mon[129293]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2023-07-14T19:30:32.220 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:30:31 smithi139 ceph-mon[129293]: Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2023-07-14T19:30:32.221 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:30:31 smithi139 ceph-mon[129293]: Cluster is now healthy 2023-07-14T19:30:32.221 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:30:31 smithi139 ceph-mon[129293]: mds.? [v2:172.21.15.101:6835/3226284830,v1:172.21.15.101:6837/3226284830] up:boot 2023-07-14T19:30:32.221 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:30:31 smithi139 ceph-mon[129293]: mds.? [v2:172.21.15.139:6838/4008487421,v1:172.21.15.139:6839/4008487421] up:boot 2023-07-14T19:30:32.221 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:30:31 smithi139 ceph-mon[129293]: mds.? [v2:172.21.15.101:6838/2160653963,v1:172.21.15.101:6839/2160653963] up:boot 2023-07-14T19:30:32.222 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:30:31 smithi139 ceph-mon[129293]: mds.? [v2:172.21.15.101:6834/3960790974,v1:172.21.15.101:6836/3960790974] up:boot 2023-07-14T19:30:32.222 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:30:31 smithi139 ceph-mon[129293]: mds.? [v2:172.21.15.139:6834/2077338435,v1:172.21.15.139:6835/2077338435] up:boot 2023-07-14T19:30:32.222 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:30:31 smithi139 ceph-mon[129293]: mds.? [v2:172.21.15.139:6836/3057937674,v1:172.21.15.139:6837/3057937674] up:boot 2023-07-14T19:30:32.222 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:30:31 smithi139 ceph-mon[129293]: from='client.? 172.21.15.98:0/302434192' entity='client.admin' cmd='[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]': finished 2023-07-14T19:30:32.223 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:30:31 smithi139 ceph-mon[129293]: fsmap 12 up:standby 2023-07-14T19:30:32.223 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:30:31 smithi139 ceph-mon[129293]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2023-07-14T19:30:32.223 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:30:31 smithi139 ceph-mon[129293]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2023-07-14T19:30:32.223 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:30:31 smithi139 ceph-mon[129293]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2023-07-14T19:30:32.224 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:30:31 smithi139 ceph-mon[129293]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2023-07-14T19:30:32.224 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:30:31 smithi139 ceph-mon[129293]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2023-07-14T19:30:32.224 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:30:31 smithi139 ceph-mon[129293]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2023-07-14T19:30:32.224 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:30:31 smithi139 ceph-mon[129293]: from='client.? 172.21.15.98:0/3818728386' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]: dispatch 2023-07-14T19:30:32.246 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:32.244+0000 7f00a132f700 1 -- 172.21.15.98:0/2126444364 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f009c10a4a0 con 0x7f009c100690 2023-07-14T19:30:32.281 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:30:31 smithi098 ceph-mon[127486]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2023-07-14T19:30:32.281 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:30:31 smithi098 ceph-mon[127486]: Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2023-07-14T19:30:32.282 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:30:31 smithi098 ceph-mon[127486]: Cluster is now healthy 2023-07-14T19:30:32.282 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:30:31 smithi098 ceph-mon[127486]: mds.? [v2:172.21.15.101:6835/3226284830,v1:172.21.15.101:6837/3226284830] up:boot 2023-07-14T19:30:32.282 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:30:31 smithi098 ceph-mon[127486]: mds.? [v2:172.21.15.139:6838/4008487421,v1:172.21.15.139:6839/4008487421] up:boot 2023-07-14T19:30:32.282 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:30:31 smithi098 ceph-mon[127486]: mds.? [v2:172.21.15.101:6838/2160653963,v1:172.21.15.101:6839/2160653963] up:boot 2023-07-14T19:30:32.282 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:30:31 smithi098 ceph-mon[127486]: mds.? [v2:172.21.15.101:6834/3960790974,v1:172.21.15.101:6836/3960790974] up:boot 2023-07-14T19:30:32.283 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:30:31 smithi098 ceph-mon[127486]: mds.? [v2:172.21.15.139:6834/2077338435,v1:172.21.15.139:6835/2077338435] up:boot 2023-07-14T19:30:32.283 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:30:31 smithi098 ceph-mon[127486]: mds.? [v2:172.21.15.139:6836/3057937674,v1:172.21.15.139:6837/3057937674] up:boot 2023-07-14T19:30:32.283 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:30:31 smithi098 ceph-mon[127486]: from='client.? 172.21.15.98:0/302434192' entity='client.admin' cmd='[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]': finished 2023-07-14T19:30:32.283 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:30:31 smithi098 ceph-mon[127486]: fsmap 12 up:standby 2023-07-14T19:30:32.283 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:30:31 smithi098 ceph-mon[127486]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2023-07-14T19:30:32.284 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:30:31 smithi098 ceph-mon[127486]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2023-07-14T19:30:32.284 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:30:31 smithi098 ceph-mon[127486]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2023-07-14T19:30:32.284 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:30:31 smithi098 ceph-mon[127486]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2023-07-14T19:30:32.284 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:30:31 smithi098 ceph-mon[127486]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2023-07-14T19:30:32.284 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:30:31 smithi098 ceph-mon[127486]: from='mgr.14152 172.21.15.98:0/2785157538' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2023-07-14T19:30:32.285 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:30:31 smithi098 ceph-mon[127486]: from='client.? 172.21.15.98:0/3818728386' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]: dispatch 2023-07-14T19:30:32.902 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:32.900+0000 7f0083fff700 1 -- 172.21.15.98:0/2126444364 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v88) v1 ==== 169+0+0 (secure 0 0 0) 0x7f0094098db0 con 0x7f009c100690 2023-07-14T19:30:32.905 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:32.904+0000 7f00a132f700 1 -- 172.21.15.98:0/2126444364 >> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] conn(0x7f00840a11e0 msgr2=0x7f00840a3690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:30:32.905 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:32.904+0000 7f00a132f700 1 --2- 172.21.15.98:0/2126444364 >> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] conn(0x7f00840a11e0 0x7f00840a3690 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f009000b5c0 tx=0x7f0090005960 comp rx=0 tx=0).stop 2023-07-14T19:30:32.906 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:32.904+0000 7f00a132f700 1 -- 172.21.15.98:0/2126444364 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f009c100690 msgr2=0x7f009c19cf30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:30:32.906 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:32.904+0000 7f00a132f700 1 --2- 172.21.15.98:0/2126444364 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f009c100690 0x7f009c19cf30 secure :-1 s=READY pgs=423 cs=0 l=1 rev1=1 crypto rx=0x7f0094007ee0 tx=0x7f009400d3b0 comp rx=0 tx=0).stop 2023-07-14T19:30:32.906 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:32.904+0000 7f00a132f700 1 -- 172.21.15.98:0/2126444364 shutdown_connections 2023-07-14T19:30:32.906 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:32.904+0000 7f00a132f700 1 --2- 172.21.15.98:0/2126444364 >> [v2:172.21.15.98:6800/452042164,v1:172.21.15.98:6801/452042164] conn(0x7f00840a11e0 0x7f00840a3690 unknown :-1 s=CLOSED pgs=253 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:30:32.906 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:32.904+0000 7f00a132f700 1 --2- 172.21.15.98:0/2126444364 >> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] conn(0x7f009c0feb10 0x7f009c195980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:30:32.907 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:32.904+0000 7f00a132f700 1 --2- 172.21.15.98:0/2126444364 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f009c100690 0x7f009c19cf30 unknown :-1 s=CLOSED pgs=423 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:30:32.907 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:32.904+0000 7f00a132f700 1 --2- 172.21.15.98:0/2126444364 >> [v2:172.21.15.101:3300/0,v1:172.21.15.101:6789/0] conn(0x7f009c0ffd00 0x7f009c195ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:30:32.907 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:32.904+0000 7f00a132f700 1 -- 172.21.15.98:0/2126444364 >> 172.21.15.98:0/2126444364 conn(0x7f009c0fa130 msgr2=0x7f009c068890 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:30:32.907 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:32.904+0000 7f00a132f700 1 -- 172.21.15.98:0/2126444364 shutdown_connections 2023-07-14T19:30:32.908 INFO:teuthology.orchestra.run.smithi098.stderr:2023-07-14T19:30:32.904+0000 7f00a132f700 1 -- 172.21.15.98:0/2126444364 wait complete. 2023-07-14T19:30:32.908 INFO:teuthology.orchestra.run.smithi098.stderr:pool 'cephfs_data' removed 2023-07-14T19:30:32.921 DEBUG:teuthology.orchestra.run.smithi098:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2023-07-14T19:30:32.950 DEBUG:teuthology.orchestra.run.smithi101:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2023-07-14T19:30:32.986 DEBUG:teuthology.orchestra.run.smithi139:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2023-07-14T19:30:33.026 INFO:tasks.cephadm:Cleaning up testdir ceph.* files... 2023-07-14T19:30:33.026 DEBUG:teuthology.orchestra.run.smithi098:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2023-07-14T19:30:33.045 DEBUG:teuthology.orchestra.run.smithi101:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2023-07-14T19:30:33.063 DEBUG:teuthology.orchestra.run.smithi139:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2023-07-14T19:30:33.085 INFO:tasks.cephadm:Stopping all daemons... 2023-07-14T19:30:33.086 INFO:tasks.cephadm.mon.a:Stopping mon.a... 2023-07-14T19:30:33.086 DEBUG:teuthology.orchestra.run.smithi098:> sudo systemctl stop ceph-9adea648-227b-11ee-9b33-001a4aab830c@mon.a 2023-07-14T19:30:33.191 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:30:32 smithi101 ceph-mon[128807]: pgmap v301: 97 pgs: 97 active+clean; 639 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 1.1 KiB/s wr, 6 op/s 2023-07-14T19:30:33.191 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:30:32 smithi101 ceph-mon[128807]: from='client.? 172.21.15.98:0/3818728386' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]': finished 2023-07-14T19:30:33.192 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:30:32 smithi101 ceph-mon[128807]: osdmap e87: 12 total, 12 up, 12 in 2023-07-14T19:30:33.192 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:30:32 smithi101 ceph-mon[128807]: from='client.? 172.21.15.98:0/2126444364' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]: dispatch 2023-07-14T19:30:33.220 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:30:32 smithi139 ceph-mon[129293]: pgmap v301: 97 pgs: 97 active+clean; 639 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 1.1 KiB/s wr, 6 op/s 2023-07-14T19:30:33.220 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:30:32 smithi139 ceph-mon[129293]: from='client.? 172.21.15.98:0/3818728386' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]': finished 2023-07-14T19:30:33.221 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:30:32 smithi139 ceph-mon[129293]: osdmap e87: 12 total, 12 up, 12 in 2023-07-14T19:30:33.221 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:30:32 smithi139 ceph-mon[129293]: from='client.? 172.21.15.98:0/2126444364' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]: dispatch 2023-07-14T19:30:33.281 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:30:32 smithi098 ceph-mon[127486]: pgmap v301: 97 pgs: 97 active+clean; 639 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 1.1 KiB/s wr, 6 op/s 2023-07-14T19:30:33.282 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:30:32 smithi098 ceph-mon[127486]: from='client.? 172.21.15.98:0/3818728386' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]': finished 2023-07-14T19:30:33.282 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:30:32 smithi098 ceph-mon[127486]: osdmap e87: 12 total, 12 up, 12 in 2023-07-14T19:30:33.282 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:30:32 smithi098 ceph-mon[127486]: from='client.? 172.21.15.98:0/2126444364' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]: dispatch 2023-07-14T19:30:33.282 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:30:33 smithi098 systemd[1]: Stopping Ceph mon.a for 9adea648-227b-11ee-9b33-001a4aab830c... 2023-07-14T19:30:33.690 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:30:33 smithi098 ceph-9adea648-227b-11ee-9b33-001a4aab830c-mon-a[127462]: 2023-07-14T19:30:33.424+0000 7f6e64ac2700 -1 received signal: Terminated from /dev/init -- /usr/bin/ceph-mon -n mon.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2023-07-14T19:30:33.691 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:30:33 smithi098 ceph-9adea648-227b-11ee-9b33-001a4aab830c-mon-a[127462]: 2023-07-14T19:30:33.424+0000 7f6e64ac2700 -1 mon.a@0(leader) e3 *** Got Signal Terminated *** 2023-07-14T19:30:33.971 INFO:journalctl@ceph.mon.a.smithi098.stdout:Jul 14 19:30:33 smithi098 bash[176190]: ceph-9adea648-227b-11ee-9b33-001a4aab830c-mon-a 2023-07-14T19:30:33.979 DEBUG:teuthology.orchestra.run.smithi098:> sudo pkill -f 'journalctl -f -n 0 -u ceph-9adea648-227b-11ee-9b33-001a4aab830c@mon.a.service' 2023-07-14T19:30:34.052 DEBUG:teuthology.orchestra.run:got remote process result: None 2023-07-14T19:30:34.052 INFO:tasks.cephadm.mon.a:Stopped mon.a 2023-07-14T19:30:34.053 INFO:tasks.cephadm.mon.c:Stopping mon.b... 2023-07-14T19:30:34.053 DEBUG:teuthology.orchestra.run.smithi101:> sudo systemctl stop ceph-9adea648-227b-11ee-9b33-001a4aab830c@mon.b 2023-07-14T19:30:34.429 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:30:34 smithi101 systemd[1]: Stopping Ceph mon.b for 9adea648-227b-11ee-9b33-001a4aab830c... 2023-07-14T19:30:34.691 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:30:34 smithi101 ceph-9adea648-227b-11ee-9b33-001a4aab830c-mon-b[128782]: 2023-07-14T19:30:34.426+0000 7fa5feb59700 -1 received signal: Terminated from /dev/init -- /usr/bin/ceph-mon -n mon.b -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2023-07-14T19:30:34.691 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:30:34 smithi101 ceph-9adea648-227b-11ee-9b33-001a4aab830c-mon-b[128782]: 2023-07-14T19:30:34.426+0000 7fa5feb59700 -1 mon.b@2(peon) e3 *** Got Signal Terminated *** 2023-07-14T19:30:35.071 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 14 19:30:35 smithi101 bash[155820]: ceph-9adea648-227b-11ee-9b33-001a4aab830c-mon-b 2023-07-14T19:30:35.459 DEBUG:teuthology.orchestra.run.smithi101:> sudo pkill -f 'journalctl -f -n 0 -u ceph-9adea648-227b-11ee-9b33-001a4aab830c@mon.b.service' 2023-07-14T19:30:35.515 DEBUG:teuthology.orchestra.run:got remote process result: None 2023-07-14T19:30:35.516 INFO:tasks.cephadm.mon.c:Stopped mon.b 2023-07-14T19:30:35.516 INFO:tasks.cephadm.mon.c:Stopping mon.c... 2023-07-14T19:30:35.517 DEBUG:teuthology.orchestra.run.smithi139:> sudo systemctl stop ceph-9adea648-227b-11ee-9b33-001a4aab830c@mon.c 2023-07-14T19:30:35.887 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:30:35 smithi139 systemd[1]: Stopping Ceph mon.c for 9adea648-227b-11ee-9b33-001a4aab830c... 2023-07-14T19:30:36.201 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:30:35 smithi139 ceph-9adea648-227b-11ee-9b33-001a4aab830c-mon-c[129269]: 2023-07-14T19:30:35.884+0000 7f70d1f21700 -1 received signal: Terminated from /dev/init -- /usr/bin/ceph-mon -n mon.c -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2023-07-14T19:30:36.202 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:30:35 smithi139 ceph-9adea648-227b-11ee-9b33-001a4aab830c-mon-c[129269]: 2023-07-14T19:30:35.884+0000 7f70d1f21700 -1 mon.c@1(peon) e3 *** Got Signal Terminated *** 2023-07-14T19:30:36.429 DEBUG:teuthology.orchestra.run.smithi139:> sudo pkill -f 'journalctl -f -n 0 -u ceph-9adea648-227b-11ee-9b33-001a4aab830c@mon.c.service' 2023-07-14T19:30:36.454 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:30:36 smithi139 bash[157489]: ceph-9adea648-227b-11ee-9b33-001a4aab830c-mon-c 2023-07-14T19:30:36.455 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:30:36 smithi139 systemd[1]: ceph-9adea648-227b-11ee-9b33-001a4aab830c@mon.c.service: Succeeded. 2023-07-14T19:30:36.455 INFO:journalctl@ceph.mon.c.smithi139.stdout:Jul 14 19:30:36 smithi139 systemd[1]: Stopped Ceph mon.c for 9adea648-227b-11ee-9b33-001a4aab830c. 2023-07-14T19:30:36.484 DEBUG:teuthology.orchestra.run:got remote process result: None 2023-07-14T19:30:36.484 INFO:tasks.cephadm.mon.c:Stopped mon.c 2023-07-14T19:30:36.485 INFO:tasks.cephadm.mgr.x:Stopping mgr.x... 2023-07-14T19:30:36.486 DEBUG:teuthology.orchestra.run.smithi098:> sudo systemctl stop ceph-9adea648-227b-11ee-9b33-001a4aab830c@mgr.x 2023-07-14T19:30:36.781 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Jul 14 19:30:36 smithi098 systemd[1]: Stopping Ceph mgr.x for 9adea648-227b-11ee-9b33-001a4aab830c... 2023-07-14T19:30:37.123 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Jul 14 19:30:36 smithi098 ceph-9adea648-227b-11ee-9b33-001a4aab830c-mgr-x[127734]: 2023-07-14T19:30:36.832+0000 7f24e8021700 -1 received signal: Terminated from /dev/init -- /usr/bin/ceph-mgr -n mgr.x -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2023-07-14T19:30:37.371 DEBUG:teuthology.orchestra.run.smithi098:> sudo pkill -f 'journalctl -f -n 0 -u ceph-9adea648-227b-11ee-9b33-001a4aab830c@mgr.x.service' 2023-07-14T19:30:37.397 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Jul 14 19:30:37 smithi098 bash[176701]: ceph-9adea648-227b-11ee-9b33-001a4aab830c-mgr-x 2023-07-14T19:30:37.397 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Jul 14 19:30:37 smithi098 systemd[1]: ceph-9adea648-227b-11ee-9b33-001a4aab830c@mgr.x.service: Succeeded. 2023-07-14T19:30:37.397 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Jul 14 19:30:37 smithi098 systemd[1]: Stopped Ceph mgr.x for 9adea648-227b-11ee-9b33-001a4aab830c. 2023-07-14T19:30:37.411 DEBUG:teuthology.orchestra.run:got remote process result: None 2023-07-14T19:30:37.411 INFO:tasks.cephadm.mgr.x:Stopped mgr.x 2023-07-14T19:30:37.412 INFO:tasks.cephadm.mgr.y:Stopping mgr.y... 2023-07-14T19:30:37.412 DEBUG:teuthology.orchestra.run.smithi101:> sudo systemctl stop ceph-9adea648-227b-11ee-9b33-001a4aab830c@mgr.y 2023-07-14T19:30:37.691 INFO:journalctl@ceph.mgr.y.smithi101.stdout:Jul 14 19:30:37 smithi101 systemd[1]: Stopping Ceph mgr.y for 9adea648-227b-11ee-9b33-001a4aab830c... 2023-07-14T19:30:38.112 INFO:journalctl@ceph.mgr.y.smithi101.stdout:Jul 14 19:30:38 smithi101 bash[156210]: ceph-9adea648-227b-11ee-9b33-001a4aab830c-mgr-y 2023-07-14T19:30:38.295 INFO:journalctl@ceph.mgr.y.smithi101.stdout:Jul 14 19:30:38 smithi101 systemd[1]: ceph-9adea648-227b-11ee-9b33-001a4aab830c@mgr.y.service: Main process exited, code=exited, status=143/n/a 2023-07-14T19:30:38.298 DEBUG:teuthology.orchestra.run.smithi101:> sudo pkill -f 'journalctl -f -n 0 -u ceph-9adea648-227b-11ee-9b33-001a4aab830c@mgr.y.service' 2023-07-14T19:30:38.336 DEBUG:teuthology.orchestra.run:got remote process result: None 2023-07-14T19:30:38.337 INFO:tasks.cephadm.mgr.y:Stopped mgr.y 2023-07-14T19:30:38.337 INFO:tasks.cephadm.mgr.z:Stopping mgr.z... 2023-07-14T19:30:38.338 DEBUG:teuthology.orchestra.run.smithi139:> sudo systemctl stop ceph-9adea648-227b-11ee-9b33-001a4aab830c@mgr.z 2023-07-14T19:30:38.720 INFO:journalctl@ceph.mgr.z.smithi139.stdout:Jul 14 19:30:38 smithi139 systemd[1]: Stopping Ceph mgr.z for 9adea648-227b-11ee-9b33-001a4aab830c... 2023-07-14T19:30:39.043 INFO:journalctl@ceph.mgr.z.smithi139.stdout:Jul 14 19:30:38 smithi139 bash[157674]: ceph-9adea648-227b-11ee-9b33-001a4aab830c-mgr-z 2023-07-14T19:30:39.221 INFO:journalctl@ceph.mgr.z.smithi139.stdout:Jul 14 19:30:39 smithi139 systemd[1]: ceph-9adea648-227b-11ee-9b33-001a4aab830c@mgr.z.service: Main process exited, code=exited, status=143/n/a 2023-07-14T19:30:39.226 DEBUG:teuthology.orchestra.run.smithi139:> sudo pkill -f 'journalctl -f -n 0 -u ceph-9adea648-227b-11ee-9b33-001a4aab830c@mgr.z.service' 2023-07-14T19:30:39.262 DEBUG:teuthology.orchestra.run:got remote process result: None 2023-07-14T19:30:39.263 INFO:tasks.cephadm.mgr.z:Stopped mgr.z 2023-07-14T19:30:39.263 INFO:tasks.cephadm.osd.0:Stopping osd.0... 2023-07-14T19:30:39.264 DEBUG:teuthology.orchestra.run.smithi098:> sudo systemctl stop ceph-9adea648-227b-11ee-9b33-001a4aab830c@osd.0 2023-07-14T19:30:39.642 INFO:journalctl@ceph.osd.0.smithi098.stdout:Jul 14 19:30:39 smithi098 systemd[1]: Stopping Ceph osd.0 for 9adea648-227b-11ee-9b33-001a4aab830c... 2023-07-14T19:30:40.031 INFO:journalctl@ceph.osd.0.smithi098.stdout:Jul 14 19:30:39 smithi098 ceph-9adea648-227b-11ee-9b33-001a4aab830c-osd-0[140489]: 2023-07-14T19:30:39.640+0000 7fdc5ae24700 -1 received signal: Terminated from /dev/init -- /usr/bin/ceph-osd -n osd.0 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2023-07-14T19:30:40.031 INFO:journalctl@ceph.osd.0.smithi098.stdout:Jul 14 19:30:39 smithi098 ceph-9adea648-227b-11ee-9b33-001a4aab830c-osd-0[140489]: 2023-07-14T19:30:39.640+0000 7fdc5ae24700 -1 osd.0 88 *** Got signal Terminated *** 2023-07-14T19:30:40.032 INFO:journalctl@ceph.osd.0.smithi098.stdout:Jul 14 19:30:39 smithi098 ceph-9adea648-227b-11ee-9b33-001a4aab830c-osd-0[140489]: 2023-07-14T19:30:39.640+0000 7fdc5ae24700 -1 osd.0 88 *** Immediate shutdown (osd_fast_shutdown=true) *** 2023-07-14T19:30:45.781 INFO:journalctl@ceph.osd.0.smithi098.stdout:Jul 14 19:30:45 smithi098 bash[176841]: ceph-9adea648-227b-11ee-9b33-001a4aab830c-osd-0 2023-07-14T19:30:46.740 DEBUG:teuthology.orchestra.run.smithi098:> sudo pkill -f 'journalctl -f -n 0 -u ceph-9adea648-227b-11ee-9b33-001a4aab830c@osd.0.service' 2023-07-14T19:30:46.781 DEBUG:teuthology.orchestra.run:got remote process result: None 2023-07-14T19:30:46.781 INFO:tasks.cephadm.osd.0:Stopped osd.0 2023-07-14T19:30:46.782 INFO:tasks.cephadm.osd.1:Stopping osd.1... 2023-07-14T19:30:46.782 DEBUG:teuthology.orchestra.run.smithi101:> sudo systemctl stop ceph-9adea648-227b-11ee-9b33-001a4aab830c@osd.1 2023-07-14T19:30:47.148 INFO:journalctl@ceph.osd.1.smithi101.stdout:Jul 14 19:30:46 smithi101 systemd[1]: Stopping Ceph osd.1 for 9adea648-227b-11ee-9b33-001a4aab830c... 2023-07-14T19:30:47.441 INFO:journalctl@ceph.osd.1.smithi101.stdout:Jul 14 19:30:47 smithi101 ceph-9adea648-227b-11ee-9b33-001a4aab830c-osd-1[133877]: 2023-07-14T19:30:47.146+0000 7fab7a4f8700 -1 received signal: Terminated from /dev/init -- /usr/bin/ceph-osd -n osd.1 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2023-07-14T19:30:47.441 INFO:journalctl@ceph.osd.1.smithi101.stdout:Jul 14 19:30:47 smithi101 ceph-9adea648-227b-11ee-9b33-001a4aab830c-osd-1[133877]: 2023-07-14T19:30:47.146+0000 7fab7a4f8700 -1 osd.1 88 *** Got signal Terminated *** 2023-07-14T19:30:47.442 INFO:journalctl@ceph.osd.1.smithi101.stdout:Jul 14 19:30:47 smithi101 ceph-9adea648-227b-11ee-9b33-001a4aab830c-osd-1[133877]: 2023-07-14T19:30:47.146+0000 7fab7a4f8700 -1 osd.1 88 *** Immediate shutdown (osd_fast_shutdown=true) *** 2023-07-14T19:30:53.191 INFO:journalctl@ceph.osd.1.smithi101.stdout:Jul 14 19:30:52 smithi101 bash[156357]: ceph-9adea648-227b-11ee-9b33-001a4aab830c-osd-1 2023-07-14T19:30:54.183 DEBUG:teuthology.orchestra.run.smithi101:> sudo pkill -f 'journalctl -f -n 0 -u ceph-9adea648-227b-11ee-9b33-001a4aab830c@osd.1.service' 2023-07-14T19:30:54.223 DEBUG:teuthology.orchestra.run:got remote process result: None 2023-07-14T19:30:54.223 INFO:tasks.cephadm.osd.1:Stopped osd.1 2023-07-14T19:30:54.224 INFO:tasks.cephadm.osd.2:Stopping osd.2... 2023-07-14T19:30:54.224 DEBUG:teuthology.orchestra.run.smithi139:> sudo systemctl stop ceph-9adea648-227b-11ee-9b33-001a4aab830c@osd.2 2023-07-14T19:30:54.592 INFO:journalctl@ceph.osd.2.smithi139.stdout:Jul 14 19:30:54 smithi139 systemd[1]: Stopping Ceph osd.2 for 9adea648-227b-11ee-9b33-001a4aab830c... 2023-07-14T19:30:54.970 INFO:journalctl@ceph.osd.2.smithi139.stdout:Jul 14 19:30:54 smithi139 ceph-9adea648-227b-11ee-9b33-001a4aab830c-osd-2[135088]: 2023-07-14T19:30:54.590+0000 7fb2e3196700 -1 received signal: Terminated from /dev/init -- /usr/bin/ceph-osd -n osd.2 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2023-07-14T19:30:54.970 INFO:journalctl@ceph.osd.2.smithi139.stdout:Jul 14 19:30:54 smithi139 ceph-9adea648-227b-11ee-9b33-001a4aab830c-osd-2[135088]: 2023-07-14T19:30:54.590+0000 7fb2e3196700 -1 osd.2 88 *** Got signal Terminated *** 2023-07-14T19:30:54.970 INFO:journalctl@ceph.osd.2.smithi139.stdout:Jul 14 19:30:54 smithi139 ceph-9adea648-227b-11ee-9b33-001a4aab830c-osd-2[135088]: 2023-07-14T19:30:54.590+0000 7fb2e3196700 -1 osd.2 88 *** Immediate shutdown (osd_fast_shutdown=true) *** 2023-07-14T19:31:00.361 INFO:journalctl@ceph.osd.2.smithi139.stdout:Jul 14 19:31:00 smithi139 bash[157818]: ceph-9adea648-227b-11ee-9b33-001a4aab830c-osd-2 2023-07-14T19:31:01.825 DEBUG:teuthology.orchestra.run.smithi139:> sudo pkill -f 'journalctl -f -n 0 -u ceph-9adea648-227b-11ee-9b33-001a4aab830c@osd.2.service' 2023-07-14T19:31:01.866 DEBUG:teuthology.orchestra.run:got remote process result: None 2023-07-14T19:31:01.867 INFO:tasks.cephadm.osd.2:Stopped osd.2 2023-07-14T19:31:01.867 INFO:tasks.cephadm.osd.3:Stopping osd.3... 2023-07-14T19:31:01.867 DEBUG:teuthology.orchestra.run.smithi098:> sudo systemctl stop ceph-9adea648-227b-11ee-9b33-001a4aab830c@osd.3 2023-07-14T19:31:02.222 INFO:journalctl@ceph.osd.3.smithi098.stdout:Jul 14 19:31:01 smithi098 systemd[1]: Stopping Ceph osd.3 for 9adea648-227b-11ee-9b33-001a4aab830c... 2023-07-14T19:31:02.531 INFO:journalctl@ceph.osd.3.smithi098.stdout:Jul 14 19:31:02 smithi098 ceph-9adea648-227b-11ee-9b33-001a4aab830c-osd-3[145819]: 2023-07-14T19:31:02.220+0000 7f6b1a270700 -1 received signal: Terminated from /dev/init -- /usr/bin/ceph-osd -n osd.3 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2023-07-14T19:31:02.531 INFO:journalctl@ceph.osd.3.smithi098.stdout:Jul 14 19:31:02 smithi098 ceph-9adea648-227b-11ee-9b33-001a4aab830c-osd-3[145819]: 2023-07-14T19:31:02.220+0000 7f6b1a270700 -1 osd.3 88 *** Got signal Terminated *** 2023-07-14T19:31:02.531 INFO:journalctl@ceph.osd.3.smithi098.stdout:Jul 14 19:31:02 smithi098 ceph-9adea648-227b-11ee-9b33-001a4aab830c-osd-3[145819]: 2023-07-14T19:31:02.220+0000 7f6b1a270700 -1 osd.3 88 *** Immediate shutdown (osd_fast_shutdown=true) *** 2023-07-14T19:31:08.031 INFO:journalctl@ceph.osd.3.smithi098.stdout:Jul 14 19:31:07 smithi098 bash[177116]: ceph-9adea648-227b-11ee-9b33-001a4aab830c-osd-3 2023-07-14T19:31:09.265 DEBUG:teuthology.orchestra.run.smithi098:> sudo pkill -f 'journalctl -f -n 0 -u ceph-9adea648-227b-11ee-9b33-001a4aab830c@osd.3.service' 2023-07-14T19:31:09.306 DEBUG:teuthology.orchestra.run:got remote process result: None 2023-07-14T19:31:09.306 INFO:tasks.cephadm.osd.3:Stopped osd.3 2023-07-14T19:31:09.307 INFO:tasks.cephadm.osd.4:Stopping osd.4... 2023-07-14T19:31:09.307 DEBUG:teuthology.orchestra.run.smithi101:> sudo systemctl stop ceph-9adea648-227b-11ee-9b33-001a4aab830c@osd.4 2023-07-14T19:31:09.670 INFO:journalctl@ceph.osd.4.smithi101.stdout:Jul 14 19:31:09 smithi101 systemd[1]: Stopping Ceph osd.4 for 9adea648-227b-11ee-9b33-001a4aab830c... 2023-07-14T19:31:09.941 INFO:journalctl@ceph.osd.4.smithi101.stdout:Jul 14 19:31:09 smithi101 ceph-9adea648-227b-11ee-9b33-001a4aab830c-osd-4[139397]: 2023-07-14T19:31:09.669+0000 7f3f06e1c700 -1 received signal: Terminated from /dev/init -- /usr/bin/ceph-osd -n osd.4 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2023-07-14T19:31:09.941 INFO:journalctl@ceph.osd.4.smithi101.stdout:Jul 14 19:31:09 smithi101 ceph-9adea648-227b-11ee-9b33-001a4aab830c-osd-4[139397]: 2023-07-14T19:31:09.669+0000 7f3f06e1c700 -1 osd.4 88 *** Got signal Terminated *** 2023-07-14T19:31:09.942 INFO:journalctl@ceph.osd.4.smithi101.stdout:Jul 14 19:31:09 smithi101 ceph-9adea648-227b-11ee-9b33-001a4aab830c-osd-4[139397]: 2023-07-14T19:31:09.669+0000 7f3f06e1c700 -1 osd.4 88 *** Immediate shutdown (osd_fast_shutdown=true) *** 2023-07-14T19:31:15.346 INFO:journalctl@ceph.osd.4.smithi101.stdout:Jul 14 19:31:15 smithi101 bash[156630]: ceph-9adea648-227b-11ee-9b33-001a4aab830c-osd-4 2023-07-14T19:31:16.611 DEBUG:teuthology.orchestra.run.smithi101:> sudo pkill -f 'journalctl -f -n 0 -u ceph-9adea648-227b-11ee-9b33-001a4aab830c@osd.4.service' 2023-07-14T19:31:16.651 DEBUG:teuthology.orchestra.run:got remote process result: None 2023-07-14T19:31:16.652 INFO:tasks.cephadm.osd.4:Stopped osd.4 2023-07-14T19:31:16.653 INFO:tasks.cephadm.osd.5:Stopping osd.5... 2023-07-14T19:31:16.653 DEBUG:teuthology.orchestra.run.smithi139:> sudo systemctl stop ceph-9adea648-227b-11ee-9b33-001a4aab830c@osd.5 2023-07-14T19:31:16.970 INFO:journalctl@ceph.osd.5.smithi139.stdout:Jul 14 19:31:16 smithi139 systemd[1]: Stopping Ceph osd.5 for 9adea648-227b-11ee-9b33-001a4aab830c... 2023-07-14T19:31:17.470 INFO:journalctl@ceph.osd.5.smithi139.stdout:Jul 14 19:31:17 smithi139 ceph-9adea648-227b-11ee-9b33-001a4aab830c-osd-5[140626]: 2023-07-14T19:31:17.028+0000 7f0e530ec700 -1 received signal: Terminated from /dev/init -- /usr/bin/ceph-osd -n osd.5 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2023-07-14T19:31:17.470 INFO:journalctl@ceph.osd.5.smithi139.stdout:Jul 14 19:31:17 smithi139 ceph-9adea648-227b-11ee-9b33-001a4aab830c-osd-5[140626]: 2023-07-14T19:31:17.028+0000 7f0e530ec700 -1 osd.5 88 *** Got signal Terminated *** 2023-07-14T19:31:17.471 INFO:journalctl@ceph.osd.5.smithi139.stdout:Jul 14 19:31:17 smithi139 ceph-9adea648-227b-11ee-9b33-001a4aab830c-osd-5[140626]: 2023-07-14T19:31:17.028+0000 7f0e530ec700 -1 osd.5 88 *** Immediate shutdown (osd_fast_shutdown=true) *** 2023-07-14T19:31:22.796 INFO:journalctl@ceph.osd.5.smithi139.stdout:Jul 14 19:31:22 smithi139 bash[158115]: ceph-9adea648-227b-11ee-9b33-001a4aab830c-osd-5 2023-07-14T19:31:24.161 DEBUG:teuthology.orchestra.run.smithi139:> sudo pkill -f 'journalctl -f -n 0 -u ceph-9adea648-227b-11ee-9b33-001a4aab830c@osd.5.service' 2023-07-14T19:31:24.203 DEBUG:teuthology.orchestra.run:got remote process result: None 2023-07-14T19:31:24.203 INFO:tasks.cephadm.osd.5:Stopped osd.5 2023-07-14T19:31:24.204 INFO:tasks.cephadm.osd.6:Stopping osd.6... 2023-07-14T19:31:24.205 DEBUG:teuthology.orchestra.run.smithi098:> sudo systemctl stop ceph-9adea648-227b-11ee-9b33-001a4aab830c@osd.6 2023-07-14T19:31:24.531 INFO:journalctl@ceph.osd.6.smithi098.stdout:Jul 14 19:31:24 smithi098 systemd[1]: Stopping Ceph osd.6 for 9adea648-227b-11ee-9b33-001a4aab830c... 2023-07-14T19:31:25.031 INFO:journalctl@ceph.osd.6.smithi098.stdout:Jul 14 19:31:24 smithi098 ceph-9adea648-227b-11ee-9b33-001a4aab830c-osd-6[151312]: 2023-07-14T19:31:24.585+0000 7f4645401700 -1 received signal: Terminated from /dev/init -- /usr/bin/ceph-osd -n osd.6 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2023-07-14T19:31:25.031 INFO:journalctl@ceph.osd.6.smithi098.stdout:Jul 14 19:31:24 smithi098 ceph-9adea648-227b-11ee-9b33-001a4aab830c-osd-6[151312]: 2023-07-14T19:31:24.585+0000 7f4645401700 -1 osd.6 88 *** Got signal Terminated *** 2023-07-14T19:31:25.032 INFO:journalctl@ceph.osd.6.smithi098.stdout:Jul 14 19:31:24 smithi098 ceph-9adea648-227b-11ee-9b33-001a4aab830c-osd-6[151312]: 2023-07-14T19:31:24.585+0000 7f4645401700 -1 osd.6 88 *** Immediate shutdown (osd_fast_shutdown=true) *** 2023-07-14T19:31:30.531 INFO:journalctl@ceph.osd.6.smithi098.stdout:Jul 14 19:31:30 smithi098 bash[177387]: ceph-9adea648-227b-11ee-9b33-001a4aab830c-osd-6 2023-07-14T19:31:31.532 DEBUG:teuthology.orchestra.run.smithi098:> sudo pkill -f 'journalctl -f -n 0 -u ceph-9adea648-227b-11ee-9b33-001a4aab830c@osd.6.service' 2023-07-14T19:31:31.584 DEBUG:teuthology.orchestra.run:got remote process result: None 2023-07-14T19:31:31.584 INFO:tasks.cephadm.osd.6:Stopped osd.6 2023-07-14T19:31:31.585 INFO:tasks.cephadm.osd.7:Stopping osd.7... 2023-07-14T19:31:31.585 DEBUG:teuthology.orchestra.run.smithi101:> sudo systemctl stop ceph-9adea648-227b-11ee-9b33-001a4aab830c@osd.7 2023-07-14T19:31:31.941 INFO:journalctl@ceph.osd.7.smithi101.stdout:Jul 14 19:31:31 smithi101 systemd[1]: Stopping Ceph osd.7 for 9adea648-227b-11ee-9b33-001a4aab830c... 2023-07-14T19:31:32.441 INFO:journalctl@ceph.osd.7.smithi101.stdout:Jul 14 19:31:31 smithi101 ceph-9adea648-227b-11ee-9b33-001a4aab830c-osd-7[144864]: 2023-07-14T19:31:31.957+0000 7f42f54a7700 -1 received signal: Terminated from /dev/init -- /usr/bin/ceph-osd -n osd.7 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2023-07-14T19:31:32.442 INFO:journalctl@ceph.osd.7.smithi101.stdout:Jul 14 19:31:31 smithi101 ceph-9adea648-227b-11ee-9b33-001a4aab830c-osd-7[144864]: 2023-07-14T19:31:31.957+0000 7f42f54a7700 -1 osd.7 88 *** Got signal Terminated *** 2023-07-14T19:31:32.442 INFO:journalctl@ceph.osd.7.smithi101.stdout:Jul 14 19:31:31 smithi101 ceph-9adea648-227b-11ee-9b33-001a4aab830c-osd-7[144864]: 2023-07-14T19:31:31.957+0000 7f42f54a7700 -1 osd.7 88 *** Immediate shutdown (osd_fast_shutdown=true) *** 2023-07-14T19:31:37.734 INFO:journalctl@ceph.osd.7.smithi101.stdout:Jul 14 19:31:37 smithi101 bash[156891]: ceph-9adea648-227b-11ee-9b33-001a4aab830c-osd-7 2023-07-14T19:31:39.200 DEBUG:teuthology.orchestra.run.smithi101:> sudo pkill -f 'journalctl -f -n 0 -u ceph-9adea648-227b-11ee-9b33-001a4aab830c@osd.7.service' 2023-07-14T19:31:39.240 DEBUG:teuthology.orchestra.run:got remote process result: None 2023-07-14T19:31:39.241 INFO:tasks.cephadm.osd.7:Stopped osd.7 2023-07-14T19:31:39.241 INFO:tasks.cephadm.osd.8:Stopping osd.8... 2023-07-14T19:31:39.242 DEBUG:teuthology.orchestra.run.smithi139:> sudo systemctl stop ceph-9adea648-227b-11ee-9b33-001a4aab830c@osd.8 2023-07-14T19:31:39.625 INFO:journalctl@ceph.osd.8.smithi139.stdout:Jul 14 19:31:39 smithi139 systemd[1]: Stopping Ceph osd.8 for 9adea648-227b-11ee-9b33-001a4aab830c... 2023-07-14T19:31:39.970 INFO:journalctl@ceph.osd.8.smithi139.stdout:Jul 14 19:31:39 smithi139 ceph-9adea648-227b-11ee-9b33-001a4aab830c-osd-8[146146]: 2023-07-14T19:31:39.623+0000 7f35dfcb5700 -1 received signal: Terminated from /dev/init -- /usr/bin/ceph-osd -n osd.8 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2023-07-14T19:31:39.970 INFO:journalctl@ceph.osd.8.smithi139.stdout:Jul 14 19:31:39 smithi139 ceph-9adea648-227b-11ee-9b33-001a4aab830c-osd-8[146146]: 2023-07-14T19:31:39.623+0000 7f35dfcb5700 -1 osd.8 88 *** Got signal Terminated *** 2023-07-14T19:31:39.970 INFO:journalctl@ceph.osd.8.smithi139.stdout:Jul 14 19:31:39 smithi139 ceph-9adea648-227b-11ee-9b33-001a4aab830c-osd-8[146146]: 2023-07-14T19:31:39.623+0000 7f35dfcb5700 -1 osd.8 88 *** Immediate shutdown (osd_fast_shutdown=true) *** 2023-07-14T19:31:45.470 INFO:journalctl@ceph.osd.8.smithi139.stdout:Jul 14 19:31:45 smithi139 bash[158387]: ceph-9adea648-227b-11ee-9b33-001a4aab830c-osd-8 2023-07-14T19:31:46.440 INFO:journalctl@ceph.osd.10.smithi101.stdout:Jul 14 19:31:46 smithi101 ceph-9adea648-227b-11ee-9b33-001a4aab830c-osd-10[150362]: 2023-07-14T19:31:46.077+0000 7f2e7769f700 -1 osd.10 88 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2023-07-14T19:30:40.876664+0000 front 2023-07-14T19:30:40.876575+0000 (oldest deadline 2023-07-14T19:31:45.576582+0000) 2023-07-14T19:31:46.708 DEBUG:teuthology.orchestra.run.smithi139:> sudo pkill -f 'journalctl -f -n 0 -u ceph-9adea648-227b-11ee-9b33-001a4aab830c@osd.8.service' 2023-07-14T19:31:46.748 DEBUG:teuthology.orchestra.run:got remote process result: None 2023-07-14T19:31:46.748 INFO:tasks.cephadm.osd.8:Stopped osd.8 2023-07-14T19:31:46.749 INFO:tasks.cephadm.osd.9:Stopping osd.9... 2023-07-14T19:31:46.749 DEBUG:teuthology.orchestra.run.smithi098:> sudo systemctl stop ceph-9adea648-227b-11ee-9b33-001a4aab830c@osd.9 2023-07-14T19:31:47.030 INFO:journalctl@ceph.osd.9.smithi098.stdout:Jul 14 19:31:46 smithi098 systemd[1]: Stopping Ceph osd.9 for 9adea648-227b-11ee-9b33-001a4aab830c... 2023-07-14T19:31:47.440 INFO:journalctl@ceph.osd.10.smithi101.stdout:Jul 14 19:31:47 smithi101 ceph-9adea648-227b-11ee-9b33-001a4aab830c-osd-10[150362]: 2023-07-14T19:31:47.126+0000 7f2e7769f700 -1 osd.10 88 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2023-07-14T19:30:40.876664+0000 front 2023-07-14T19:30:40.876575+0000 (oldest deadline 2023-07-14T19:31:45.576582+0000) 2023-07-14T19:31:47.531 INFO:journalctl@ceph.osd.9.smithi098.stdout:Jul 14 19:31:47 smithi098 ceph-9adea648-227b-11ee-9b33-001a4aab830c-osd-9[156860]: 2023-07-14T19:31:47.090+0000 7fd87b3a5700 -1 received signal: Terminated from /dev/init -- /usr/bin/ceph-osd -n osd.9 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2023-07-14T19:31:47.531 INFO:journalctl@ceph.osd.9.smithi098.stdout:Jul 14 19:31:47 smithi098 ceph-9adea648-227b-11ee-9b33-001a4aab830c-osd-9[156860]: 2023-07-14T19:31:47.090+0000 7fd87b3a5700 -1 osd.9 88 *** Got signal Terminated *** 2023-07-14T19:31:47.531 INFO:journalctl@ceph.osd.9.smithi098.stdout:Jul 14 19:31:47 smithi098 ceph-9adea648-227b-11ee-9b33-001a4aab830c-osd-9[156860]: 2023-07-14T19:31:47.090+0000 7fd87b3a5700 -1 osd.9 88 *** Immediate shutdown (osd_fast_shutdown=true) *** 2023-07-14T19:31:48.220 INFO:journalctl@ceph.osd.11.smithi139.stdout:Jul 14 19:31:47 smithi139 ceph-9adea648-227b-11ee-9b33-001a4aab830c-osd-11[151655]: 2023-07-14T19:31:47.732+0000 7f50a8f97700 -1 osd.11 88 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2023-07-14T19:30:44.591688+0000 front 2023-07-14T19:30:44.591727+0000 (oldest deadline 2023-07-14T19:31:47.491900+0000) 2023-07-14T19:31:48.440 INFO:journalctl@ceph.osd.10.smithi101.stdout:Jul 14 19:31:48 smithi101 ceph-9adea648-227b-11ee-9b33-001a4aab830c-osd-10[150362]: 2023-07-14T19:31:48.159+0000 7f2e7769f700 -1 osd.10 88 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2023-07-14T19:30:40.876664+0000 front 2023-07-14T19:30:40.876575+0000 (oldest deadline 2023-07-14T19:31:45.576582+0000) 2023-07-14T19:31:49.220 INFO:journalctl@ceph.osd.11.smithi139.stdout:Jul 14 19:31:48 smithi139 ceph-9adea648-227b-11ee-9b33-001a4aab830c-osd-11[151655]: 2023-07-14T19:31:48.769+0000 7f50a8f97700 -1 osd.11 88 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2023-07-14T19:30:44.591688+0000 front 2023-07-14T19:30:44.591727+0000 (oldest deadline 2023-07-14T19:31:47.491900+0000) 2023-07-14T19:31:49.281 INFO:journalctl@ceph.osd.9.smithi098.stdout:Jul 14 19:31:48 smithi098 ceph-9adea648-227b-11ee-9b33-001a4aab830c-osd-9[156860]: 2023-07-14T19:31:48.934+0000 7fd875998700 -1 osd.9 88 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2023-07-14T19:30:44.079955+0000 front 2023-07-14T19:30:44.080151+0000 (oldest deadline 2023-07-14T19:31:48.779953+0000) 2023-07-14T19:31:49.440 INFO:journalctl@ceph.osd.10.smithi101.stdout:Jul 14 19:31:49 smithi101 ceph-9adea648-227b-11ee-9b33-001a4aab830c-osd-10[150362]: 2023-07-14T19:31:49.182+0000 7f2e7769f700 -1 osd.10 88 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2023-07-14T19:30:40.876664+0000 front 2023-07-14T19:30:40.876575+0000 (oldest deadline 2023-07-14T19:31:45.576582+0000) 2023-07-14T19:31:50.220 INFO:journalctl@ceph.osd.11.smithi139.stdout:Jul 14 19:31:49 smithi139 ceph-9adea648-227b-11ee-9b33-001a4aab830c-osd-11[151655]: 2023-07-14T19:31:49.791+0000 7f50a8f97700 -1 osd.11 88 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2023-07-14T19:30:44.591688+0000 front 2023-07-14T19:30:44.591727+0000 (oldest deadline 2023-07-14T19:31:47.491900+0000) 2023-07-14T19:31:50.281 INFO:journalctl@ceph.osd.9.smithi098.stdout:Jul 14 19:31:49 smithi098 ceph-9adea648-227b-11ee-9b33-001a4aab830c-osd-9[156860]: 2023-07-14T19:31:49.954+0000 7fd875998700 -1 osd.9 88 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2023-07-14T19:30:44.079955+0000 front 2023-07-14T19:30:44.080151+0000 (oldest deadline 2023-07-14T19:31:48.779953+0000) 2023-07-14T19:31:50.690 INFO:journalctl@ceph.osd.10.smithi101.stdout:Jul 14 19:31:50 smithi101 ceph-9adea648-227b-11ee-9b33-001a4aab830c-osd-10[150362]: 2023-07-14T19:31:50.211+0000 7f2e7769f700 -1 osd.10 88 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2023-07-14T19:30:40.876664+0000 front 2023-07-14T19:30:40.876575+0000 (oldest deadline 2023-07-14T19:31:45.576582+0000) 2023-07-14T19:31:51.220 INFO:journalctl@ceph.osd.11.smithi139.stdout:Jul 14 19:31:50 smithi139 ceph-9adea648-227b-11ee-9b33-001a4aab830c-osd-11[151655]: 2023-07-14T19:31:50.815+0000 7f50a8f97700 -1 osd.11 88 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2023-07-14T19:30:44.591688+0000 front 2023-07-14T19:30:44.591727+0000 (oldest deadline 2023-07-14T19:31:47.491900+0000) 2023-07-14T19:31:51.281 INFO:journalctl@ceph.osd.9.smithi098.stdout:Jul 14 19:31:50 smithi098 ceph-9adea648-227b-11ee-9b33-001a4aab830c-osd-9[156860]: 2023-07-14T19:31:50.931+0000 7fd875998700 -1 osd.9 88 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2023-07-14T19:30:44.079955+0000 front 2023-07-14T19:30:44.080151+0000 (oldest deadline 2023-07-14T19:31:48.779953+0000) 2023-07-14T19:31:51.690 INFO:journalctl@ceph.osd.10.smithi101.stdout:Jul 14 19:31:51 smithi101 ceph-9adea648-227b-11ee-9b33-001a4aab830c-osd-10[150362]: 2023-07-14T19:31:51.240+0000 7f2e7769f700 -1 osd.10 88 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2023-07-14T19:30:40.876664+0000 front 2023-07-14T19:30:40.876575+0000 (oldest deadline 2023-07-14T19:31:45.576582+0000) 2023-07-14T19:31:52.220 INFO:journalctl@ceph.osd.11.smithi139.stdout:Jul 14 19:31:51 smithi139 ceph-9adea648-227b-11ee-9b33-001a4aab830c-osd-11[151655]: 2023-07-14T19:31:51.828+0000 7f50a8f97700 -1 osd.11 88 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2023-07-14T19:30:44.591688+0000 front 2023-07-14T19:30:44.591727+0000 (oldest deadline 2023-07-14T19:31:47.491900+0000) 2023-07-14T19:31:52.281 INFO:journalctl@ceph.osd.9.smithi098.stdout:Jul 14 19:31:51 smithi098 ceph-9adea648-227b-11ee-9b33-001a4aab830c-osd-9[156860]: 2023-07-14T19:31:51.914+0000 7fd875998700 -1 osd.9 88 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2023-07-14T19:30:44.079955+0000 front 2023-07-14T19:30:44.080151+0000 (oldest deadline 2023-07-14T19:31:48.779953+0000) 2023-07-14T19:31:52.690 INFO:journalctl@ceph.osd.10.smithi101.stdout:Jul 14 19:31:52 smithi101 ceph-9adea648-227b-11ee-9b33-001a4aab830c-osd-10[150362]: 2023-07-14T19:31:52.233+0000 7f2e7769f700 -1 osd.10 88 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2023-07-14T19:30:40.876664+0000 front 2023-07-14T19:30:40.876575+0000 (oldest deadline 2023-07-14T19:31:45.576582+0000) 2023-07-14T19:31:52.781 INFO:journalctl@ceph.osd.9.smithi098.stdout:Jul 14 19:31:52 smithi098 bash[177655]: ceph-9adea648-227b-11ee-9b33-001a4aab830c-osd-9 2023-07-14T19:31:53.220 INFO:journalctl@ceph.osd.11.smithi139.stdout:Jul 14 19:31:52 smithi139 ceph-9adea648-227b-11ee-9b33-001a4aab830c-osd-11[151655]: 2023-07-14T19:31:52.818+0000 7f50a8f97700 -1 osd.11 88 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2023-07-14T19:30:44.591688+0000 front 2023-07-14T19:30:44.591727+0000 (oldest deadline 2023-07-14T19:31:47.491900+0000) 2023-07-14T19:31:53.690 INFO:journalctl@ceph.osd.10.smithi101.stdout:Jul 14 19:31:53 smithi101 ceph-9adea648-227b-11ee-9b33-001a4aab830c-osd-10[150362]: 2023-07-14T19:31:53.274+0000 7f2e7769f700 -1 osd.10 88 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2023-07-14T19:30:40.876664+0000 front 2023-07-14T19:30:40.876575+0000 (oldest deadline 2023-07-14T19:31:45.576582+0000) 2023-07-14T19:31:53.836 DEBUG:teuthology.orchestra.run.smithi098:> sudo pkill -f 'journalctl -f -n 0 -u ceph-9adea648-227b-11ee-9b33-001a4aab830c@osd.9.service' 2023-07-14T19:31:53.876 DEBUG:teuthology.orchestra.run:got remote process result: None 2023-07-14T19:31:53.877 INFO:tasks.cephadm.osd.9:Stopped osd.9 2023-07-14T19:31:53.877 INFO:tasks.cephadm.osd.10:Stopping osd.10... 2023-07-14T19:31:53.877 DEBUG:teuthology.orchestra.run.smithi101:> sudo systemctl stop ceph-9adea648-227b-11ee-9b33-001a4aab830c@osd.10 2023-07-14T19:31:54.190 INFO:journalctl@ceph.osd.10.smithi101.stdout:Jul 14 19:31:53 smithi101 systemd[1]: Stopping Ceph osd.10 for 9adea648-227b-11ee-9b33-001a4aab830c... 2023-07-14T19:31:54.220 INFO:journalctl@ceph.osd.11.smithi139.stdout:Jul 14 19:31:53 smithi139 ceph-9adea648-227b-11ee-9b33-001a4aab830c-osd-11[151655]: 2023-07-14T19:31:53.795+0000 7f50a8f97700 -1 osd.11 88 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2023-07-14T19:30:44.591688+0000 front 2023-07-14T19:30:44.591727+0000 (oldest deadline 2023-07-14T19:31:47.491900+0000) 2023-07-14T19:31:54.220 INFO:journalctl@ceph.osd.11.smithi139.stdout:Jul 14 19:31:53 smithi139 ceph-9adea648-227b-11ee-9b33-001a4aab830c-osd-11[151655]: 2023-07-14T19:31:53.795+0000 7f50a8f97700 -1 osd.11 88 heartbeat_check: no reply from 172.21.15.101:6804 osd.1 since back 2023-07-14T19:30:47.492939+0000 front 2023-07-14T19:30:47.493059+0000 (oldest deadline 2023-07-14T19:31:53.392823+0000) 2023-07-14T19:31:54.690 INFO:journalctl@ceph.osd.10.smithi101.stdout:Jul 14 19:31:54 smithi101 ceph-9adea648-227b-11ee-9b33-001a4aab830c-osd-10[150362]: 2023-07-14T19:31:54.238+0000 7f2e7d0ac700 -1 received signal: Terminated from /dev/init -- /usr/bin/ceph-osd -n osd.10 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2023-07-14T19:31:54.691 INFO:journalctl@ceph.osd.10.smithi101.stdout:Jul 14 19:31:54 smithi101 ceph-9adea648-227b-11ee-9b33-001a4aab830c-osd-10[150362]: 2023-07-14T19:31:54.238+0000 7f2e7d0ac700 -1 osd.10 88 *** Got signal Terminated *** 2023-07-14T19:31:54.691 INFO:journalctl@ceph.osd.10.smithi101.stdout:Jul 14 19:31:54 smithi101 ceph-9adea648-227b-11ee-9b33-001a4aab830c-osd-10[150362]: 2023-07-14T19:31:54.238+0000 7f2e7d0ac700 -1 osd.10 88 *** Immediate shutdown (osd_fast_shutdown=true) *** 2023-07-14T19:31:54.691 INFO:journalctl@ceph.osd.10.smithi101.stdout:Jul 14 19:31:54 smithi101 ceph-9adea648-227b-11ee-9b33-001a4aab830c-osd-10[150362]: 2023-07-14T19:31:54.258+0000 7f2e7769f700 -1 osd.10 88 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2023-07-14T19:30:40.876664+0000 front 2023-07-14T19:30:40.876575+0000 (oldest deadline 2023-07-14T19:31:45.576582+0000) 2023-07-14T19:31:55.220 INFO:journalctl@ceph.osd.11.smithi139.stdout:Jul 14 19:31:54 smithi139 ceph-9adea648-227b-11ee-9b33-001a4aab830c-osd-11[151655]: 2023-07-14T19:31:54.754+0000 7f50a8f97700 -1 osd.11 88 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2023-07-14T19:30:44.591688+0000 front 2023-07-14T19:30:44.591727+0000 (oldest deadline 2023-07-14T19:31:47.491900+0000) 2023-07-14T19:31:55.220 INFO:journalctl@ceph.osd.11.smithi139.stdout:Jul 14 19:31:54 smithi139 ceph-9adea648-227b-11ee-9b33-001a4aab830c-osd-11[151655]: 2023-07-14T19:31:54.754+0000 7f50a8f97700 -1 osd.11 88 heartbeat_check: no reply from 172.21.15.101:6804 osd.1 since back 2023-07-14T19:30:47.492939+0000 front 2023-07-14T19:30:47.493059+0000 (oldest deadline 2023-07-14T19:31:53.392823+0000) 2023-07-14T19:31:55.690 INFO:journalctl@ceph.osd.10.smithi101.stdout:Jul 14 19:31:55 smithi101 ceph-9adea648-227b-11ee-9b33-001a4aab830c-osd-10[150362]: 2023-07-14T19:31:55.229+0000 7f2e7769f700 -1 osd.10 88 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2023-07-14T19:30:40.876664+0000 front 2023-07-14T19:30:40.876575+0000 (oldest deadline 2023-07-14T19:31:45.576582+0000) 2023-07-14T19:31:56.220 INFO:journalctl@ceph.osd.11.smithi139.stdout:Jul 14 19:31:55 smithi139 ceph-9adea648-227b-11ee-9b33-001a4aab830c-osd-11[151655]: 2023-07-14T19:31:55.729+0000 7f50a8f97700 -1 osd.11 88 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2023-07-14T19:30:44.591688+0000 front 2023-07-14T19:30:44.591727+0000 (oldest deadline 2023-07-14T19:31:47.491900+0000) 2023-07-14T19:31:56.220 INFO:journalctl@ceph.osd.11.smithi139.stdout:Jul 14 19:31:55 smithi139 ceph-9adea648-227b-11ee-9b33-001a4aab830c-osd-11[151655]: 2023-07-14T19:31:55.729+0000 7f50a8f97700 -1 osd.11 88 heartbeat_check: no reply from 172.21.15.101:6804 osd.1 since back 2023-07-14T19:30:47.492939+0000 front 2023-07-14T19:30:47.493059+0000 (oldest deadline 2023-07-14T19:31:53.392823+0000) 2023-07-14T19:31:56.440 INFO:journalctl@ceph.osd.10.smithi101.stdout:Jul 14 19:31:56 smithi101 ceph-9adea648-227b-11ee-9b33-001a4aab830c-osd-10[150362]: 2023-07-14T19:31:56.187+0000 7f2e7769f700 -1 osd.10 88 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2023-07-14T19:30:40.876664+0000 front 2023-07-14T19:30:40.876575+0000 (oldest deadline 2023-07-14T19:31:45.576582+0000) 2023-07-14T19:31:57.220 INFO:journalctl@ceph.osd.11.smithi139.stdout:Jul 14 19:31:56 smithi139 ceph-9adea648-227b-11ee-9b33-001a4aab830c-osd-11[151655]: 2023-07-14T19:31:56.754+0000 7f50a8f97700 -1 osd.11 88 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2023-07-14T19:30:44.591688+0000 front 2023-07-14T19:30:44.591727+0000 (oldest deadline 2023-07-14T19:31:47.491900+0000) 2023-07-14T19:31:57.220 INFO:journalctl@ceph.osd.11.smithi139.stdout:Jul 14 19:31:56 smithi139 ceph-9adea648-227b-11ee-9b33-001a4aab830c-osd-11[151655]: 2023-07-14T19:31:56.754+0000 7f50a8f97700 -1 osd.11 88 heartbeat_check: no reply from 172.21.15.101:6804 osd.1 since back 2023-07-14T19:30:47.492939+0000 front 2023-07-14T19:30:47.493059+0000 (oldest deadline 2023-07-14T19:31:53.392823+0000) 2023-07-14T19:31:57.690 INFO:journalctl@ceph.osd.10.smithi101.stdout:Jul 14 19:31:57 smithi101 ceph-9adea648-227b-11ee-9b33-001a4aab830c-osd-10[150362]: 2023-07-14T19:31:57.220+0000 7f2e7769f700 -1 osd.10 88 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2023-07-14T19:30:40.876664+0000 front 2023-07-14T19:30:40.876575+0000 (oldest deadline 2023-07-14T19:31:45.576582+0000) 2023-07-14T19:31:57.690 INFO:journalctl@ceph.osd.10.smithi101.stdout:Jul 14 19:31:57 smithi101 ceph-9adea648-227b-11ee-9b33-001a4aab830c-osd-10[150362]: 2023-07-14T19:31:57.220+0000 7f2e7769f700 -1 osd.10 88 heartbeat_check: no reply from 172.21.15.101:6804 osd.1 since back 2023-07-14T19:30:50.678588+0000 front 2023-07-14T19:30:50.678678+0000 (oldest deadline 2023-07-14T19:31:56.578799+0000) 2023-07-14T19:31:58.220 INFO:journalctl@ceph.osd.11.smithi139.stdout:Jul 14 19:31:57 smithi139 ceph-9adea648-227b-11ee-9b33-001a4aab830c-osd-11[151655]: 2023-07-14T19:31:57.762+0000 7f50a8f97700 -1 osd.11 88 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2023-07-14T19:30:44.591688+0000 front 2023-07-14T19:30:44.591727+0000 (oldest deadline 2023-07-14T19:31:47.491900+0000) 2023-07-14T19:31:58.220 INFO:journalctl@ceph.osd.11.smithi139.stdout:Jul 14 19:31:57 smithi139 ceph-9adea648-227b-11ee-9b33-001a4aab830c-osd-11[151655]: 2023-07-14T19:31:57.762+0000 7f50a8f97700 -1 osd.11 88 heartbeat_check: no reply from 172.21.15.101:6804 osd.1 since back 2023-07-14T19:30:47.492939+0000 front 2023-07-14T19:30:47.493059+0000 (oldest deadline 2023-07-14T19:31:53.392823+0000) 2023-07-14T19:31:58.440 INFO:journalctl@ceph.osd.10.smithi101.stdout:Jul 14 19:31:58 smithi101 ceph-9adea648-227b-11ee-9b33-001a4aab830c-osd-10[150362]: 2023-07-14T19:31:58.186+0000 7f2e7769f700 -1 osd.10 88 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2023-07-14T19:30:40.876664+0000 front 2023-07-14T19:30:40.876575+0000 (oldest deadline 2023-07-14T19:31:45.576582+0000) 2023-07-14T19:31:58.441 INFO:journalctl@ceph.osd.10.smithi101.stdout:Jul 14 19:31:58 smithi101 ceph-9adea648-227b-11ee-9b33-001a4aab830c-osd-10[150362]: 2023-07-14T19:31:58.186+0000 7f2e7769f700 -1 osd.10 88 heartbeat_check: no reply from 172.21.15.101:6804 osd.1 since back 2023-07-14T19:30:50.678588+0000 front 2023-07-14T19:30:50.678678+0000 (oldest deadline 2023-07-14T19:31:56.578799+0000) 2023-07-14T19:31:59.220 INFO:journalctl@ceph.osd.11.smithi139.stdout:Jul 14 19:31:58 smithi139 ceph-9adea648-227b-11ee-9b33-001a4aab830c-osd-11[151655]: 2023-07-14T19:31:58.799+0000 7f50a8f97700 -1 osd.11 88 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2023-07-14T19:30:44.591688+0000 front 2023-07-14T19:30:44.591727+0000 (oldest deadline 2023-07-14T19:31:47.491900+0000) 2023-07-14T19:31:59.220 INFO:journalctl@ceph.osd.11.smithi139.stdout:Jul 14 19:31:58 smithi139 ceph-9adea648-227b-11ee-9b33-001a4aab830c-osd-11[151655]: 2023-07-14T19:31:58.799+0000 7f50a8f97700 -1 osd.11 88 heartbeat_check: no reply from 172.21.15.101:6804 osd.1 since back 2023-07-14T19:30:47.492939+0000 front 2023-07-14T19:30:47.493059+0000 (oldest deadline 2023-07-14T19:31:53.392823+0000) 2023-07-14T19:31:59.634 INFO:journalctl@ceph.osd.10.smithi101.stdout:Jul 14 19:31:59 smithi101 ceph-9adea648-227b-11ee-9b33-001a4aab830c-osd-10[150362]: 2023-07-14T19:31:59.189+0000 7f2e7769f700 -1 osd.10 88 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2023-07-14T19:30:40.876664+0000 front 2023-07-14T19:30:40.876575+0000 (oldest deadline 2023-07-14T19:31:45.576582+0000) 2023-07-14T19:31:59.634 INFO:journalctl@ceph.osd.10.smithi101.stdout:Jul 14 19:31:59 smithi101 ceph-9adea648-227b-11ee-9b33-001a4aab830c-osd-10[150362]: 2023-07-14T19:31:59.189+0000 7f2e7769f700 -1 osd.10 88 heartbeat_check: no reply from 172.21.15.101:6804 osd.1 since back 2023-07-14T19:30:50.678588+0000 front 2023-07-14T19:30:50.678678+0000 (oldest deadline 2023-07-14T19:31:56.578799+0000) 2023-07-14T19:31:59.914 INFO:journalctl@ceph.osd.10.smithi101.stdout:Jul 14 19:31:59 smithi101 bash[157150]: ceph-9adea648-227b-11ee-9b33-001a4aab830c-osd-10 2023-07-14T19:32:00.220 INFO:journalctl@ceph.osd.11.smithi139.stdout:Jul 14 19:31:59 smithi139 ceph-9adea648-227b-11ee-9b33-001a4aab830c-osd-11[151655]: 2023-07-14T19:31:59.838+0000 7f50a8f97700 -1 osd.11 88 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2023-07-14T19:30:44.591688+0000 front 2023-07-14T19:30:44.591727+0000 (oldest deadline 2023-07-14T19:31:47.491900+0000) 2023-07-14T19:32:00.220 INFO:journalctl@ceph.osd.11.smithi139.stdout:Jul 14 19:31:59 smithi139 ceph-9adea648-227b-11ee-9b33-001a4aab830c-osd-11[151655]: 2023-07-14T19:31:59.838+0000 7f50a8f97700 -1 osd.11 88 heartbeat_check: no reply from 172.21.15.101:6804 osd.1 since back 2023-07-14T19:30:47.492939+0000 front 2023-07-14T19:30:47.493059+0000 (oldest deadline 2023-07-14T19:31:53.392823+0000) 2023-07-14T19:32:01.172 DEBUG:teuthology.orchestra.run.smithi101:> sudo pkill -f 'journalctl -f -n 0 -u ceph-9adea648-227b-11ee-9b33-001a4aab830c@osd.10.service' 2023-07-14T19:32:01.213 DEBUG:teuthology.orchestra.run:got remote process result: None 2023-07-14T19:32:01.213 INFO:tasks.cephadm.osd.10:Stopped osd.10 2023-07-14T19:32:01.214 INFO:tasks.cephadm.osd.11:Stopping osd.11... 2023-07-14T19:32:01.214 DEBUG:teuthology.orchestra.run.smithi139:> sudo systemctl stop ceph-9adea648-227b-11ee-9b33-001a4aab830c@osd.11 2023-07-14T19:32:01.219 INFO:journalctl@ceph.osd.11.smithi139.stdout:Jul 14 19:32:00 smithi139 ceph-9adea648-227b-11ee-9b33-001a4aab830c-osd-11[151655]: 2023-07-14T19:32:00.872+0000 7f50a8f97700 -1 osd.11 88 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2023-07-14T19:30:44.591688+0000 front 2023-07-14T19:30:44.591727+0000 (oldest deadline 2023-07-14T19:31:47.491900+0000) 2023-07-14T19:32:01.220 INFO:journalctl@ceph.osd.11.smithi139.stdout:Jul 14 19:32:00 smithi139 ceph-9adea648-227b-11ee-9b33-001a4aab830c-osd-11[151655]: 2023-07-14T19:32:00.872+0000 7f50a8f97700 -1 osd.11 88 heartbeat_check: no reply from 172.21.15.101:6804 osd.1 since back 2023-07-14T19:30:47.492939+0000 front 2023-07-14T19:30:47.493059+0000 (oldest deadline 2023-07-14T19:31:53.392823+0000) 2023-07-14T19:32:01.610 INFO:journalctl@ceph.osd.11.smithi139.stdout:Jul 14 19:32:01 smithi139 systemd[1]: Stopping Ceph osd.11 for 9adea648-227b-11ee-9b33-001a4aab830c... 2023-07-14T19:32:01.870 INFO:journalctl@ceph.osd.11.smithi139.stdout:Jul 14 19:32:01 smithi139 ceph-9adea648-227b-11ee-9b33-001a4aab830c-osd-11[151655]: 2023-07-14T19:32:01.610+0000 7f50ae9a4700 -1 received signal: Terminated from /dev/init -- /usr/bin/ceph-osd -n osd.11 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2023-07-14T19:32:01.871 INFO:journalctl@ceph.osd.11.smithi139.stdout:Jul 14 19:32:01 smithi139 ceph-9adea648-227b-11ee-9b33-001a4aab830c-osd-11[151655]: 2023-07-14T19:32:01.610+0000 7f50ae9a4700 -1 osd.11 88 *** Got signal Terminated *** 2023-07-14T19:32:01.871 INFO:journalctl@ceph.osd.11.smithi139.stdout:Jul 14 19:32:01 smithi139 ceph-9adea648-227b-11ee-9b33-001a4aab830c-osd-11[151655]: 2023-07-14T19:32:01.610+0000 7f50ae9a4700 -1 osd.11 88 *** Immediate shutdown (osd_fast_shutdown=true) *** 2023-07-14T19:32:02.220 INFO:journalctl@ceph.osd.11.smithi139.stdout:Jul 14 19:32:01 smithi139 ceph-9adea648-227b-11ee-9b33-001a4aab830c-osd-11[151655]: 2023-07-14T19:32:01.871+0000 7f50a8f97700 -1 osd.11 88 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2023-07-14T19:30:44.591688+0000 front 2023-07-14T19:30:44.591727+0000 (oldest deadline 2023-07-14T19:31:47.491900+0000) 2023-07-14T19:32:02.220 INFO:journalctl@ceph.osd.11.smithi139.stdout:Jul 14 19:32:01 smithi139 ceph-9adea648-227b-11ee-9b33-001a4aab830c-osd-11[151655]: 2023-07-14T19:32:01.871+0000 7f50a8f97700 -1 osd.11 88 heartbeat_check: no reply from 172.21.15.101:6804 osd.1 since back 2023-07-14T19:30:47.492939+0000 front 2023-07-14T19:30:47.493059+0000 (oldest deadline 2023-07-14T19:31:53.392823+0000) 2023-07-14T19:32:02.220 INFO:journalctl@ceph.osd.11.smithi139.stdout:Jul 14 19:32:01 smithi139 ceph-9adea648-227b-11ee-9b33-001a4aab830c-osd-11[151655]: 2023-07-14T19:32:01.871+0000 7f50a8f97700 -1 osd.11 88 heartbeat_check: no reply from 172.21.15.139:6804 osd.2 since back 2023-07-14T19:30:57.493861+0000 front 2023-07-14T19:30:57.493837+0000 (oldest deadline 2023-07-14T19:32:01.594037+0000) 2023-07-14T19:32:03.220 INFO:journalctl@ceph.osd.11.smithi139.stdout:Jul 14 19:32:02 smithi139 ceph-9adea648-227b-11ee-9b33-001a4aab830c-osd-11[151655]: 2023-07-14T19:32:02.866+0000 7f50a8f97700 -1 osd.11 88 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2023-07-14T19:30:44.591688+0000 front 2023-07-14T19:30:44.591727+0000 (oldest deadline 2023-07-14T19:31:47.491900+0000) 2023-07-14T19:32:03.220 INFO:journalctl@ceph.osd.11.smithi139.stdout:Jul 14 19:32:02 smithi139 ceph-9adea648-227b-11ee-9b33-001a4aab830c-osd-11[151655]: 2023-07-14T19:32:02.866+0000 7f50a8f97700 -1 osd.11 88 heartbeat_check: no reply from 172.21.15.101:6804 osd.1 since back 2023-07-14T19:30:47.492939+0000 front 2023-07-14T19:30:47.493059+0000 (oldest deadline 2023-07-14T19:31:53.392823+0000) 2023-07-14T19:32:03.220 INFO:journalctl@ceph.osd.11.smithi139.stdout:Jul 14 19:32:02 smithi139 ceph-9adea648-227b-11ee-9b33-001a4aab830c-osd-11[151655]: 2023-07-14T19:32:02.866+0000 7f50a8f97700 -1 osd.11 88 heartbeat_check: no reply from 172.21.15.139:6804 osd.2 since back 2023-07-14T19:30:57.493861+0000 front 2023-07-14T19:30:57.493837+0000 (oldest deadline 2023-07-14T19:32:01.594037+0000) 2023-07-14T19:32:04.220 INFO:journalctl@ceph.osd.11.smithi139.stdout:Jul 14 19:32:03 smithi139 ceph-9adea648-227b-11ee-9b33-001a4aab830c-osd-11[151655]: 2023-07-14T19:32:03.828+0000 7f50a8f97700 -1 osd.11 88 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2023-07-14T19:30:44.591688+0000 front 2023-07-14T19:30:44.591727+0000 (oldest deadline 2023-07-14T19:31:47.491900+0000) 2023-07-14T19:32:04.220 INFO:journalctl@ceph.osd.11.smithi139.stdout:Jul 14 19:32:03 smithi139 ceph-9adea648-227b-11ee-9b33-001a4aab830c-osd-11[151655]: 2023-07-14T19:32:03.828+0000 7f50a8f97700 -1 osd.11 88 heartbeat_check: no reply from 172.21.15.101:6804 osd.1 since back 2023-07-14T19:30:47.492939+0000 front 2023-07-14T19:30:47.493059+0000 (oldest deadline 2023-07-14T19:31:53.392823+0000) 2023-07-14T19:32:04.220 INFO:journalctl@ceph.osd.11.smithi139.stdout:Jul 14 19:32:03 smithi139 ceph-9adea648-227b-11ee-9b33-001a4aab830c-osd-11[151655]: 2023-07-14T19:32:03.828+0000 7f50a8f97700 -1 osd.11 88 heartbeat_check: no reply from 172.21.15.139:6804 osd.2 since back 2023-07-14T19:30:57.493861+0000 front 2023-07-14T19:30:57.493837+0000 (oldest deadline 2023-07-14T19:32:01.594037+0000) 2023-07-14T19:32:05.220 INFO:journalctl@ceph.osd.11.smithi139.stdout:Jul 14 19:32:04 smithi139 ceph-9adea648-227b-11ee-9b33-001a4aab830c-osd-11[151655]: 2023-07-14T19:32:04.847+0000 7f50a8f97700 -1 osd.11 88 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2023-07-14T19:30:44.591688+0000 front 2023-07-14T19:30:44.591727+0000 (oldest deadline 2023-07-14T19:31:47.491900+0000) 2023-07-14T19:32:05.220 INFO:journalctl@ceph.osd.11.smithi139.stdout:Jul 14 19:32:04 smithi139 ceph-9adea648-227b-11ee-9b33-001a4aab830c-osd-11[151655]: 2023-07-14T19:32:04.847+0000 7f50a8f97700 -1 osd.11 88 heartbeat_check: no reply from 172.21.15.101:6804 osd.1 since back 2023-07-14T19:30:47.492939+0000 front 2023-07-14T19:30:47.493059+0000 (oldest deadline 2023-07-14T19:31:53.392823+0000) 2023-07-14T19:32:05.220 INFO:journalctl@ceph.osd.11.smithi139.stdout:Jul 14 19:32:04 smithi139 ceph-9adea648-227b-11ee-9b33-001a4aab830c-osd-11[151655]: 2023-07-14T19:32:04.847+0000 7f50a8f97700 -1 osd.11 88 heartbeat_check: no reply from 172.21.15.139:6804 osd.2 since back 2023-07-14T19:30:57.493861+0000 front 2023-07-14T19:30:57.493837+0000 (oldest deadline 2023-07-14T19:32:01.594037+0000) 2023-07-14T19:32:06.220 INFO:journalctl@ceph.osd.11.smithi139.stdout:Jul 14 19:32:05 smithi139 ceph-9adea648-227b-11ee-9b33-001a4aab830c-osd-11[151655]: 2023-07-14T19:32:05.815+0000 7f50a8f97700 -1 osd.11 88 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2023-07-14T19:30:44.591688+0000 front 2023-07-14T19:30:44.591727+0000 (oldest deadline 2023-07-14T19:31:47.491900+0000) 2023-07-14T19:32:06.220 INFO:journalctl@ceph.osd.11.smithi139.stdout:Jul 14 19:32:05 smithi139 ceph-9adea648-227b-11ee-9b33-001a4aab830c-osd-11[151655]: 2023-07-14T19:32:05.815+0000 7f50a8f97700 -1 osd.11 88 heartbeat_check: no reply from 172.21.15.101:6804 osd.1 since back 2023-07-14T19:30:47.492939+0000 front 2023-07-14T19:30:47.493059+0000 (oldest deadline 2023-07-14T19:31:53.392823+0000) 2023-07-14T19:32:06.220 INFO:journalctl@ceph.osd.11.smithi139.stdout:Jul 14 19:32:05 smithi139 ceph-9adea648-227b-11ee-9b33-001a4aab830c-osd-11[151655]: 2023-07-14T19:32:05.815+0000 7f50a8f97700 -1 osd.11 88 heartbeat_check: no reply from 172.21.15.139:6804 osd.2 since back 2023-07-14T19:30:57.493861+0000 front 2023-07-14T19:30:57.493837+0000 (oldest deadline 2023-07-14T19:32:01.594037+0000) 2023-07-14T19:32:07.360 INFO:journalctl@ceph.osd.11.smithi139.stdout:Jul 14 19:32:07 smithi139 bash[158658]: ceph-9adea648-227b-11ee-9b33-001a4aab830c-osd-11 2023-07-14T19:32:08.722 DEBUG:teuthology.orchestra.run.smithi139:> sudo pkill -f 'journalctl -f -n 0 -u ceph-9adea648-227b-11ee-9b33-001a4aab830c@osd.11.service' 2023-07-14T19:32:08.803 DEBUG:teuthology.orchestra.run:got remote process result: None 2023-07-14T19:32:08.803 INFO:tasks.cephadm.osd.11:Stopped osd.11 2023-07-14T19:32:08.804 INFO:tasks.cephadm.mds.a:Stopping mds.a... 2023-07-14T19:32:08.804 DEBUG:teuthology.orchestra.run.smithi098:> sudo systemctl stop ceph-9adea648-227b-11ee-9b33-001a4aab830c@mds.a 2023-07-14T19:32:09.155 INFO:journalctl@ceph.mds.a.smithi098.stdout:Jul 14 19:32:08 smithi098 systemd[1]: Stopping Ceph mds.a for 9adea648-227b-11ee-9b33-001a4aab830c... 2023-07-14T19:32:09.531 INFO:journalctl@ceph.mds.a.smithi098.stdout:Jul 14 19:32:09 smithi098 ceph-9adea648-227b-11ee-9b33-001a4aab830c-mds-a[162148]: 2023-07-14T19:32:09.153+0000 7f9d3f557700 -1 received signal: Terminated from /dev/init -- /usr/bin/ceph-mds -n mds.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2023-07-14T19:32:09.531 INFO:journalctl@ceph.mds.a.smithi098.stdout:Jul 14 19:32:09 smithi098 ceph-9adea648-227b-11ee-9b33-001a4aab830c-mds-a[162148]: 2023-07-14T19:32:09.153+0000 7f9d3f557700 -1 mds.a *** got signal Terminated *** 2023-07-14T19:32:13.408 INFO:journalctl@ceph.mds.a.smithi098.stdout:Jul 14 19:32:13 smithi098 bash[177933]: ceph-9adea648-227b-11ee-9b33-001a4aab830c-mds-a 2023-07-14T19:32:13.414 DEBUG:teuthology.orchestra.run.smithi098:> sudo pkill -f 'journalctl -f -n 0 -u ceph-9adea648-227b-11ee-9b33-001a4aab830c@mds.a.service' 2023-07-14T19:32:13.450 DEBUG:teuthology.orchestra.run:got remote process result: None 2023-07-14T19:32:13.451 INFO:tasks.cephadm.mds.a:Stopped mds.a 2023-07-14T19:32:13.451 INFO:tasks.cephadm.mds.d:Stopping mds.d... 2023-07-14T19:32:13.452 DEBUG:teuthology.orchestra.run.smithi098:> sudo systemctl stop ceph-9adea648-227b-11ee-9b33-001a4aab830c@mds.d 2023-07-14T19:32:13.781 INFO:journalctl@ceph.mds.d.smithi098.stdout:Jul 14 19:32:13 smithi098 systemd[1]: Stopping Ceph mds.d for 9adea648-227b-11ee-9b33-001a4aab830c... 2023-07-14T19:32:14.281 INFO:journalctl@ceph.mds.d.smithi098.stdout:Jul 14 19:32:13 smithi098 ceph-9adea648-227b-11ee-9b33-001a4aab830c-mds-d[161151]: 2023-07-14T19:32:13.826+0000 7f6743193700 -1 received signal: Terminated from /dev/init -- /usr/bin/ceph-mds -n mds.d -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2023-07-14T19:32:14.282 INFO:journalctl@ceph.mds.d.smithi098.stdout:Jul 14 19:32:13 smithi098 ceph-9adea648-227b-11ee-9b33-001a4aab830c-mds-d[161151]: 2023-07-14T19:32:13.826+0000 7f6743193700 -1 mds.d *** got signal Terminated *** 2023-07-14T19:32:18.258 INFO:journalctl@ceph.mds.d.smithi098.stdout:Jul 14 19:32:17 smithi098 bash[178072]: ceph-9adea648-227b-11ee-9b33-001a4aab830c-mds-d 2023-07-14T19:32:18.264 DEBUG:teuthology.orchestra.run.smithi098:> sudo pkill -f 'journalctl -f -n 0 -u ceph-9adea648-227b-11ee-9b33-001a4aab830c@mds.d.service' 2023-07-14T19:32:18.297 DEBUG:teuthology.orchestra.run:got remote process result: None 2023-07-14T19:32:18.297 INFO:tasks.cephadm.mds.d:Stopped mds.d 2023-07-14T19:32:18.298 INFO:tasks.cephadm.mds.g:Stopping mds.g... 2023-07-14T19:32:18.298 DEBUG:teuthology.orchestra.run.smithi098:> sudo systemctl stop ceph-9adea648-227b-11ee-9b33-001a4aab830c@mds.g 2023-07-14T19:32:18.532 INFO:journalctl@ceph.mds.g.smithi098.stdout:Jul 14 19:32:18 smithi098 systemd[1]: Stopping Ceph mds.g for 9adea648-227b-11ee-9b33-001a4aab830c... 2023-07-14T19:32:19.031 INFO:journalctl@ceph.mds.g.smithi098.stdout:Jul 14 19:32:18 smithi098 ceph-9adea648-227b-11ee-9b33-001a4aab830c-mds-g[161764]: 2023-07-14T19:32:18.659+0000 7f67372ff700 -1 received signal: Terminated from /dev/init -- /usr/bin/ceph-mds -n mds.g -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2023-07-14T19:32:19.031 INFO:journalctl@ceph.mds.g.smithi098.stdout:Jul 14 19:32:18 smithi098 ceph-9adea648-227b-11ee-9b33-001a4aab830c-mds-g[161764]: 2023-07-14T19:32:18.659+0000 7f67372ff700 -1 mds.g *** got signal Terminated *** 2023-07-14T19:32:23.105 INFO:journalctl@ceph.mds.g.smithi098.stdout:Jul 14 19:32:22 smithi098 bash[178216]: ceph-9adea648-227b-11ee-9b33-001a4aab830c-mds-g 2023-07-14T19:32:23.112 DEBUG:teuthology.orchestra.run.smithi098:> sudo pkill -f 'journalctl -f -n 0 -u ceph-9adea648-227b-11ee-9b33-001a4aab830c@mds.g.service' 2023-07-14T19:32:23.150 DEBUG:teuthology.orchestra.run:got remote process result: None 2023-07-14T19:32:23.151 INFO:tasks.cephadm.mds.g:Stopped mds.g 2023-07-14T19:32:23.151 INFO:tasks.cephadm.mds.j:Stopping mds.j... 2023-07-14T19:32:23.152 DEBUG:teuthology.orchestra.run.smithi098:> sudo systemctl stop ceph-9adea648-227b-11ee-9b33-001a4aab830c@mds.j 2023-07-14T19:32:23.528 INFO:journalctl@ceph.mds.j.smithi098.stdout:Jul 14 19:32:23 smithi098 systemd[1]: Stopping Ceph mds.j for 9adea648-227b-11ee-9b33-001a4aab830c... 2023-07-14T19:32:23.781 INFO:journalctl@ceph.mds.j.smithi098.stdout:Jul 14 19:32:23 smithi098 ceph-9adea648-227b-11ee-9b33-001a4aab830c-mds-j[160808]: 2023-07-14T19:32:23.526+0000 7fa62a12d700 -1 received signal: Terminated from /dev/init -- /usr/bin/ceph-mds -n mds.j -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2023-07-14T19:32:23.781 INFO:journalctl@ceph.mds.j.smithi098.stdout:Jul 14 19:32:23 smithi098 ceph-9adea648-227b-11ee-9b33-001a4aab830c-mds-j[160808]: 2023-07-14T19:32:23.526+0000 7fa62a12d700 -1 mds.j *** got signal Terminated *** 2023-07-14T19:32:28.031 INFO:journalctl@ceph.mds.j.smithi098.stdout:Jul 14 19:32:27 smithi098 bash[178368]: ceph-9adea648-227b-11ee-9b33-001a4aab830c-mds-j 2023-07-14T19:32:28.079 DEBUG:teuthology.orchestra.run.smithi098:> sudo pkill -f 'journalctl -f -n 0 -u ceph-9adea648-227b-11ee-9b33-001a4aab830c@mds.j.service' 2023-07-14T19:32:28.113 DEBUG:teuthology.orchestra.run:got remote process result: None 2023-07-14T19:32:28.114 INFO:tasks.cephadm.mds.j:Stopped mds.j 2023-07-14T19:32:28.114 INFO:tasks.cephadm.mds.b:Stopping mds.b... 2023-07-14T19:32:28.115 DEBUG:teuthology.orchestra.run.smithi101:> sudo systemctl stop ceph-9adea648-227b-11ee-9b33-001a4aab830c@mds.b 2023-07-14T19:32:28.440 INFO:journalctl@ceph.mds.b.smithi101.stdout:Jul 14 19:32:28 smithi101 systemd[1]: Stopping Ceph mds.b for 9adea648-227b-11ee-9b33-001a4aab830c... 2023-07-14T19:32:28.940 INFO:journalctl@ceph.mds.b.smithi101.stdout:Jul 14 19:32:28 smithi101 ceph-9adea648-227b-11ee-9b33-001a4aab830c-mds-b[154069]: 2023-07-14T19:32:28.476+0000 7fed66556700 -1 received signal: Terminated from /dev/init -- /usr/bin/ceph-mds -n mds.b -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2023-07-14T19:32:28.941 INFO:journalctl@ceph.mds.b.smithi101.stdout:Jul 14 19:32:28 smithi101 ceph-9adea648-227b-11ee-9b33-001a4aab830c-mds-b[154069]: 2023-07-14T19:32:28.476+0000 7fed66556700 -1 mds.b *** got signal Terminated *** 2023-07-14T19:32:30.440 INFO:journalctl@ceph.mds.b.smithi101.stdout:Jul 14 19:32:30 smithi101 bash[157419]: ceph-9adea648-227b-11ee-9b33-001a4aab830c-mds-b 2023-07-14T19:32:30.525 DEBUG:teuthology.orchestra.run.smithi101:> sudo pkill -f 'journalctl -f -n 0 -u ceph-9adea648-227b-11ee-9b33-001a4aab830c@mds.b.service' 2023-07-14T19:32:30.562 DEBUG:teuthology.orchestra.run:got remote process result: None 2023-07-14T19:32:30.562 INFO:tasks.cephadm.mds.b:Stopped mds.b 2023-07-14T19:32:30.563 INFO:tasks.cephadm.mds.e:Stopping mds.e... 2023-07-14T19:32:30.563 DEBUG:teuthology.orchestra.run.smithi101:> sudo systemctl stop ceph-9adea648-227b-11ee-9b33-001a4aab830c@mds.e 2023-07-14T19:32:30.940 INFO:journalctl@ceph.mds.e.smithi101.stdout:Jul 14 19:32:30 smithi101 systemd[1]: Stopping Ceph mds.e for 9adea648-227b-11ee-9b33-001a4aab830c... 2023-07-14T19:32:31.440 INFO:journalctl@ceph.mds.e.smithi101.stdout:Jul 14 19:32:30 smithi101 ceph-9adea648-227b-11ee-9b33-001a4aab830c-mds-e[153382]: 2023-07-14T19:32:30.943+0000 7f54f5214700 -1 received signal: Terminated from /dev/init -- /usr/bin/ceph-mds -n mds.e -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2023-07-14T19:32:31.441 INFO:journalctl@ceph.mds.e.smithi101.stdout:Jul 14 19:32:30 smithi101 ceph-9adea648-227b-11ee-9b33-001a4aab830c-mds-e[153382]: 2023-07-14T19:32:30.943+0000 7f54f5214700 -1 mds.e *** got signal Terminated *** 2023-07-14T19:32:34.440 INFO:journalctl@ceph.mds.e.smithi101.stdout:Jul 14 19:32:34 smithi101 bash[157568]: ceph-9adea648-227b-11ee-9b33-001a4aab830c-mds-e 2023-07-14T19:32:34.467 DEBUG:teuthology.orchestra.run.smithi101:> sudo pkill -f 'journalctl -f -n 0 -u ceph-9adea648-227b-11ee-9b33-001a4aab830c@mds.e.service' 2023-07-14T19:32:34.503 DEBUG:teuthology.orchestra.run:got remote process result: None 2023-07-14T19:32:34.503 INFO:tasks.cephadm.mds.e:Stopped mds.e 2023-07-14T19:32:34.504 INFO:tasks.cephadm.mds.h:Stopping mds.h... 2023-07-14T19:32:34.505 DEBUG:teuthology.orchestra.run.smithi101:> sudo systemctl stop ceph-9adea648-227b-11ee-9b33-001a4aab830c@mds.h 2023-07-14T19:32:34.883 INFO:journalctl@ceph.mds.h.smithi101.stdout:Jul 14 19:32:34 smithi101 systemd[1]: Stopping Ceph mds.h for 9adea648-227b-11ee-9b33-001a4aab830c... 2023-07-14T19:32:35.190 INFO:journalctl@ceph.mds.h.smithi101.stdout:Jul 14 19:32:34 smithi101 ceph-9adea648-227b-11ee-9b33-001a4aab830c-mds-h[153725]: 2023-07-14T19:32:34.883+0000 7f8698482700 -1 received signal: Terminated from /dev/init -- /usr/bin/ceph-mds -n mds.h -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2023-07-14T19:32:35.190 INFO:journalctl@ceph.mds.h.smithi101.stdout:Jul 14 19:32:34 smithi101 ceph-9adea648-227b-11ee-9b33-001a4aab830c-mds-h[153725]: 2023-07-14T19:32:34.883+0000 7f8698482700 -1 mds.h *** got signal Terminated *** 2023-07-14T19:32:38.440 INFO:journalctl@ceph.mds.h.smithi101.stdout:Jul 14 19:32:38 smithi101 bash[157713]: ceph-9adea648-227b-11ee-9b33-001a4aab830c-mds-h 2023-07-14T19:32:38.559 DEBUG:teuthology.orchestra.run.smithi101:> sudo pkill -f 'journalctl -f -n 0 -u ceph-9adea648-227b-11ee-9b33-001a4aab830c@mds.h.service' 2023-07-14T19:32:38.593 DEBUG:teuthology.orchestra.run:got remote process result: None 2023-07-14T19:32:38.594 INFO:tasks.cephadm.mds.h:Stopped mds.h 2023-07-14T19:32:38.594 INFO:tasks.cephadm.mds.k:Stopping mds.k... 2023-07-14T19:32:38.594 DEBUG:teuthology.orchestra.run.smithi101:> sudo systemctl stop ceph-9adea648-227b-11ee-9b33-001a4aab830c@mds.k 2023-07-14T19:32:38.940 INFO:journalctl@ceph.mds.k.smithi101.stdout:Jul 14 19:32:38 smithi101 systemd[1]: Stopping Ceph mds.k for 9adea648-227b-11ee-9b33-001a4aab830c... 2023-07-14T19:32:39.440 INFO:journalctl@ceph.mds.k.smithi101.stdout:Jul 14 19:32:38 smithi101 ceph-9adea648-227b-11ee-9b33-001a4aab830c-mds-k[153039]: 2023-07-14T19:32:38.990+0000 7f7fdc4b5700 -1 received signal: Terminated from /dev/init -- /usr/bin/ceph-mds -n mds.k -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2023-07-14T19:32:39.441 INFO:journalctl@ceph.mds.k.smithi101.stdout:Jul 14 19:32:38 smithi101 ceph-9adea648-227b-11ee-9b33-001a4aab830c-mds-k[153039]: 2023-07-14T19:32:38.991+0000 7f7fdc4b5700 -1 mds.k *** got signal Terminated *** 2023-07-14T19:32:41.847 INFO:journalctl@ceph.mds.k.smithi101.stdout:Jul 14 19:32:41 smithi101 bash[157871]: ceph-9adea648-227b-11ee-9b33-001a4aab830c-mds-k 2023-07-14T19:32:42.036 DEBUG:teuthology.orchestra.run.smithi101:> sudo pkill -f 'journalctl -f -n 0 -u ceph-9adea648-227b-11ee-9b33-001a4aab830c@mds.k.service' 2023-07-14T19:32:42.069 DEBUG:teuthology.orchestra.run:got remote process result: None 2023-07-14T19:32:42.070 INFO:tasks.cephadm.mds.k:Stopped mds.k 2023-07-14T19:32:42.070 INFO:tasks.cephadm.mds.c:Stopping mds.c... 2023-07-14T19:32:42.071 DEBUG:teuthology.orchestra.run.smithi139:> sudo systemctl stop ceph-9adea648-227b-11ee-9b33-001a4aab830c@mds.c 2023-07-14T19:32:42.439 INFO:journalctl@ceph.mds.c.smithi139.stdout:Jul 14 19:32:42 smithi139 systemd[1]: Stopping Ceph mds.c for 9adea648-227b-11ee-9b33-001a4aab830c... 2023-07-14T19:32:42.720 INFO:journalctl@ceph.mds.c.smithi139.stdout:Jul 14 19:32:42 smithi139 ceph-9adea648-227b-11ee-9b33-001a4aab830c-mds-c[155201]: 2023-07-14T19:32:42.438+0000 7f451e9a6700 -1 received signal: Terminated from /dev/init -- /usr/bin/ceph-mds -n mds.c -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2023-07-14T19:32:42.720 INFO:journalctl@ceph.mds.c.smithi139.stdout:Jul 14 19:32:42 smithi139 ceph-9adea648-227b-11ee-9b33-001a4aab830c-mds-c[155201]: 2023-07-14T19:32:42.438+0000 7f451e9a6700 -1 mds.c *** got signal Terminated *** 2023-07-14T19:32:46.300 INFO:journalctl@ceph.mds.c.smithi139.stdout:Jul 14 19:32:46 smithi139 bash[158928]: ceph-9adea648-227b-11ee-9b33-001a4aab830c-mds-c 2023-07-14T19:32:46.486 DEBUG:teuthology.orchestra.run.smithi139:> sudo pkill -f 'journalctl -f -n 0 -u ceph-9adea648-227b-11ee-9b33-001a4aab830c@mds.c.service' 2023-07-14T19:32:46.524 DEBUG:teuthology.orchestra.run:got remote process result: None 2023-07-14T19:32:46.524 INFO:tasks.cephadm.mds.c:Stopped mds.c 2023-07-14T19:32:46.525 INFO:tasks.cephadm.mds.f:Stopping mds.f... 2023-07-14T19:32:46.525 DEBUG:teuthology.orchestra.run.smithi139:> sudo systemctl stop ceph-9adea648-227b-11ee-9b33-001a4aab830c@mds.f 2023-07-14T19:32:46.914 INFO:journalctl@ceph.mds.f.smithi139.stdout:Jul 14 19:32:46 smithi139 systemd[1]: Stopping Ceph mds.f for 9adea648-227b-11ee-9b33-001a4aab830c... 2023-07-14T19:32:47.220 INFO:journalctl@ceph.mds.f.smithi139.stdout:Jul 14 19:32:46 smithi139 ceph-9adea648-227b-11ee-9b33-001a4aab830c-mds-f[154861]: 2023-07-14T19:32:46.913+0000 7fad4b12d700 -1 received signal: Terminated from /dev/init -- /usr/bin/ceph-mds -n mds.f -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2023-07-14T19:32:47.220 INFO:journalctl@ceph.mds.f.smithi139.stdout:Jul 14 19:32:46 smithi139 ceph-9adea648-227b-11ee-9b33-001a4aab830c-mds-f[154861]: 2023-07-14T19:32:46.913+0000 7fad4b12d700 -1 mds.f *** got signal Terminated *** 2023-07-14T19:32:50.272 INFO:journalctl@ceph.mds.f.smithi139.stdout:Jul 14 19:32:50 smithi139 bash[159068]: ceph-9adea648-227b-11ee-9b33-001a4aab830c-mds-f 2023-07-14T19:32:50.461 DEBUG:teuthology.orchestra.run.smithi139:> sudo pkill -f 'journalctl -f -n 0 -u ceph-9adea648-227b-11ee-9b33-001a4aab830c@mds.f.service' 2023-07-14T19:32:50.495 DEBUG:teuthology.orchestra.run:got remote process result: None 2023-07-14T19:32:50.496 INFO:tasks.cephadm.mds.f:Stopped mds.f 2023-07-14T19:32:50.497 INFO:tasks.cephadm.mds.i:Stopping mds.i... 2023-07-14T19:32:50.498 DEBUG:teuthology.orchestra.run.smithi139:> sudo systemctl stop ceph-9adea648-227b-11ee-9b33-001a4aab830c@mds.i 2023-07-14T19:32:50.720 INFO:journalctl@ceph.mds.i.smithi139.stdout:Jul 14 19:32:50 smithi139 systemd[1]: Stopping Ceph mds.i for 9adea648-227b-11ee-9b33-001a4aab830c... 2023-07-14T19:32:51.220 INFO:journalctl@ceph.mds.i.smithi139.stdout:Jul 14 19:32:50 smithi139 ceph-9adea648-227b-11ee-9b33-001a4aab830c-mds-i[155543]: 2023-07-14T19:32:50.886+0000 7fdb49e12700 -1 received signal: Terminated from /dev/init -- /usr/bin/ceph-mds -n mds.i -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2023-07-14T19:32:51.220 INFO:journalctl@ceph.mds.i.smithi139.stdout:Jul 14 19:32:50 smithi139 ceph-9adea648-227b-11ee-9b33-001a4aab830c-mds-i[155543]: 2023-07-14T19:32:50.886+0000 7fdb49e12700 -1 mds.i *** got signal Terminated *** 2023-07-14T19:32:54.423 INFO:journalctl@ceph.mds.i.smithi139.stdout:Jul 14 19:32:54 smithi139 bash[159206]: ceph-9adea648-227b-11ee-9b33-001a4aab830c-mds-i 2023-07-14T19:32:54.429 DEBUG:teuthology.orchestra.run.smithi139:> sudo pkill -f 'journalctl -f -n 0 -u ceph-9adea648-227b-11ee-9b33-001a4aab830c@mds.i.service' 2023-07-14T19:32:54.463 DEBUG:teuthology.orchestra.run:got remote process result: None 2023-07-14T19:32:54.464 INFO:tasks.cephadm.mds.i:Stopped mds.i 2023-07-14T19:32:54.464 INFO:tasks.cephadm.mds.l:Stopping mds.l... 2023-07-14T19:32:54.464 DEBUG:teuthology.orchestra.run.smithi139:> sudo systemctl stop ceph-9adea648-227b-11ee-9b33-001a4aab830c@mds.l 2023-07-14T19:32:54.720 INFO:journalctl@ceph.mds.l.smithi139.stdout:Jul 14 19:32:54 smithi139 systemd[1]: Stopping Ceph mds.l for 9adea648-227b-11ee-9b33-001a4aab830c... 2023-07-14T19:32:55.220 INFO:journalctl@ceph.mds.l.smithi139.stdout:Jul 14 19:32:54 smithi139 ceph-9adea648-227b-11ee-9b33-001a4aab830c-mds-l[154519]: 2023-07-14T19:32:54.843+0000 7f4b29402700 -1 received signal: Terminated from /dev/init -- /usr/bin/ceph-mds -n mds.l -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2023-07-14T19:32:55.220 INFO:journalctl@ceph.mds.l.smithi139.stdout:Jul 14 19:32:54 smithi139 ceph-9adea648-227b-11ee-9b33-001a4aab830c-mds-l[154519]: 2023-07-14T19:32:54.843+0000 7f4b29402700 -1 mds.l *** got signal Terminated *** 2023-07-14T19:32:59.397 INFO:journalctl@ceph.mds.l.smithi139.stdout:Jul 14 19:32:59 smithi139 bash[159344]: ceph-9adea648-227b-11ee-9b33-001a4aab830c-mds-l 2023-07-14T19:32:59.489 DEBUG:teuthology.orchestra.run.smithi139:> sudo pkill -f 'journalctl -f -n 0 -u ceph-9adea648-227b-11ee-9b33-001a4aab830c@mds.l.service' 2023-07-14T19:32:59.522 DEBUG:teuthology.orchestra.run:got remote process result: None 2023-07-14T19:32:59.523 INFO:tasks.cephadm.mds.l:Stopped mds.l 2023-07-14T19:32:59.523 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 9adea648-227b-11ee-9b33-001a4aab830c --force --keep-logs 2023-07-14T19:32:59.826 INFO:teuthology.orchestra.run.smithi098.stdout:Deleting cluster with fsid: 9adea648-227b-11ee-9b33-001a4aab830c 2023-07-14T19:33:01.386 DEBUG:teuthology.orchestra.run.smithi101:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 9adea648-227b-11ee-9b33-001a4aab830c --force --keep-logs 2023-07-14T19:33:01.681 INFO:teuthology.orchestra.run.smithi101.stdout:Deleting cluster with fsid: 9adea648-227b-11ee-9b33-001a4aab830c 2023-07-14T19:33:03.248 DEBUG:teuthology.orchestra.run.smithi139:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 9adea648-227b-11ee-9b33-001a4aab830c --force --keep-logs 2023-07-14T19:33:03.549 INFO:teuthology.orchestra.run.smithi139.stdout:Deleting cluster with fsid: 9adea648-227b-11ee-9b33-001a4aab830c 2023-07-14T19:33:05.079 DEBUG:teuthology.orchestra.run.smithi098:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2023-07-14T19:33:05.115 DEBUG:teuthology.orchestra.run.smithi101:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2023-07-14T19:33:05.152 DEBUG:teuthology.orchestra.run.smithi139:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2023-07-14T19:33:05.181 INFO:tasks.cephadm:Archiving crash dumps... 2023-07-14T19:33:05.189 DEBUG:teuthology.misc:Transferring archived files from smithi098:/var/lib/ceph/9adea648-227b-11ee-9b33-001a4aab830c/crash to /home/teuthworker/archive/pdonnell-2023-07-14_16:44:47-fs:workload-wip-pdonnell-testing-20230714.134335-distro-default-smithi/7338055/remote/smithi098/crash 2023-07-14T19:33:05.191 DEBUG:teuthology.orchestra.run.smithi098:> sudo tar c -f - -C /var/lib/ceph/9adea648-227b-11ee-9b33-001a4aab830c/crash -- . 2023-07-14T19:33:05.237 INFO:teuthology.orchestra.run.smithi098.stderr:tar: /var/lib/ceph/9adea648-227b-11ee-9b33-001a4aab830c/crash: Cannot open: No such file or directory 2023-07-14T19:33:05.238 INFO:teuthology.orchestra.run.smithi098.stderr:tar: Error is not recoverable: exiting now 2023-07-14T19:33:05.240 DEBUG:teuthology.misc:Transferring archived files from smithi101:/var/lib/ceph/9adea648-227b-11ee-9b33-001a4aab830c/crash to /home/teuthworker/archive/pdonnell-2023-07-14_16:44:47-fs:workload-wip-pdonnell-testing-20230714.134335-distro-default-smithi/7338055/remote/smithi101/crash 2023-07-14T19:33:05.241 DEBUG:teuthology.orchestra.run.smithi101:> sudo tar c -f - -C /var/lib/ceph/9adea648-227b-11ee-9b33-001a4aab830c/crash -- . 2023-07-14T19:33:05.282 INFO:teuthology.orchestra.run.smithi101.stderr:tar: /var/lib/ceph/9adea648-227b-11ee-9b33-001a4aab830c/crash: Cannot open: No such file or directory 2023-07-14T19:33:05.282 INFO:teuthology.orchestra.run.smithi101.stderr:tar: Error is not recoverable: exiting now 2023-07-14T19:33:05.285 DEBUG:teuthology.misc:Transferring archived files from smithi139:/var/lib/ceph/9adea648-227b-11ee-9b33-001a4aab830c/crash to /home/teuthworker/archive/pdonnell-2023-07-14_16:44:47-fs:workload-wip-pdonnell-testing-20230714.134335-distro-default-smithi/7338055/remote/smithi139/crash 2023-07-14T19:33:05.286 DEBUG:teuthology.orchestra.run.smithi139:> sudo tar c -f - -C /var/lib/ceph/9adea648-227b-11ee-9b33-001a4aab830c/crash -- . 2023-07-14T19:33:05.327 INFO:teuthology.orchestra.run.smithi139.stderr:tar: /var/lib/ceph/9adea648-227b-11ee-9b33-001a4aab830c/crash: Cannot open: No such file or directory 2023-07-14T19:33:05.328 INFO:teuthology.orchestra.run.smithi139.stderr:tar: Error is not recoverable: exiting now 2023-07-14T19:33:05.329 INFO:tasks.cephadm:Checking cluster log for badness... 2023-07-14T19:33:05.330 DEBUG:teuthology.orchestra.run.smithi098:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/9adea648-227b-11ee-9b33-001a4aab830c/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v 'overall HEALTH_' | egrep -v '\(FS_DEGRADED\)' | egrep -v '\(MDS_FAILED\)' | egrep -v '\(MDS_DEGRADED\)' | egrep -v '\(FS_WITH_FAILED_MDS\)' | egrep -v '\(MDS_DAMAGE\)' | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v '\(FS_INLINE_DATA_DEPRECATED\)' | egrep -v 'overall HEALTH_' | egrep -v '\(OSD_DOWN\)' | egrep -v '\(OSD_' | egrep -v 'but it is still running' | egrep -v 'is not responding' | egrep -v 'slow metadata IO' | egrep -v SLOW_OPS | egrep -v 'slow request' | egrep -v SLOW_OPS | egrep -v 'slow request' | head -n 1 2023-07-14T19:33:05.364 INFO:teuthology.orchestra.run.smithi098.stderr:grep: /var/log/ceph/9adea648-227b-11ee-9b33-001a4aab830c/ceph.log: No such file or directory 2023-07-14T19:33:05.366 INFO:tasks.cephadm:Compressing logs... 2023-07-14T19:33:05.366 DEBUG:teuthology.orchestra.run.smithi098:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2023-07-14T19:33:05.409 DEBUG:teuthology.orchestra.run.smithi101:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2023-07-14T19:33:05.412 DEBUG:teuthology.orchestra.run.smithi139:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2023-07-14T19:33:05.435 INFO:teuthology.orchestra.run.smithi098.stderr:find: ‘/var/log/rbd-target-api’gzip -5 --verbose -- /var/log/ceph/cephadm.log 2023-07-14T19:33:05.436 INFO:teuthology.orchestra.run.smithi098.stderr:: No such file or directory 2023-07-14T19:33:05.436 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/9adea648-227b-11ee-9b33-001a4aab830c/ceph-osd.0.log 2023-07-14T19:33:05.436 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/9adea648-227b-11ee-9b33-001a4aab830c/ceph-mds.g.log 2023-07-14T19:33:05.437 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cephadm.log: /var/log/ceph/9adea648-227b-11ee-9b33-001a4aab830c/ceph-osd.0.log: gzip -5 --verbose -- /var/log/ceph/9adea648-227b-11ee-9b33-001a4aab830c/ceph-mds.a.log 2023-07-14T19:33:05.438 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/9adea648-227b-11ee-9b33-001a4aab830c/ceph-mds.g.log: gzip -5 --verbose -- /var/log/ceph/9adea648-227b-11ee-9b33-001a4aab830c/ceph-osd.6.log 2023-07-14T19:33:05.438 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/9adea648-227b-11ee-9b33-001a4aab830c/ceph-mds.a.log: gzip -5 --verbose -- /var/log/ceph/9adea648-227b-11ee-9b33-001a4aab830c/ceph-osd.3.log 2023-07-14T19:33:05.438 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/9adea648-227b-11ee-9b33-001a4aab830c/ceph-osd.6.log: /var/log/ceph/9adea648-227b-11ee-9b33-001a4aab830c/ceph-osd.3.log: 94.8% -- replaced with /var/log/ceph/9adea648-227b-11ee-9b33-001a4aab830c/ceph-mds.g.log.gz 2023-07-14T19:33:05.439 INFO:teuthology.orchestra.run.smithi098.stderr: 94.7% -- replaced with /var/log/ceph/9adea648-227b-11ee-9b33-001a4aab830c/ceph-mds.a.log.gz 2023-07-14T19:33:05.440 INFO:teuthology.orchestra.run.smithi139.stderr:find: ‘/var/log/rbd-target-api’gzip -5 --verbose -- /var/log/ceph/cephadm.log 2023-07-14T19:33:05.440 INFO:teuthology.orchestra.run.smithi139.stderr:: No such file or directory 2023-07-14T19:33:05.440 INFO:teuthology.orchestra.run.smithi139.stderr:gzip -5 --verbose -- /var/log/ceph/9adea648-227b-11ee-9b33-001a4aab830c/ceph-osd.5.log 2023-07-14T19:33:05.441 INFO:teuthology.orchestra.run.smithi139.stderr:gzip -5 --verbose -- /var/log/ceph/9adea648-227b-11ee-9b33-001a4aab830c/ceph-osd.2.log 2023-07-14T19:33:05.441 INFO:teuthology.orchestra.run.smithi139.stderr:/var/log/ceph/cephadm.log: gzip -5 --verbose -- /var/log/ceph/9adea648-227b-11ee-9b33-001a4aab830c/ceph-osd.11.log 2023-07-14T19:33:05.441 INFO:teuthology.orchestra.run.smithi139.stderr:/var/log/ceph/9adea648-227b-11ee-9b33-001a4aab830c/ceph-osd.5.log: gzip -5 --verbose -- /var/log/ceph/9adea648-227b-11ee-9b33-001a4aab830c/ceph-mgr.z.log 2023-07-14T19:33:05.441 INFO:teuthology.orchestra.run.smithi139.stderr:/var/log/ceph/9adea648-227b-11ee-9b33-001a4aab830c/ceph-osd.2.log: /var/log/ceph/9adea648-227b-11ee-9b33-001a4aab830c/ceph-osd.11.log: gzip -5 --verbose -- /var/log/ceph/9adea648-227b-11ee-9b33-001a4aab830c/ceph-osd.8.log 2023-07-14T19:33:05.441 INFO:teuthology.orchestra.run.smithi139.stderr:/var/log/ceph/9adea648-227b-11ee-9b33-001a4aab830c/ceph-mgr.z.log: 92.1% -- replaced with /var/log/ceph/cephadm.log.gz 2023-07-14T19:33:05.442 INFO:teuthology.orchestra.run.smithi139.stderr:gzip -5 --verbose -- /var/log/ceph/9adea648-227b-11ee-9b33-001a4aab830c/ceph-mds.f.log 2023-07-14T19:33:05.444 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/9adea648-227b-11ee-9b33-001a4aab830c/ceph-mds.d.log 2023-07-14T19:33:05.444 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/9adea648-227b-11ee-9b33-001a4aab830c/ceph-mgr.x.log 2023-07-14T19:33:05.444 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/9adea648-227b-11ee-9b33-001a4aab830c/ceph-mds.d.log: /var/log/ceph/9adea648-227b-11ee-9b33-001a4aab830c/ceph-mgr.x.log: gzip -5 --verbose -- /var/log/ceph/9adea648-227b-11ee-9b33-001a4aab830c/ceph-osd.9.log 2023-07-14T19:33:05.444 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/9adea648-227b-11ee-9b33-001a4aab830c/ceph-osd.9.log: 94.7% -- replaced with /var/log/ceph/9adea648-227b-11ee-9b33-001a4aab830c/ceph-mds.d.log.gz 2023-07-14T19:33:05.445 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/9adea648-227b-11ee-9b33-001a4aab830c/ceph-volume.log 2023-07-14T19:33:05.445 INFO:teuthology.orchestra.run.smithi098.stderr: 93.4% -- replaced with /var/log/ceph/cephadm.log.gz 2023-07-14T19:33:05.445 INFO:teuthology.orchestra.run.smithi101.stderr:find: ‘/var/log/rbd-target-api’gzip -5 --verbose -- /var/log/ceph/cephadm.log 2023-07-14T19:33:05.446 INFO:teuthology.orchestra.run.smithi101.stderr:: No such file or directory 2023-07-14T19:33:05.446 INFO:teuthology.orchestra.run.smithi101.stderr:gzip -5 --verbose -- /var/log/ceph/9adea648-227b-11ee-9b33-001a4aab830c/ceph-osd.10.log 2023-07-14T19:33:05.446 INFO:teuthology.orchestra.run.smithi101.stderr:/var/log/ceph/cephadm.log: gzip -5 --verbose -- /var/log/ceph/9adea648-227b-11ee-9b33-001a4aab830c/ceph-mds.h.log 2023-07-14T19:33:05.446 INFO:teuthology.orchestra.run.smithi101.stderr:gzip -5 --verbose -- /var/log/ceph/9adea648-227b-11ee-9b33-001a4aab830c/ceph-mds.e.log 2023-07-14T19:33:05.447 INFO:teuthology.orchestra.run.smithi101.stderr:/var/log/ceph/9adea648-227b-11ee-9b33-001a4aab830c/ceph-osd.10.log: gzip -5 --verbose -- /var/log/ceph/9adea648-227b-11ee-9b33-001a4aab830c/ceph-osd.4.log 2023-07-14T19:33:05.447 INFO:teuthology.orchestra.run.smithi101.stderr:/var/log/ceph/9adea648-227b-11ee-9b33-001a4aab830c/ceph-mds.h.log: 91.7% -- replaced with /var/log/ceph/cephadm.log.gz 2023-07-14T19:33:05.447 INFO:teuthology.orchestra.run.smithi101.stderr:/var/log/ceph/9adea648-227b-11ee-9b33-001a4aab830c/ceph-mds.e.log: gzip -5 --verbose -- /var/log/ceph/9adea648-227b-11ee-9b33-001a4aab830c/ceph-osd.1.log 2023-07-14T19:33:05.447 INFO:teuthology.orchestra.run.smithi101.stderr:/var/log/ceph/9adea648-227b-11ee-9b33-001a4aab830c/ceph-osd.4.log: gzip -5 --verbose -- /var/log/ceph/9adea648-227b-11ee-9b33-001a4aab830c/ceph-mds.k.log 2023-07-14T19:33:05.447 INFO:teuthology.orchestra.run.smithi101.stderr:/var/log/ceph/9adea648-227b-11ee-9b33-001a4aab830c/ceph-osd.1.log: /var/log/ceph/9adea648-227b-11ee-9b33-001a4aab830c/ceph-mds.k.log: 94.8% -- replaced with /var/log/ceph/9adea648-227b-11ee-9b33-001a4aab830c/ceph-mds.k.log.gz 2023-07-14T19:33:05.448 INFO:teuthology.orchestra.run.smithi101.stderr:gzip -5 --verbose -- /var/log/ceph/9adea648-227b-11ee-9b33-001a4aab830c/ceph-volume.log 2023-07-14T19:33:05.448 INFO:teuthology.orchestra.run.smithi101.stderr:gzip -5 --verbose -- /var/log/ceph/9adea648-227b-11ee-9b33-001a4aab830c/ceph-osd.7.log 2023-07-14T19:33:05.449 INFO:teuthology.orchestra.run.smithi139.stderr:/var/log/ceph/9adea648-227b-11ee-9b33-001a4aab830c/ceph-osd.8.log: gzip -5 --verbose -- /var/log/ceph/9adea648-227b-11ee-9b33-001a4aab830c/ceph-mds.c.log 2023-07-14T19:33:05.449 INFO:teuthology.orchestra.run.smithi139.stderr:/var/log/ceph/9adea648-227b-11ee-9b33-001a4aab830c/ceph-mds.f.log: /var/log/ceph/9adea648-227b-11ee-9b33-001a4aab830c/ceph-mds.c.log: 91.5% -- replaced with /var/log/ceph/9adea648-227b-11ee-9b33-001a4aab830c/ceph-mgr.z.log.gz 2023-07-14T19:33:05.449 INFO:teuthology.orchestra.run.smithi139.stderr:gzip -5 --verbose -- /var/log/ceph/9adea648-227b-11ee-9b33-001a4aab830c/ceph-mds.i.log 2023-07-14T19:33:05.449 INFO:teuthology.orchestra.run.smithi139.stderr:gzip -5 --verbose -- /var/log/ceph/9adea648-227b-11ee-9b33-001a4aab830c/ceph-mon.c.log 2023-07-14T19:33:05.450 INFO:teuthology.orchestra.run.smithi139.stderr:/var/log/ceph/9adea648-227b-11ee-9b33-001a4aab830c/ceph-mds.i.log: /var/log/ceph/9adea648-227b-11ee-9b33-001a4aab830c/ceph-mon.c.log: gzip -5 --verbose -- /var/log/ceph/9adea648-227b-11ee-9b33-001a4aab830c/ceph-volume.log 2023-07-14T19:33:05.450 INFO:teuthology.orchestra.run.smithi101.stderr:/var/log/ceph/9adea648-227b-11ee-9b33-001a4aab830c/ceph-volume.log: /var/log/ceph/9adea648-227b-11ee-9b33-001a4aab830c/ceph-osd.7.log: gzip -5 --verbose -- /var/log/ceph/9adea648-227b-11ee-9b33-001a4aab830c/ceph-mon.b.log 2023-07-14T19:33:05.451 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/9adea648-227b-11ee-9b33-001a4aab830c/ceph-volume.log: gzip -5 --verbose -- /var/log/ceph/9adea648-227b-11ee-9b33-001a4aab830c/ceph-mds.j.log 2023-07-14T19:33:05.451 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/9adea648-227b-11ee-9b33-001a4aab830c/ceph-mds.j.log: 94.7% -- replaced with /var/log/ceph/9adea648-227b-11ee-9b33-001a4aab830c/ceph-mds.j.log.gz 2023-07-14T19:33:05.451 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/9adea648-227b-11ee-9b33-001a4aab830c/ceph-mon.a.log 2023-07-14T19:33:05.451 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/9adea648-227b-11ee-9b33-001a4aab830c/ceph-mon.a.log: 91.6% -- replaced with /var/log/ceph/9adea648-227b-11ee-9b33-001a4aab830c/ceph-volume.log.gz 2023-07-14T19:33:05.453 INFO:teuthology.orchestra.run.smithi139.stderr: 92.6% -- replaced with /var/log/ceph/9adea648-227b-11ee-9b33-001a4aab830c/ceph-mds.f.log.gz 2023-07-14T19:33:05.460 INFO:teuthology.orchestra.run.smithi101.stderr:/var/log/ceph/9adea648-227b-11ee-9b33-001a4aab830c/ceph-mon.b.log: 91.5% -- replaced with /var/log/ceph/9adea648-227b-11ee-9b33-001a4aab830c/ceph-volume.log.gz 2023-07-14T19:33:05.460 INFO:teuthology.orchestra.run.smithi101.stderr: 92.6% -- replaced with /var/log/ceph/9adea648-227b-11ee-9b33-001a4aab830c/ceph-mds.e.log.gz 2023-07-14T19:33:05.460 INFO:teuthology.orchestra.run.smithi101.stderr: 91.2% -- replaced with /var/log/ceph/9adea648-227b-11ee-9b33-001a4aab830c/ceph-mds.h.log.gz 2023-07-14T19:33:05.461 INFO:teuthology.orchestra.run.smithi101.stderr:gzip -5 --verbose -- /var/log/ceph/9adea648-227b-11ee-9b33-001a4aab830c/ceph-mds.b.log 2023-07-14T19:33:05.461 INFO:teuthology.orchestra.run.smithi101.stderr:gzip -5 --verbose -- /var/log/ceph/9adea648-227b-11ee-9b33-001a4aab830c/ceph-mgr.y.log 2023-07-14T19:33:05.461 INFO:teuthology.orchestra.run.smithi101.stderr:/var/log/ceph/9adea648-227b-11ee-9b33-001a4aab830c/ceph-mds.b.log: /var/log/ceph/9adea648-227b-11ee-9b33-001a4aab830c/ceph-mgr.y.log: 91.4% -- replaced with /var/log/ceph/9adea648-227b-11ee-9b33-001a4aab830c/ceph-mgr.y.log.gz 2023-07-14T19:33:05.462 INFO:teuthology.orchestra.run.smithi139.stderr:/var/log/ceph/9adea648-227b-11ee-9b33-001a4aab830c/ceph-volume.log: 92.4% -- replaced with /var/log/ceph/9adea648-227b-11ee-9b33-001a4aab830c/ceph-mds.c.log.gz 2023-07-14T19:33:05.462 INFO:teuthology.orchestra.run.smithi139.stderr:gzip -5 --verbose -- /var/log/ceph/9adea648-227b-11ee-9b33-001a4aab830c/ceph-mds.l.log 2023-07-14T19:33:05.462 INFO:teuthology.orchestra.run.smithi139.stderr:/var/log/ceph/9adea648-227b-11ee-9b33-001a4aab830c/ceph-mds.l.log: 91.4% -- replaced with /var/log/ceph/9adea648-227b-11ee-9b33-001a4aab830c/ceph-volume.log.gz 2023-07-14T19:33:05.462 INFO:teuthology.orchestra.run.smithi139.stderr: 91.5% -- replaced with /var/log/ceph/9adea648-227b-11ee-9b33-001a4aab830c/ceph-mds.i.log.gz 2023-07-14T19:33:05.463 INFO:teuthology.orchestra.run.smithi139.stderr: 94.9% -- replaced with /var/log/ceph/9adea648-227b-11ee-9b33-001a4aab830c/ceph-mds.l.log.gz 2023-07-14T19:33:05.527 INFO:teuthology.orchestra.run.smithi098.stderr: 92.8% -- replaced with /var/log/ceph/9adea648-227b-11ee-9b33-001a4aab830c/ceph-osd.9.log.gz 2023-07-14T19:33:05.531 INFO:teuthology.orchestra.run.smithi101.stderr: 95.6% -- replaced with /var/log/ceph/9adea648-227b-11ee-9b33-001a4aab830c/ceph-mds.b.log.gz 2023-07-14T19:33:05.537 INFO:teuthology.orchestra.run.smithi098.stderr: 92.9% -- replaced with /var/log/ceph/9adea648-227b-11ee-9b33-001a4aab830c/ceph-osd.6.log.gz 2023-07-14T19:33:05.542 INFO:teuthology.orchestra.run.smithi139.stderr: 93.3% -- replaced with /var/log/ceph/9adea648-227b-11ee-9b33-001a4aab830c/ceph-osd.11.log.gz 2023-07-14T19:33:05.550 INFO:teuthology.orchestra.run.smithi101.stderr: 92.8% -- replaced with /var/log/ceph/9adea648-227b-11ee-9b33-001a4aab830c/ceph-osd.4.log.gz 2023-07-14T19:33:05.571 INFO:teuthology.orchestra.run.smithi139.stderr: 92.9% -- replaced with /var/log/ceph/9adea648-227b-11ee-9b33-001a4aab830c/ceph-osd.8.log.gz 2023-07-14T19:33:05.578 INFO:teuthology.orchestra.run.smithi139.stderr: 92.0% -- replaced with /var/log/ceph/9adea648-227b-11ee-9b33-001a4aab830c/ceph-mon.c.log.gz 2023-07-14T19:33:05.579 INFO:teuthology.orchestra.run.smithi098.stderr: 92.8% -- replaced with /var/log/ceph/9adea648-227b-11ee-9b33-001a4aab830c/ceph-osd.3.log.gz 2023-07-14T19:33:05.585 INFO:teuthology.orchestra.run.smithi098.stderr: 89.0% -- replaced with /var/log/ceph/9adea648-227b-11ee-9b33-001a4aab830c/ceph-mgr.x.log.gz 2023-07-14T19:33:05.589 INFO:teuthology.orchestra.run.smithi098.stderr: 93.0% -- replaced with /var/log/ceph/9adea648-227b-11ee-9b33-001a4aab830c/ceph-osd.0.log.gz 2023-07-14T19:33:05.590 INFO:teuthology.orchestra.run.smithi139.stderr: 92.9% -- replaced with /var/log/ceph/9adea648-227b-11ee-9b33-001a4aab830c/ceph-osd.5.log.gz 2023-07-14T19:33:05.592 INFO:teuthology.orchestra.run.smithi139.stderr: 92.9% -- replaced with /var/log/ceph/9adea648-227b-11ee-9b33-001a4aab830c/ceph-osd.2.log.gz 2023-07-14T19:33:05.593 INFO:teuthology.orchestra.run.smithi101.stderr: 92.9% -- replaced with /var/log/ceph/9adea648-227b-11ee-9b33-001a4aab830c/ceph-osd.7.log.gz 2023-07-14T19:33:05.594 INFO:teuthology.orchestra.run.smithi139.stderr: 2023-07-14T19:33:05.594 INFO:teuthology.orchestra.run.smithi139.stderr:real 0m0.166s 2023-07-14T19:33:05.594 INFO:teuthology.orchestra.run.smithi139.stderr:user 0m0.721s 2023-07-14T19:33:05.595 INFO:teuthology.orchestra.run.smithi139.stderr:sys 0m0.036s 2023-07-14T19:33:05.599 INFO:teuthology.orchestra.run.smithi101.stderr: 93.1% -- replaced with /var/log/ceph/9adea648-227b-11ee-9b33-001a4aab830c/ceph-osd.10.log.gz 2023-07-14T19:33:05.617 INFO:teuthology.orchestra.run.smithi101.stderr: 91.9% -- replaced with /var/log/ceph/9adea648-227b-11ee-9b33-001a4aab830c/ceph-mon.b.log.gz 2023-07-14T19:33:05.624 INFO:teuthology.orchestra.run.smithi101.stderr: 93.3% -- replaced with /var/log/ceph/9adea648-227b-11ee-9b33-001a4aab830c/ceph-osd.1.log.gz 2023-07-14T19:33:05.625 INFO:teuthology.orchestra.run.smithi101.stderr: 2023-07-14T19:33:05.626 INFO:teuthology.orchestra.run.smithi101.stderr:real 0m0.198s 2023-07-14T19:33:05.626 INFO:teuthology.orchestra.run.smithi101.stderr:user 0m0.858s 2023-07-14T19:33:05.626 INFO:teuthology.orchestra.run.smithi101.stderr:sys 0m0.053s 2023-07-14T19:33:05.831 INFO:teuthology.orchestra.run.smithi098.stderr: 90.9% -- replaced with /var/log/ceph/9adea648-227b-11ee-9b33-001a4aab830c/ceph-mon.a.log.gz 2023-07-14T19:33:05.833 INFO:teuthology.orchestra.run.smithi098.stderr: 2023-07-14T19:33:05.833 INFO:teuthology.orchestra.run.smithi098.stderr:real 0m0.409s 2023-07-14T19:33:05.833 INFO:teuthology.orchestra.run.smithi098.stderr:user 0m1.006s 2023-07-14T19:33:05.833 INFO:teuthology.orchestra.run.smithi098.stderr:sys 0m0.055s 2023-07-14T19:33:05.834 INFO:tasks.cephadm:Archiving logs... 2023-07-14T19:33:05.834 DEBUG:teuthology.misc:Transferring archived files from smithi098:/var/log/ceph to /home/teuthworker/archive/pdonnell-2023-07-14_16:44:47-fs:workload-wip-pdonnell-testing-20230714.134335-distro-default-smithi/7338055/remote/smithi098/log 2023-07-14T19:33:05.836 DEBUG:teuthology.orchestra.run.smithi098:> sudo tar c -f - -C /var/log/ceph -- . 2023-07-14T19:33:06.034 DEBUG:teuthology.misc:Transferring archived files from smithi101:/var/log/ceph to /home/teuthworker/archive/pdonnell-2023-07-14_16:44:47-fs:workload-wip-pdonnell-testing-20230714.134335-distro-default-smithi/7338055/remote/smithi101/log 2023-07-14T19:33:06.035 DEBUG:teuthology.orchestra.run.smithi101:> sudo tar c -f - -C /var/log/ceph -- . 2023-07-14T19:33:06.157 DEBUG:teuthology.misc:Transferring archived files from smithi139:/var/log/ceph to /home/teuthworker/archive/pdonnell-2023-07-14_16:44:47-fs:workload-wip-pdonnell-testing-20230714.134335-distro-default-smithi/7338055/remote/smithi139/log 2023-07-14T19:33:06.158 DEBUG:teuthology.orchestra.run.smithi139:> sudo tar c -f - -C /var/log/ceph -- . 2023-07-14T19:33:06.261 INFO:tasks.cephadm:Removing cluster... 2023-07-14T19:33:06.262 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 9adea648-227b-11ee-9b33-001a4aab830c --force 2023-07-14T19:33:06.545 INFO:teuthology.orchestra.run.smithi098.stdout:Deleting cluster with fsid: 9adea648-227b-11ee-9b33-001a4aab830c 2023-07-14T19:33:06.996 DEBUG:teuthology.orchestra.run.smithi101:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 9adea648-227b-11ee-9b33-001a4aab830c --force 2023-07-14T19:33:07.303 INFO:teuthology.orchestra.run.smithi101.stdout:Deleting cluster with fsid: 9adea648-227b-11ee-9b33-001a4aab830c 2023-07-14T19:33:07.759 DEBUG:teuthology.orchestra.run.smithi139:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 9adea648-227b-11ee-9b33-001a4aab830c --force 2023-07-14T19:33:08.061 INFO:teuthology.orchestra.run.smithi139.stdout:Deleting cluster with fsid: 9adea648-227b-11ee-9b33-001a4aab830c 2023-07-14T19:33:08.511 INFO:tasks.cephadm:Removing cephadm ... 2023-07-14T19:33:08.512 DEBUG:teuthology.orchestra.run.smithi098:> rm -rf /home/ubuntu/cephtest/cephadm 2023-07-14T19:33:08.533 DEBUG:teuthology.orchestra.run.smithi101:> rm -rf /home/ubuntu/cephtest/cephadm 2023-07-14T19:33:08.554 DEBUG:teuthology.orchestra.run.smithi139:> rm -rf /home/ubuntu/cephtest/cephadm 2023-07-14T19:33:08.572 INFO:tasks.cephadm:Teardown complete 2023-07-14T19:33:08.573 DEBUG:teuthology.run_tasks:Unwinding manager install 2023-07-14T19:33:08.584 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2023-07-14T19:33:08.585 DEBUG:teuthology.orchestra.run.smithi098:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2023-07-14T19:33:08.589 DEBUG:teuthology.orchestra.run.smithi101:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2023-07-14T19:33:08.597 DEBUG:teuthology.orchestra.run.smithi139:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2023-07-14T19:33:23.740 DEBUG:teuthology.run_tasks:Unwinding manager clock 2023-07-14T19:33:23.750 INFO:teuthology.task.clock:Checking final clock skew... 2023-07-14T19:33:23.751 DEBUG:teuthology.orchestra.run.smithi098:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2023-07-14T19:33:23.756 DEBUG:teuthology.orchestra.run.smithi101:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2023-07-14T19:33:23.759 DEBUG:teuthology.orchestra.run.smithi139:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2023-07-14T19:33:23.774 INFO:teuthology.orchestra.run.smithi098.stderr:bash: ntpq: command not found 2023-07-14T19:33:23.776 INFO:teuthology.orchestra.run.smithi101.stderr:bash: ntpq: command not found 2023-07-14T19:33:23.778 INFO:teuthology.orchestra.run.smithi098.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2023-07-14T19:33:23.778 INFO:teuthology.orchestra.run.smithi098.stdout:=============================================================================== 2023-07-14T19:33:23.779 INFO:teuthology.orchestra.run.smithi098.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 36 +1031us[+1030us] +/- 98ms 2023-07-14T19:33:23.779 INFO:teuthology.orchestra.run.smithi098.stdout:^* hv02.front.sepia.ceph.com 3 6 377 36 -3963us[-3964us] +/- 62ms 2023-07-14T19:33:23.779 INFO:teuthology.orchestra.run.smithi098.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 38 +4660us[+4677us] +/- 87ms 2023-07-14T19:33:23.780 INFO:teuthology.orchestra.run.smithi098.stdout:^? hv04.front.sepia.ceph.com 0 8 0 - +0ns[ +0ns] +/- 0ns 2023-07-14T19:33:23.780 INFO:teuthology.orchestra.run.smithi139.stderr:bash: ntpq: command not found 2023-07-14T19:33:23.782 INFO:teuthology.orchestra.run.smithi101.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2023-07-14T19:33:23.782 INFO:teuthology.orchestra.run.smithi101.stdout:=============================================================================== 2023-07-14T19:33:23.782 INFO:teuthology.orchestra.run.smithi101.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 37 +1055us[+1055us] +/- 98ms 2023-07-14T19:33:23.783 INFO:teuthology.orchestra.run.smithi101.stdout:^* hv02.front.sepia.ceph.com 3 7 377 102 -3993us[-3993us] +/- 61ms 2023-07-14T19:33:23.783 INFO:teuthology.orchestra.run.smithi101.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 37 +4649us[+4649us] +/- 87ms 2023-07-14T19:33:23.783 INFO:teuthology.orchestra.run.smithi101.stdout:^? hv04.front.sepia.ceph.com 0 8 0 - +0ns[ +0ns] +/- 0ns 2023-07-14T19:33:23.785 INFO:teuthology.orchestra.run.smithi139.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2023-07-14T19:33:23.785 INFO:teuthology.orchestra.run.smithi139.stdout:=============================================================================== 2023-07-14T19:33:23.785 INFO:teuthology.orchestra.run.smithi139.stdout:^+ hv01.front.sepia.ceph.com 2 7 377 104 +1010us[ +856us] +/- 97ms 2023-07-14T19:33:23.786 INFO:teuthology.orchestra.run.smithi139.stdout:^* hv02.front.sepia.ceph.com 3 6 377 40 -3978us[-3961us] +/- 61ms 2023-07-14T19:33:23.786 INFO:teuthology.orchestra.run.smithi139.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 38 +4645us[+4662us] +/- 87ms 2023-07-14T19:33:23.786 INFO:teuthology.orchestra.run.smithi139.stdout:^? hv04.front.sepia.ceph.com 0 8 0 - +0ns[ +0ns] +/- 0ns 2023-07-14T19:33:23.786 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2023-07-14T19:33:23.839 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2023-07-14T19:33:23.840 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2023-07-14T19:33:23.851 DEBUG:teuthology.orchestra.run.smithi098:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2023-07-14T19:33:23.957 DEBUG:teuthology.orchestra.run.smithi101:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2023-07-14T19:33:24.058 DEBUG:teuthology.orchestra.run.smithi139:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2023-07-14T19:33:24.161 DEBUG:teuthology.orchestra.run.smithi098:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2023-07-14T19:33:24.196 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-07-14T19:33:24.197 DEBUG:teuthology.orchestra.run.smithi101:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2023-07-14T19:33:24.231 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-07-14T19:33:24.231 DEBUG:teuthology.orchestra.run.smithi139:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2023-07-14T19:33:24.264 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-07-14T19:33:24.265 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2023-07-14T19:33:24.276 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2023-07-14T19:33:24.285 INFO:teuthology.task.internal:Duration was 1986.730923 seconds 2023-07-14T19:33:24.286 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2023-07-14T19:33:24.295 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2023-07-14T19:33:24.296 DEBUG:teuthology.orchestra.run.smithi098:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2023-07-14T19:33:24.298 DEBUG:teuthology.orchestra.run.smithi101:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2023-07-14T19:33:24.300 DEBUG:teuthology.orchestra.run.smithi139:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2023-07-14T19:33:24.345 INFO:teuthology.orchestra.run.smithi098.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2023-07-14T19:33:24.346 INFO:teuthology.orchestra.run.smithi101.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2023-07-14T19:33:24.353 INFO:teuthology.orchestra.run.smithi139.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2023-07-14T19:33:24.732 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2023-07-14T19:33:24.733 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi098.front.sepia.ceph.com 2023-07-14T19:33:24.733 DEBUG:teuthology.orchestra.run.smithi098:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2023-07-14T19:33:24.758 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi101.front.sepia.ceph.com 2023-07-14T19:33:24.758 DEBUG:teuthology.orchestra.run.smithi101:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2023-07-14T19:33:24.781 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi139.front.sepia.ceph.com 2023-07-14T19:33:24.782 DEBUG:teuthology.orchestra.run.smithi139:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2023-07-14T19:33:24.804 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2023-07-14T19:33:24.805 DEBUG:teuthology.orchestra.run.smithi098:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2023-07-14T19:33:24.808 DEBUG:teuthology.orchestra.run.smithi101:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2023-07-14T19:33:24.823 DEBUG:teuthology.orchestra.run.smithi139:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2023-07-14T19:33:24.890 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2023-07-14T19:33:24.890 DEBUG:teuthology.orchestra.run.smithi098:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2023-07-14T19:33:24.893 DEBUG:teuthology.orchestra.run.smithi101:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2023-07-14T19:33:24.908 DEBUG:teuthology.orchestra.run.smithi139:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2023-07-14T19:33:24.996 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2023-07-14T19:33:25.009 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2023-07-14T19:33:25.010 DEBUG:teuthology.orchestra.run.smithi098:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2023-07-14T19:33:25.037 DEBUG:teuthology.orchestra.run.smithi101:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2023-07-14T19:33:25.064 DEBUG:teuthology.orchestra.run.smithi139:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2023-07-14T19:33:25.091 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2023-07-14T19:33:25.106 DEBUG:teuthology.orchestra.run.smithi098:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2023-07-14T19:33:25.109 DEBUG:teuthology.orchestra.run.smithi101:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2023-07-14T19:33:25.112 DEBUG:teuthology.orchestra.run.smithi139:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2023-07-14T19:33:25.133 INFO:teuthology.orchestra.run.smithi098.stdout:kernel.core_pattern = core 2023-07-14T19:33:25.135 INFO:teuthology.orchestra.run.smithi101.stdout:kernel.core_pattern = core 2023-07-14T19:33:25.157 INFO:teuthology.orchestra.run.smithi139.stdout:kernel.core_pattern = core 2023-07-14T19:33:25.198 DEBUG:teuthology.orchestra.run.smithi098:> test -e /home/ubuntu/cephtest/archive/coredump 2023-07-14T19:33:25.213 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-07-14T19:33:25.214 DEBUG:teuthology.orchestra.run.smithi101:> test -e /home/ubuntu/cephtest/archive/coredump 2023-07-14T19:33:25.230 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-07-14T19:33:25.231 DEBUG:teuthology.orchestra.run.smithi139:> test -e /home/ubuntu/cephtest/archive/coredump 2023-07-14T19:33:25.253 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-07-14T19:33:25.254 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2023-07-14T19:33:25.265 INFO:teuthology.task.internal:Transferring archived files... 2023-07-14T19:33:25.266 DEBUG:teuthology.misc:Transferring archived files from smithi098:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/pdonnell-2023-07-14_16:44:47-fs:workload-wip-pdonnell-testing-20230714.134335-distro-default-smithi/7338055/remote/smithi098 2023-07-14T19:33:25.266 DEBUG:teuthology.orchestra.run.smithi098:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2023-07-14T19:33:25.307 DEBUG:teuthology.misc:Transferring archived files from smithi101:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/pdonnell-2023-07-14_16:44:47-fs:workload-wip-pdonnell-testing-20230714.134335-distro-default-smithi/7338055/remote/smithi101 2023-07-14T19:33:25.308 DEBUG:teuthology.orchestra.run.smithi101:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2023-07-14T19:33:25.344 DEBUG:teuthology.misc:Transferring archived files from smithi139:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/pdonnell-2023-07-14_16:44:47-fs:workload-wip-pdonnell-testing-20230714.134335-distro-default-smithi/7338055/remote/smithi139 2023-07-14T19:33:25.345 DEBUG:teuthology.orchestra.run.smithi139:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2023-07-14T19:33:25.388 INFO:teuthology.task.internal:Removing archive directory... 2023-07-14T19:33:25.389 DEBUG:teuthology.orchestra.run.smithi098:> rm -rf -- /home/ubuntu/cephtest/archive 2023-07-14T19:33:25.391 DEBUG:teuthology.orchestra.run.smithi101:> rm -rf -- /home/ubuntu/cephtest/archive 2023-07-14T19:33:25.393 DEBUG:teuthology.orchestra.run.smithi139:> rm -rf -- /home/ubuntu/cephtest/archive 2023-07-14T19:33:25.435 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2023-07-14T19:33:25.448 INFO:teuthology.task.internal:Not uploading archives. 2023-07-14T19:33:25.449 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2023-07-14T19:33:25.465 INFO:teuthology.task.internal:Tidying up after the test... 2023-07-14T19:33:25.466 DEBUG:teuthology.orchestra.run.smithi098:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2023-07-14T19:33:25.468 DEBUG:teuthology.orchestra.run.smithi101:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2023-07-14T19:33:25.470 DEBUG:teuthology.orchestra.run.smithi139:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2023-07-14T19:33:25.482 INFO:teuthology.orchestra.run.smithi098.stdout: 262307 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Jul 14 19:33 /home/ubuntu/cephtest 2023-07-14T19:33:25.484 INFO:teuthology.orchestra.run.smithi101.stdout: 262169 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Jul 14 19:33 /home/ubuntu/cephtest 2023-07-14T19:33:25.491 INFO:teuthology.orchestra.run.smithi139.stdout: 262169 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Jul 14 19:33 /home/ubuntu/cephtest 2023-07-14T19:33:25.492 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2023-07-14T19:33:25.510 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2023-07-14T19:33:25.537 INFO:teuthology.nuke:Checking targets against current locks 2023-07-14T19:33:30.567 DEBUG:teuthology.nuke:shortname: smithi098 2023-07-14T19:33:30.567 INFO:teuthology.task.internal.check_lock:Checking locks... 2023-07-14T19:33:38.955 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi098.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/pdonnell-2023-07-14_16:44:47-fs:workload-wip-pdonnell-testing-20230714.134335-distro-default-smithi/7338055', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'rhel', 'os_version': '8.6', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-07-14 18:51:51.079221', 'locked_by': 'scheduled_pdonnell@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBLVvN3+LmzPZ4ixiwyCkS3KZLZ9R6MOPm+kPtaJbl8fExpzbk8TudXCcJWHBL7DGPWFvye2jT6sei9eiX0EUQYs='} 2023-07-14T19:33:38.960 DEBUG:teuthology.nuke:shortname: smithi101 2023-07-14T19:33:38.960 INFO:teuthology.task.internal.check_lock:Checking locks... 2023-07-14T19:33:38.985 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi101.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/pdonnell-2023-07-14_16:44:47-fs:workload-wip-pdonnell-testing-20230714.134335-distro-default-smithi/7338055', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'rhel', 'os_version': '8.6', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-07-14 18:51:51.082748', 'locked_by': 'scheduled_pdonnell@teuthology', 'mac_address': None, 'ssh_pub_key': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCpUxo5ay/ucdkX9rleNMm8Ofp24XRXUtJPOK3Ji3XBpTiUgmmwneaY6316+3CG+k4K0p3IebanuyTvZApjm02/dNCvu8yD3rjEpWVgIuqYH8O3lEdN9DXcCX+7xHJE0m18+ZPsJQVsNrNJxY+5zqyNnBc8jpZq2LStsWwPOKPWhauANRkOBFSGDoLppWZbLx/yJs+R0qoNLLK9LpKQTUnqsapXvbD27OpPahzj/qbNxFPPyY6XrrldZ6qEO+EjJOujFLH3gzv0kJu10DCuzxrpz6ston4RPi1O3GgcHnF4Mqr/W9VwZ94UTMh+k4dIFRVqJEKrcwRHa5kFsvH/WhVx0Ygr/MiV/Q86mmV7JT+/MpzIRTtWyei5L9IBeZIE9cwzj+mOk6SD/t1roIVEGVV/uF2AyFDVBPr6F8oZ822OGpUyyaDsWTitPNdTX4TC3zdSrLc9tyDa1P74eH/ZLeTSf+C6pM/cepOhQaa6XzpDiPSuJUm9HOQxUlEAvcqa7Ac='} 2023-07-14T19:33:38.989 DEBUG:teuthology.nuke:shortname: smithi139 2023-07-14T19:33:38.989 INFO:teuthology.task.internal.check_lock:Checking locks... 2023-07-14T19:33:39.006 INFO:teuthology.orchestra.console:Power off smithi098 2023-07-14T19:33:39.008 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi098.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2023-07-14T19:33:39.025 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi139.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/pdonnell-2023-07-14_16:44:47-fs:workload-wip-pdonnell-testing-20230714.134335-distro-default-smithi/7338055', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'rhel', 'os_version': '8.6', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-07-14 18:51:51.081325', 'locked_by': 'scheduled_pdonnell@teuthology', 'mac_address': None, 'ssh_pub_key': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCpUxo5ay/ucdkX9rleNMm8Ofp24XRXUtJPOK3Ji3XBpTiUgmmwneaY6316+3CG+k4K0p3IebanuyTvZApjm02/dNCvu8yD3rjEpWVgIuqYH8O3lEdN9DXcCX+7xHJE0m18+ZPsJQVsNrNJxY+5zqyNnBc8jpZq2LStsWwPOKPWhauANRkOBFSGDoLppWZbLx/yJs+R0qoNLLK9LpKQTUnqsapXvbD27OpPahzj/qbNxFPPyY6XrrldZ6qEO+EjJOujFLH3gzv0kJu10DCuzxrpz6ston4RPi1O3GgcHnF4Mqr/W9VwZ94UTMh+k4dIFRVqJEKrcwRHa5kFsvH/WhVx0Ygr/MiV/Q86mmV7JT+/MpzIRTtWyei5L9IBeZIE9cwzj+mOk6SD/t1roIVEGVV/uF2AyFDVBPr6F8oZ822OGpUyyaDsWTitPNdTX4TC3zdSrLc9tyDa1P74eH/ZLeTSf+C6pM/cepOhQaa6XzpDiPSuJUm9HOQxUlEAvcqa7Ac='} 2023-07-14T19:33:39.036 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi098.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-07-14T19:33:39.064 INFO:teuthology.orchestra.console:Power off smithi101 2023-07-14T19:33:39.065 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi101.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2023-07-14T19:33:39.090 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi101.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-07-14T19:33:39.112 INFO:teuthology.orchestra.console:Power off smithi139 2023-07-14T19:33:39.113 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi139.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2023-07-14T19:33:39.141 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi139.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-07-14T19:33:43.081 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi098.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-07-14T19:33:43.129 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi101.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-07-14T19:33:43.169 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi139.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-07-14T19:33:47.197 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi098.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-07-14T19:33:47.242 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi101.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-07-14T19:33:47.284 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi139.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-07-14T19:33:47.414 INFO:teuthology.orchestra.console:Power off for smithi098 completed 2023-07-14T19:33:47.454 INFO:teuthology.orchestra.console:Power off for smithi101 completed 2023-07-14T19:33:47.498 INFO:teuthology.orchestra.console:Power off for smithi139 completed 2023-07-14T19:33:47.606 INFO:teuthology.run:Summary data: description: fs:workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/crc wsync/yes} objectstore-ec/bluestore-bitmap omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{balancer/random export-check n/3 replication/default} standby-replay tasks/{0-subvolume/{with-no-extra-options} 1-check-counter 2-scrub/yes 3-snaps/yes 4-flush/yes 5-workunit/suites/ffsb}} duration: 1986.7309234142303 failure_reason: 'Command failed on smithi098 with status 1: "sudo TESTDIR=/home/ubuntu/cephtest bash -c ''ceph fs subvolumegroup pin cephfs _nogroup random 0.01''"' flavor: default owner: scheduled_pdonnell@teuthology sentry_event: https://sentry.ceph.com/organizations/ceph/?query=23a9ab8569fe49bbaf9c6dc2feefbb78 status: fail success: false 2023-07-14T19:33:47.606 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2023-07-14T19:33:47.691 INFO:teuthology.run:FAIL